WO2025089587A1 - Cargo unloading automation method based on cargo vision recognition and computing device for performing same - Google Patents

Abstract

Disclosed are a cargo unloading automation method based on cargo vision recognition and a computing device for performing same. The cargo unloading automation method disclosed according to an embodiment is a method performed by a cargo unloading automation machine including one or more processors and a memory for storing one or more programs executed by the one or more processors and includes the steps of: obtaining an RGB image and a depth image for loaded cargo by using an RGB-D (Red Green Blue Depth) sensor; obtaining a point cloud for the loaded cargo based on one of the RGB image and the depth image; obtaining boundary line information for individual cargo items from the RGB image; and recognizing each of the structured cargo items and the unstructured cargo items of the loaded cargo on the basis of the depth image, the point cloud, and the boundary line information.

Description

Cargo unloading automation method based on cargo vision recognition and computing device for performing the same

An embodiment of the present invention relates to a cargo unloading automation technology. The present invention is derived from research conducted as part of the Robot Industry Core Technology Development Project of the Ministry of Trade, Industry and Energy - Inter-ministerial Cooperation Robot Product Technology (Project Unique Number: 1415168943, Subproject Number: 20009109, Research Project Name: Development of a Trunk Freight Logistics Vehicle Unloading Work System Using a Robot, Organized by: Korea Industrial Technology Evaluation and Planning Institute, Research Period: 2020.05.01 ~ 2024.12.31).

In the past three years, the volume of parcel delivery has increased by 10-13% each year, but the average cost of parcel delivery has decreased by 2-3% each year, so automation in the logistics industry is becoming very important for companies to maintain their competitiveness. Currently, a significant portion of the parcel delivery process is automated in logistics centers using equipment such as belt sorters, but the unloading work of unloading various parcel cargoes from the vehicle's loading compartment is still dependent on manual labor.

In addition, in order to handle the nationwide daily delivery demand, trunk cargo trucks at hub terminals are usually loaded with about 2,000 pieces of cargo per truck, and two workers must unload them all within one hour. This is the most intense of the various tasks at the parcel hub terminal, and is a task that workers avoid. Therefore, the development of a robot system that can automate the unloading work of trunk cargo trucks is required.

The purpose of the present invention is to provide an automated cargo unloading method capable of recognizing cargo in a loading compartment and automatically unloading it, and a computing device for performing the same.

Meanwhile, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary knowledge in the technical field to which the present invention belongs from the description below.

According to one embodiment of the present invention, a cargo unloading automation method is provided, which is performed in a cargo unloading automation machine including one or more processors and a memory storing one or more programs executed by the one or more processors, comprising: a step of obtaining an RGB image and a depth image of loaded cargoes using an RGB-D (Red Green Blue Depth) sensor; a step of obtaining a point cloud of the loaded cargoes based on at least one of the RGB image and the depth image; a step of obtaining boundary information of individual cargoes from the RGB image; and a step of recognizing regular cargoes and irregular cargoes of the loaded cargoes based on the depth image, the point cloud, and the boundary information, respectively.

The step of obtaining the above boundary information obtains boundary information of individual cargo from the RGB image using a deep learning-based object segmentation model, and the object segmentation model may be a model learned to detect individual cargo in a state where regular cargo and irregular cargo are mixedly loaded and to detect the boundary of each cargo.

The above-described recognizing step may include, in the case of a regular cargo among the loaded cargoes, a step of performing a polyhedral transformation on a point cloud of the corresponding cargo to obtain a main plane of the corresponding cargo; a step of obtaining an outline of the corresponding cargo based on a point cloud on a boundary line projected onto the main plane; and a step of performing a rotation caliper along the obtained outline to estimate the position, direction, and size of a three-dimensional oriented bounding box of the corresponding cargo.

The above-mentioned recognizing step may include, in the case of irregular cargo among the loaded cargoes, a step of obtaining a segmented point cloud by merging the boundary line with a point cloud obtained from inside the boundary line of the corresponding cargo; and a step of estimating the position, direction, and size of the corresponding cargo based on the segmented point cloud.

A computing device according to one embodiment of the present disclosure comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, wherein the one or more programs include: a command for obtaining an RGB image and a depth image of loaded cargoes using an RGB-D (Red Green Blue Depth) sensor; a command for obtaining a point cloud of the loaded cargoes based on at least one of the RGB image and the depth image; a command for obtaining boundary information of individual cargoes from the RGB image; and a command for recognizing regular cargoes and irregular cargoes of the loaded cargoes based on the depth image, the point cloud, and the boundary information, respectively.

According to an embodiment of the present invention, a cargo unloading automation machine has a pair of robot arm devices capable of multi-axis rotation and movement, and since the robot arm devices have an unloading conveyor belt, an adsorption means, and a clamp means, the machine operates in various unloading modes, such as a down sweep mode, a side sweep mode, an adsorption mode, and a clamping mode, depending on the position of the loaded cargo, the loading pattern of the cargo, the type of the cargo, etc., so that the loaded cargo can be unloaded easily and quickly.

In addition, the speed of cargo recognition can be increased by recognizing each individual cargo without a separate 3D model conversion process based on the RGB image and depth image of the loaded cargo, and regular cargo and irregular cargo can be recognized separately.

Meanwhile, the effects obtainable from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention belongs from the description below.

Figures 1 and 2 are a front perspective view and a bottom perspective view showing an automated cargo unloading machine according to one embodiment of the present invention.

FIG. 3 is a perspective view showing a robot arm device according to one embodiment of the present invention.

FIG. 4 is a drawing showing each axis part of the arm unit of a robot arm device according to one embodiment of the present invention.

FIGS. 5 and 6 are perspective views showing one side and the other side of a hand unit according to one embodiment of the present invention.

Figures 7a and 7b are drawings showing a state in which the suction pad part moves forward in one embodiment of the present invention.

Figures 8a and 8b are drawings showing a state in which the clamping part moves forward in one embodiment of the present invention.

FIGS. 9 to 15c are drawings showing the operation of a cargo unloading automation machine according to one embodiment of the present invention unloading loaded cargo from a loading box.

FIG. 16 is a drawing showing a process of determining an unloading mode in an automated cargo unloading method according to one embodiment of the present invention.

FIG. 17 is a flow chart showing a method for recognizing individual loaded cargo in a cargo unloading automation method according to one embodiment of the present invention.

FIG. 18 is a diagram showing a state in which labeling is performed on learning data for learning an object segmentation model in one embodiment of the present invention.

FIG. 19 is a drawing showing the result of object segmentation using an RGB image of a loaded cargo and an object segmentation model in one embodiment of the present invention.

Figure 20 is a drawing showing a process of recognizing regular cargo and irregular cargo in one embodiment of the present invention.

FIG. 21 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the attached drawings. The embodiments of the present invention can be modified in various forms, and the scope of the present invention should not be construed as being limited to the embodiments below. The embodiments are provided to more completely explain the present invention to those with average knowledge in the art. Therefore, the shapes of the elements in the drawings are exaggerated to emphasize a clearer explanation.

In order to clearly explain the solution to the problem to be solved by the present invention, the composition of the invention will be described in detail based on a preferred embodiment of the present invention with reference to the attached drawings. In assigning reference numbers to components in the drawings, the same reference numbers are assigned to the same components even if they are in different drawings. It is to be noted in advance that components in other drawings may be cited when necessary when describing the drawings.

Meanwhile, directional terms such as upper, lower, one side, the other side, etc. are used in connection with the orientation of the disclosed drawings. Since the components of the embodiments of the present invention may be positioned in various orientations, directional terms are used for illustrative purposes and not for limitation.

FIGS. 1 and 2 are a front perspective view and a bottom perspective view showing an automated cargo unloading machine according to one embodiment of the present invention.

Referring to FIGS. 1 and 2, the cargo unloading automation machine (100) may include a robot arm device (102), a conveyor device (104), a track device (106), and a main body frame (108).

The cargo unloading automatic machine (100) is for unloading cargo (hereinafter, referred to as loaded cargo) loaded in a loading box (50) or transporting it after unloading. Here, the loading box (50) may be a loading box mounted on a cargo vehicle (e.g., a trunk cargo vehicle, etc.) or may be a container. The cargo unloading automatic machine (100) may unload the loaded cargo in the loading box (50) at the entrance of the loading box (50), or may enter the interior of the loading box (50) and unload the loaded cargo in the loading box.

In addition, the cargo unloading automation machine (100) may include various sensor equipment (e.g., vision sensor or Lidar or vision and Lidar fusion sensor, scanner, etc.) to recognize cargo loading pattern, cargo loading location, cargo type, etc. and determine cargo unloading mode accordingly.

The robot arm device (102) may be provided on both sides of the main body frame (108). That is, the robot arm devices (102) may be provided as a pair on both sides of the main body frame (108). The robot arm devices (102) may be provided to operate as a pair like human arms to unload various types of cargo, such as boxes, vinyl pouches, and sacks, that are piled in bulk form in a loading bin. The robot arm device (102) may unload the loaded cargo in the loading bin toward the conveyor device (104).

A conveyor device (104) may be mounted on the main body frame (108). The conveyor device (104) may be provided along the length direction of the robot arm device (102) on the main body frame (108). The conveyor device (104) may serve to transport a loaded cargo unloaded by the robot arm device (102) to the rear of the main body frame (108).

The conveyor device (104) can operate in conjunction with the robot arm device (102). The front end of the conveyor device (104) can be provided so that its height is adjusted according to the operation of the robot arm device (102). At this time, the front end of the conveyor device (104) can be adjusted in height according to the height of the loaded cargo to be unloaded by the robot arm device (102).

The track device (106) may be provided at the bottom of the main body frame (108). The track device (106) may play a role in moving the cargo unloading automation machine (100). That is, the track device (106) may move the cargo unloading automation machine (100) to the entrance of the loading box or cause it to enter the inside of the loading box. Meanwhile, although the moving means of the cargo unloading automation machine (100) is described as the track device (106) here, it is not limited thereto, and it is obvious that a wheel-type moving means may also be included.

The main body frame (108) can serve to support the cargo unloading automation machine (100). The main body frame (108) can be provided in various shapes that can support the cargo unloading automation machine (100), and its shape is not limited. The main body frame (108) can be provided to support the robot arm device (102). The main body frame (108) can be provided to support the conveyor device (104). In addition, the main body frame (108) can be connected to the track device (106) at the upper portion of the track device (106).

The main body frame (108) may be equipped with a power box (108a) that provides power to the cargo unloading automation machine (100) and includes circuit devices that control the operation of the cargo unloading automation machine (100). In addition, the main body frame (108) may be equipped with a hydraulic tank (108b) for providing hydraulic pressure to at least one of the robot arm device (102) and the track device (106). In addition, the main body frame (108) may be equipped with at least one monitor (108c) for checking the operation of the cargo unloading automation machine (100).

FIG. 3 is a perspective view showing a robot arm device (102) according to one embodiment of the present invention, and FIG. 4 is a drawing showing each axis part of the arm unit of the robot arm device (102) according to one embodiment of the present invention.

Referring to FIGS. 3 and 4, the robotic arm device (102) may include an arm unit (111) and a hand unit (113).

The robot arm device (102) can be operated in an unloading mode determined according to one or more of a loading pattern of cargo in a loading box, a loading location of cargo, and a type of cargo. The unloading mode is an operation mode for unloading cargo in a loading box, and for example, the unloading mode can include a down sweep mode, a side sweep mode, a suction mode, and a clamping mode.

The arm unit (111) may be mounted on both sides of the main body frame (108). The arm unit (111) may be provided to enable multi-axis rotation. In addition, the arm unit (111) may be provided to enable forward and backward movement. The arm unit (111) may be provided to perform multi-axis rotation and forward or backward movement simultaneously.

In an exemplary embodiment, the arm unit (111) may be configured to have six degrees of freedom. The arm unit (111) may include a first axis portion (111-1), a second axis portion (111-2), a third axis portion (111-3), a fourth axis portion (111-4), a fifth axis portion (111-5), and a sixth axis portion (111-6).

The first shaft part (111-1) may be mounted on the main body frame (108). The first shaft part (111-1) may be provided to be rotatable in a first direction (1). The second shaft part (111-2) may be provided to be connected to the first shaft part (111-1). The second shaft part (111-2) may be provided to be rotatable in a second direction (2). The third shaft part (111-3) may be provided to be connected to the second shaft part (111-2). The third shaft part (111-3) may be provided to be capable of moving forward and backward along the third direction (3).

The fourth axis part (111-4) may be provided connected to the third axis part (111-3). The fourth axis part (111-4) may be provided so as to be rotatable in the fourth direction (4). The fifth axis part (111-5) may be provided connected to the fourth axis part (111-4). The fifth axis part (111-5) may be provided so as to be rotatable in the fifth direction (5). The sixth axis part (111-6) may be provided connected to the fifth axis part (111-5). The sixth axis part (111-6) may be provided so as to be rotatable in the sixth direction (6). Here, the first direction (1) to the sixth direction (6) may be different directions from each other. In this case, the arm unit (111) can move forward and backward while rotating in the five axial directions.

The hand unit (113) may be provided connected to the end of the arm unit (111). The hand unit (113) is a portion that comes into contact with the loaded cargo and is a portion that unloads the loaded cargo toward the conveyor device (104). The hand unit (113) may be provided connected to the sixth axis portion (111-6). Since the hand unit (113) is connected to the arm unit (111), the hand unit (113) can move forward and backward while rotating in five axial directions together with the arm unit (111) according to the operation of the arm unit (111). In this way, since the robot arm device (102) is provided to have six degrees of freedom, it is possible to cover the entire range within the loading box and unload the loaded cargo.

Figures 5 and 6 are perspective views showing one side and the other side of a hand unit (113) according to one embodiment of the present invention. The hand unit (113) may include a disembarkation conveyor belt (121), an adsorption means (123), and a clamp means (125).

An unloading conveyor belt (121) may be mounted on one side of the hand unit (113). When the robot arm device (102) is likened to a human arm, one side of the hand unit (113) may be likened to a palm. The unloading conveyor belt (121) may be provided along the length direction of the arm unit (111). The unloading conveyor belt (121) may be provided to rotate in a certain direction. For example, the unloading conveyor belt (121) may be provided in a loop shape to rotate cyclically.

The unloading conveyor belt (121) may be arranged to rotate toward the main body frame (108) (i.e., inwardly). That is, the unloading conveyor belt (121) of each hand unit (113) of a pair of robot arm devices (102) may be arranged to rotate toward the main body frame (108).

As the pair of unloading conveyor belts (121) rotate toward the main body frame (108), the loaded cargoes that come into contact with the pair of unloading conveyor belts (121) between the pair of unloading conveyor belts (121) are swept toward the main body frame (108) and unloaded toward the conveyor device (104). That is, the unloading conveyor belt (121) is used in a sweep mode and can be used when the unloading mode of the robot arm device (102) is a down sweep mode or a side sweep mode.

A rough pattern portion (121a) may be formed on the surface of the unloading conveyor belt (121) to increase friction with the loaded cargo when sweeping the loaded cargo. The rough pattern portion (121a) may be provided in a form that protrudes from the surface of the unloading conveyor belt (121), but is not limited thereto. A plurality of rough pattern portions (121a) may be provided at regular intervals along the length direction of the unloading conveyor belt (121).

The other side of the hand unit (113) may be provided with a suction means (123) and a clamp means (125), respectively. When the robot arm device (102) is likened to a human arm, the other side of the hand unit (113) may be likened to the back of the hand. The suction means (123) and the clamp means (125) may be arranged vertically, respectively, on the other side of the hand unit (113). The suction means (123) and the clamp means (125) may be provided along the longitudinal direction of the hand unit (113), respectively.

In an exemplary embodiment, the suction means (123) and the clamp means (125) may be mounted to the hand unit (113) via a hand bracket (127). Both sides of the hand bracket (127) may be fixed to both sides of the unloading conveyor belt (121). The hand bracket (127) may be provided between the unloading conveyor belt (121) and the suction means (123) and the clamp means (125).

The suction means (123) can play a role in sucking and unloading the loaded cargo in the loading box. That is, the suction means (123) can be used when the unloading mode of the robot arm device (102) is the suction mode. For example, the suction means (123) can be arranged to move forward and backward along the longitudinal direction of the hand unit (113). The suction means (123) can include a suction pad portion (123a) and a suction drive portion (123b).

The suction pad section (123a) is a section that absorbs the loaded cargo. A plurality of suction pad sections (123a) may be provided. One end of each of the plurality of suction pad sections (123a) may be fixed to a mounting plate (123a-2). The mounting plate (123a-2) may be connected to the suction drive section (123b). Here, the suction pad section (123a) is illustrated as being provided in a plurality of pieces, but is not limited thereto.

A suction cup (123a-1) for vacuum-absorbing a loaded cargo may be provided at the end of the suction pad section (123a). The suction cup (123a-1) may be provided in a Javara shape so as to be able to bend in various directions, but its shape is not limited thereto.

The suction drive unit (123b) may be provided to move the suction pad unit (123a) forward and backward. In addition, the suction drive unit (123b) may include a vacuum generator so that the suction pad unit (123a) absorbs the loaded cargo.

Figures 7a and 7b are drawings showing a state in which the suction pad part (123a) moves forward in one embodiment of the present invention. The suction drive part (123b) can move the suction pad part (123a) forward in the suction mode to suction the loaded cargo. The suction drive part (123b) can move the suction pad part (123a) that has unloaded the suctioned loaded cargo backward to return it to its original position.

Here, it is described that the suction pad part (123a) moves forward to absorb the loaded cargo, but it is not limited to this, and the hand unit (111) may move toward the loaded cargo while the suction pad part (123a) is fixed, and then the loaded cargo may be absorbed through the suction pad part (123a).

The clamp means (125) can play a role in picking up and unloading a loaded cargo within a loading compartment. That is, the clamp means (125) can be used when the unloading mode of the robot arm device (102) is a clamping mode. For example, the clamp means (125) can be arranged to move forward and backward along the longitudinal direction of the hand unit (113). The clamp means (125) can include a clamping portion (125a) and a clamp driving portion (125b).

The clamping part (125a) is a part that picks up the loaded cargo. At this time, the type of the loaded cargo may be a sack. That is, the clamping part (125a) may be provided to pick up a sack among the loaded cargo. For this purpose, the end of the clamping part (125a) may be provided in the form of a pair of tongs. The clamp driving part (125b) may be provided to move the clamping part (125a) forward and backward.

Figures 8a and 8b are drawings showing a state in which the clamping part (125a) moves forward in one embodiment of the present invention. The clamp driving part (125b) can move the clamping part (125a) forward in the clamping mode to pick up the loaded cargo. The clamp driving part (125b) can move the clamping part (125a) that has unloaded the loaded cargo backward to return it to its original position.

Here, it is described that the clamping part (125a) moves forward to pick up the loaded cargo, but it is not limited to this, and it may be arranged that the hand unit (111) moves toward the loaded cargo while the clamping part (125a) is fixed and then the loaded cargo is clamped through the clamping part (125a).

FIGS. 9 to 15c are drawings showing the operation of an automated cargo unloading machine according to one embodiment of the present invention unloading loaded cargo from a loading bin.

Referring to Fig. 9, the cargo unloading automation machine (100) can recognize the distance to the cargo, the cargo loading pattern, the cargo loading position, and the type of cargo through a vision sensor, etc. At this time, the cargo unloading automation machine (100) can move toward the cargo according to the distance from the cargo. The cargo unloading automation machine (100) can sequentially unload the cargo according to the height of the loaded cargo.

Referring to FIG. 10, the cargo unloading automation machine (100) can adjust the front end height of the conveyor device (104) according to the loading height of the cargo. The cargo unloading automation machine (100) can adjust the front end height of the conveyor device (104) based on the height of the cargo located at the top among the loaded cargo. That is, since the robot arm device (102) unloads the cargo located at the top among the loaded cargo, the front end height of the conveyor device (104) can be adjusted to a height that can safely receive the cargo located at the top among the loaded cargo.

Referring to FIG. 11, the cargo unloading automation machine (100) can position the robot arm device (102) above the cargo in order to unload the cargo located at the top among the loaded cargo. In an exemplary embodiment, if the cargo located at the top among the loaded cargo is in the shape of a box and an entry space for the robot arm device (102) is secured above the cargo located at the top, the robot arm device (102) can operate in a down sweep mode to unload the cargo located at the top. At this time, the robot arm device (102) can unload the cargo located at the outermost part among the cargo located at the top among the loaded cargo in the down sweep mode, but is not limited thereto.

Referring to FIG. 12, the robot arm device (102) is shown in a state in which the cargo located at the top among the loaded cargo is unloaded toward the conveyor device (104) through the down sweep mode. Here, the down sweep mode may mean an operation mode in which the unloading conveyor belt (121) of the hand unit (113) is positioned to face the target cargo (i.e., the cargo to be unloaded) from above the target cargo (i.e., the surface of the unloading conveyor belt (121) faces the upper surface of the target cargo from above the target cargo) and the unloading conveyor belt (121) is rotated toward the main body frame (108) to sweep down the target cargo. At this time, depending on the number of cargo to be unloaded or the location of the cargo through the down sweep mode, both of the unloading conveyor belts (121) may operate in the down sweep mode, or only one of them may operate in the down sweep mode.

Referring to FIGS. 13A to 13C, the robot arm device (102) is shown in a state of unloading loaded cargo through a side sweep mode. In an exemplary embodiment, the robot arm device (102) may unload cargo located at the outermost end among the loaded cargoes located at the top end in a down sweep mode, and then unload cargoes located at the center end in a side sweep mode. However, the present invention is not limited thereto, and if the cargo located at the top end among the loaded cargoes is in a box shape and an entry space for the robot arm device (102) is secured on the side of the cargo located at the top end, the robot arm device (102) may operate in the side sweep mode to unload the cargo.

Here, the side sweep mode may mean an operation mode in which one or more target cargoes are positioned between a pair of unloading conveyor belts (121), and a pair of unloading conveyor belts (121) are positioned to face the target cargo (i.e., cargo to be unloaded) on both sides of the target cargo (i.e., a state in which the surface of the unloading conveyor belt (121) faces the side of the target cargo on the side of the target cargo) and the pair of unloading conveyor belts (121) are rotated toward the main body frame (108) to sweep down the target cargo.

Specifically, the robot arm device (102) can cause the hand unit (113) to enter the side space of the loaded cargo (FIG. 13a). Next, the robot arm device (102) can arrange a pair of unloading conveyor belts (121) of the hand unit (113) to face the side of the target cargo (FIG. 13b). Next, the robot arm device (102) can rotate the pair of unloading conveyor belts (121) toward the main body frame (108) to unload the target cargo downward (FIG. 13c). At this time, the cargo unloading automation machine (100) can continue to operate in a side sweep mode while bringing the pair of robot arm devices (102) inward to each other as the loaded cargo is unloaded.

Referring to FIGS. 14A to 14C, a state in which the robot arm device (102) unloads a loaded cargo through a clamping mode is illustrated. In an exemplary embodiment, the cargo unloading automation machine (100) can operate the robot arm device (102) in the clamping mode depending on the type of cargo to be unloaded. For example, if the type of cargo to be unloaded is a cargo that can be held through a clamping means (125), such as a sack or pouch, the cargo unloading automation machine (100) can operate the robot arm device (102) in the clamping mode.

Specifically, when the type of cargo is a sack (Fig. 14a), the robot arm device (102) can move the hand unit (113) toward the target cargo and then grab the target cargo through the clamping part (125a) of the clamping means (125) (Fig. 14b). At this time, the clamping part (125a) can also be moved forward depending on the distance from the target cargo. Next, the robot arm device (102) can pull the target cargo down to unload the target cargo (Fig. 14c).

Referring to FIGS. 15A to 15C, a state in which the robot arm device (102) unloads a loaded cargo through the suction mode is illustrated. In an exemplary embodiment, the robot arm device (102) can unload a cargo through the suction mode depending on at least one of the loading height of the cargo and the type of the cargo. For example, when the target cargo is located at the bottom or top and the cargo is in the shape of a box, the robot arm device (102) can unload the target cargo through the suction mode. FIGS. 15A to 15C illustrate a state in which the cargo is unloaded through the suction mode when the target cargo is in the shape of a box located at the bottom.

Specifically, the robot arm device (102) can adsorb the target cargo through the adsorption pad portion (123a) of the adsorption means (123) (Fig. 15a). At this time, the adsorption pad portion (123a) can also be moved forward depending on the distance from the target cargo. Next, the robot arm device (102) can move the target cargo to the upper part of the conveyor device (104) while adsorbing the target cargo (Fig. 15b). Next, the robot arm device (102) can detach the target cargo from the adsorption pad portion (123a) and unload the target cargo onto the conveyor device (104) (Fig. 15c).

According to the disclosed embodiment, a cargo unloading automation machine (100) is provided with a pair of robot arm devices (102) capable of multi-axis rotation and movement, and since the robot arm devices (102) are provided with an unloading conveyor belt (121), an adsorption means (123), and a clamp means (125), the machine operates in various unloading modes, such as a down sweep mode, a side sweep mode, an adsorption mode, and a clamping mode, depending on the position of the loaded cargo, the loading pattern of the cargo, the type of the cargo, etc., thereby enabling the loaded cargo to be unloaded easily and quickly.

FIG. 16 is a drawing showing a process for determining an unloading mode in a cargo unloading automation method according to one embodiment of the present invention. In the illustrated flowchart, the method is described by dividing it into a plurality of steps, but at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into detailed steps, or performed by adding one or more steps that are not illustrated.

Referring to Fig. 16, the cargo unloading automation machine (100) can obtain an image (loaded cargo image) of cargo loaded inside a loading box (S 101). For example, the cargo unloading automation machine (100) can obtain an image of loaded cargo through a 3D depth camera, a vision sensor or Lidar, and equipment combining these.

Next, the cargo unloading automation machine (100) can recognize each loaded cargo based on edge information and depth information of each object in the loaded cargo image (S 103).

Here, the cargo unloading automation machine (100) photographs the inside of the loading box from the front of the loading box, so the depth information may mean the distance between the object included in the loaded cargo image and the cargo unloading automation machine (100). The loaded cargo image may be photographed so that such depth information can be known. In addition, the boundary information of each object in the loaded cargo image can be obtained using an object segmentation model among artificial neural network models based on deep learning. The cargo unloading automation machine (100) can assign an index to each recognized cargo. A specific method for recognizing each loaded cargo will be described later.

Next, the cargo unloading automation machine (100) can generate loaded cargo-related information including one or more of the recognized location of each cargo, the size of each cargo, the type of each cargo, and the loading pattern of the cargo in the loading box (S 105).

That is, the cargo unloading automation machine (100) can check the location of each cargo within the loading compartment, the size of each cargo, the type of each cargo, and the pattern in which the cargo within the loading compartment is loaded.

Inside the loading compartment, various types of cargo are loaded in various patterns (for example, when boxes are loaded in an aligned manner, when boxes are loaded in an aligned manner at the bottom and unaligned at the top, when sacks are loaded in an aligned manner, when sacks are loaded in an aligned manner at the bottom and unaligned at the top, when boxes are loaded in the bottom and sacks are loaded in an aligned manner at the top, etc.), and the cargo unloading automation machine (100) can analyze the loaded cargo image to generate information related to the loaded cargo.

In an exemplary embodiment, the cargo unloading automation machine (100) may use a classification model among artificial neural network models based on deep learning when generating information related to loaded cargo.

Next, the cargo unloading automation machine (100) can determine the cargo unloading order and cargo unloading mode based on the loaded cargo-related information (S 107).

That is, the cargo unloading automation machine (100) can determine in what order to unload the cargo loaded in the loading box based on the information related to the loaded cargo. In addition, the cargo unloading automation machine (100) can determine which unloading mode to use when unloading each cargo.

Fig. 17 is a flow chart showing a method for recognizing individual loaded cargo in a cargo unloading automation method according to one embodiment of the present invention. In the illustrated flow chart, the method is described by dividing it into a plurality of steps, but at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into detailed steps, or performed by adding one or more steps that are not illustrated.

Referring to FIG. 17, the cargo unloading automation machine (100) can obtain RGB images and depth images of loaded cargo using an RGB-D (Red Green Blue Depth) sensor, and obtain a point cloud of the loaded cargo from them (S 201). In one embodiment, the cargo unloading automation machine (100) can obtain RGB images and depth images of loaded cargo using two RGB-D sensors (a left sensor and a right sensor).

Next, the cargo unloading automation machine (100) can obtain boundary information of each individual cargo loaded from the RGB image (S 203). In one embodiment, the cargo unloading automation machine (100) can obtain boundary information of each individual cargo from the RGB image using a deep learning-based object segmentation model. At this time, the object segmentation model may be an artificial intelligence model learned to detect and segment objects included in an input image from the corresponding image.

The object segmentation model can be trained to detect each individual cargo and detect the boundary of each cargo even when regular cargo and irregular cargo are mixedly loaded. FIG. 19 is a diagram showing a state in which labeling is performed on learning data for learning an object segmentation model in one embodiment of the present invention. As in FIG. 18, labeling (box, icebox, sack, etc.) is performed on each cargo in an RGB image of the loaded cargo to generate learning data, and the learning data is input to the object segmentation model to train the object segmentation model.

Fig. 19 is a diagram showing the result of object segmentation using an RGB image of loaded cargo and an object segmentation model in one embodiment of the present invention. When the RGB image of loaded cargo is input into the object segmentation model, the object segmentation model can segment each object in the RGB image to obtain boundary information of each individual cargo.

Next, the cargo unloading automation machine (100) can recognize regular cargo and irregular cargo, respectively, based on the depth image of the loaded cargo, the point cloud, and the boundary information of each cargo (S 205).

FIG. 20 is a diagram showing a process for recognizing regular cargo and irregular cargo in one embodiment of the present invention. In the case of regular cargo (e.g., a box or an ice box) among the loaded cargo, the cargo unloading automation machine (100) can obtain the depth value of the corresponding cargo from inside the boundary line of the corresponding cargo to prevent the corresponding cargo from being affected by the depth values of the surrounding cargo. In addition, the cargo unloading automation machine (100) can perform polyhedral transformation on the point cloud of the corresponding cargo to obtain a plane having the largest number of internal point clouds located on the plane as the main plane of the corresponding cargo. The cargo unloading automation machine (100) can obtain the outline of the corresponding cargo based on the point cloud on the boundary line projected onto the main plane, and can estimate the position, direction, and size of the 3D oriented bounding box of the corresponding cargo by performing a rotation caliper along the obtained outline.

Meanwhile, in the case of irregular cargo (e.g., a sling, a pouch, etc.) among the loaded cargo, the cargo unloading automation machine (100) acquires a segmented point cloud by merging the point cloud acquired inside the boundary line of the cargo and the boundary line, and can estimate the location, direction, and size of the cargo based on the segmented point cloud.

When individual cargo is recognized from the loaded cargo, the cargo unloading automation machine (100) can generate information related to the loaded cargo by recognizing the loading status of multiple cargoes by integrating the recognition information of the individual cargo.

According to the disclosed embodiment, the cargo recognition speed can be increased by recognizing each individual cargo without a process of converting a separate 3D model based on an RGB image and a depth image of the loaded cargo, and regular cargo and irregular cargo can be recognized separately.

FIG. 21 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those described below.

The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a cargo unloading automation machine (100). The computing device (12) may be a device for performing a cargo unloading automation method.

A computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) may cause the computing device (12) to operate in accordance with the exemplary embodiments described above. For example, the processor (14) may execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, which, when executed by the processor (14), may be configured to cause the computing device (12) to perform operations in accordance with the exemplary embodiments.

A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data, and/or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by the processor (14). In one embodiment, the computer-readable storage medium (16) may be a memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, any other form of storage medium that can be accessed by the computing device (12) and capable of storing desired information, or a suitable combination thereof.

A communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and computer-readable storage media (16).

The computing device (12) may also include one or more input/output interfaces (22) that provide interfaces for one or more input/output devices (24) and one or more network communication interfaces (26). The input/output interfaces (22) and the network communication interfaces (26) are coupled to the communication bus (18). The input/output devices (24) may be coupled to other components of the computing device (12) via the input/output interfaces (22). Exemplary input/output devices (24) may include input devices such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or a touchscreen), a voice or sound input device, various types of sensor devices and/or photographing devices, and/or output devices such as a display device, a printer, speakers, and/or a network card. The exemplary input/output devices (24) may be included within the computing device (12) as a component that constitutes the computing device (12), or may be coupled to the computing device (12) as a separate device distinct from the computing device (12).

The above detailed description is illustrative of the present invention. In addition, the above contents illustrate and explain the preferred embodiment of the present invention, and the present invention can be used in various other combinations, changes, and environments. That is, changes or modifications are possible within the scope of the inventive concept disclosed in this specification, the scope equivalent to the written disclosure, and/or the scope of technology or knowledge in the art. The written embodiment describes the best state for implementing the technical idea of the present invention, and various changes required for specific application fields and uses of the present invention are also possible. Therefore, the above detailed description of the invention is not intended to limit the present invention to the disclosed embodiments. In addition, the appended claims should be interpreted to include other embodiments.

Claims (5)

one or more processors, and A method performed in a cargo unloading automation machine including a memory storing one or more programs executed by said one or more processors, A step of acquiring RGB images and depth images of loaded cargo using an RGB-D (Red Green Blue Depth) sensor; A step of obtaining a point cloud for the loaded cargoes based on at least one of the RGB image and the depth image; A step of obtaining boundary information of individual cargo from the above RGB image; and A method for automatically unloading cargo, comprising a step of recognizing regular cargo and irregular cargo among the loaded cargo based on the depth image, the point cloud, and the boundary information. In claim 1, The step of obtaining the above boundary information is: Using a deep learning-based object segmentation model, boundary information of individual cargo is obtained from the RGB image. The above object segmentation model is a method for automatically unloading cargo, wherein the model is trained to detect individual cargo in a state where regular cargo and irregular cargo are mixedly loaded and to detect the boundary of each cargo. In claim 1, The above recognition step is, in the case of regular cargo among the loaded cargoes, A step of performing a polyhedral transformation on the point cloud of the cargo to obtain the principal plane of the cargo; A step of obtaining an outline of the cargo based on a point cloud on a boundary line projected onto the main plane; and A method for automatically unloading cargo, comprising the step of performing a rotation caliper along the obtained outline to estimate the position, orientation, and size of a three-dimensional oriented bounding box of the cargo. In claim 1, The above recognition step is, in the case of irregular cargo among the loaded cargo, A step of obtaining a segmented point cloud by merging the point cloud obtained inside the boundary line of the cargo and the boundary line; and A method for automating cargo unloading, comprising the step of estimating the location, direction, and size of the cargo based on the segmented point cloud. One or more processors; memory; and Contains one or more programs, The one or more programs are stored in the memory and configured to be executed by the one or more processors, One or more of the above programs, Command to acquire RGB images and depth images of loaded cargo using an RGB-D (Red Green Blue Depth) sensor; A command to obtain a point cloud for the loaded cargoes based on at least one of the RGB image and the depth image; A command for obtaining boundary information of individual cargo from the above RGB image; and A computing device including a command for recognizing regular cargo and irregular cargo among the loaded cargoes based on the depth image, the point cloud, and the boundary information.

Images