CN113524172B - Robot, article grabbing method thereof and computer-readable storage medium - Google Patents

Abstract

The application relates to the technical field of robot control, and discloses a robot, an article grabbing method thereof and a computer readable storage medium. The method comprises the following steps: acquiring a first image of a target area; identifying the first image to determine a first location and a first type of the first target item in the first image; and controlling a mechanical arm of the robot to grab the first target object according to the first position and the first type of the first target object and place the first target object in a sub-area corresponding to the first type of the first target object in the target area. By the aid of the mode, sorting efficiency can be improved, labor cost is reduced, and user experience is improved.

Description

Robot, method for grabbing objects thereof, and computer-readable storage medium

technical field

The present application relates to the technical field of robot control, in particular to a robot, its object grasping method, and a computer-readable storage medium.

Background technique

At present, the application of robots is more and more extensive, filling all aspects of our lives. But it is rarely used in daily life. For example, organizing items is a relatively common problem in life. Under the heavy study and work pressure in modern society, many people are unwilling or even have no time to organize their belongings, which greatly affects work efficiency. And as the aging population becomes more and more serious, many elderly people live alone, and it is not easy for them to organize their belongings.

Contents of the invention

The main technical problem to be solved by this application is to provide a robot and its item grabbing method, and a computer-readable storage medium, which can improve the efficiency of sorting and sorting, reduce labor costs, and improve user experience.

In order to solve the above problems, a technical solution adopted by this application is to provide a method for grabbing objects by a robot. The method includes: acquiring a first image of the target area; The first position and the first type of a target item; according to the first position and the first type of the first target item, the mechanical arm of the robot is controlled to grab the first target item and place it in the target area with the first target item in the subregion corresponding to the first type.

Wherein, identifying the first image to determine the first position and the first type of the first target item in the first image includes: comparing the first image with the background image to obtain the first target item in the first image where the background image is collected when there is no first target item in the target area; the first image is input to the pre-trained first learning model to obtain the first type corresponding to the first target item.

Wherein, comparing the first image with the background image to obtain the first position of the first target item in the first image includes: comparing the first image with the background image to obtain the position of the first target item in the first image First contour information: inputting the first image to the pre-trained first learning model to obtain the first type corresponding to the first target item, including: inputting the image formed based on the first contour information to the first learning model to obtain the contour The first type that the message corresponds to.

Wherein, inputting the image formed based on the first contour information to the first learning model to obtain the first type corresponding to the contour information includes: converting the image formed based on the first contour information to HSV format to obtain the first HSV data; obtaining The first tone histogram in the first HSV data; input the first tone histogram into the first learning model to obtain the first type corresponding to the contour information.

Wherein, according to the first position and the first type of the first target item, controlling the mechanical arm of the robot to grab the first target item includes: generating a grasping trajectory according to the first position and the first type; controlling the mechanical arm according to the grasping trajectory Move to the first position to grab the first target item.

Wherein, controlling the movement of the mechanical arm to the first position according to the grasping trajectory to grasp the first target item includes: controlling the movement of the mechanical arm to the first position according to the grasping trajectory; obtaining the first position collected by the camera assembly at the end of the mechanical arm Two images: identify the second image to determine the second position and the second type of the second target item in the second image; if the first type is the same as the second type, then determine the first target item and the second target item For the same item, control the robotic arm to grab the second target item.

Wherein, identifying the second image to determine the second position and the second type of the second target item in the second image includes: identifying the second target item in the second image to obtain the second target item in the second image the second position of the second image; inputting the second image to the pre-trained second learning model to obtain the second type of the second target item in the second image.

Wherein, before the second image is input to the pre-trained second learning model, before obtaining the second type of the second target item in the second image, it includes: obtaining the second contour information of the second target item based on the second position; The image formed by the second contour information is converted into HSV format to obtain the second HSV data; obtain the second tone histogram in the second HSV data; input the second image to the pre-trained second learning model to obtain the second image The second type of the second target item includes: inputting the second tone histogram into the pre-trained second learning model to obtain the second type corresponding to the second target item in the second image.

Wherein, generating the grasping track according to the first position and the first type includes: determining the sub-area corresponding to the first target item according to the first type; determining the first distance between the first target item and the sub-area according to the first position; A grasp trajectory is generated based on the first distance.

Wherein, when there are multiple first target items in the target area, the method further includes: identifying the first image to determine the first position and the first type of each first target item in the first image; according to the first Determining the first distance between the first target item and the sub-area at a position includes: respectively determining the corresponding sub-area according to the first type of each first target item; determining and The first distance between the corresponding sub-regions; generating the grasping trajectory based on the first distance, including: sorting the first distances of the first target items of the same first type from small to large, so as to obtain each first type The corresponding grasping sequence; the grasping trajectory is generated based on the grasping sequence.

Wherein, generating the grasping trajectory based on the grasping sequence includes: using the RRT algorithm to generate the grasping trajectory based on the grasping sequence and the first position of the first target item of the same first type.

Wherein, the robot includes at least a first mechanical arm and a second mechanical arm; when there are multiple first target items in the target area, the method further includes: determining whether there are at least two first types of the multiple first target items; If so, determine the first type of grasping and the first target item corresponding to the first type for the first mechanical arm and the second mechanical arm respectively; according to the first position and the first type of the first target item, control the mechanical The arm grabs the first target item and places it in the sub-area corresponding to the first type of the first target item in the target area, including: controlling the first mechanical arm and the second mechanical arm to grab the corresponding first target respectively item, and placed in the sub-area corresponding to the first type of the first target item.

In order to solve the above problems, another technical solution adopted by the present application is to provide a robot, the robot includes: a robot body; a mechanical arm arranged on the robot body; a camera assembly arranged on the robot body and/or the robot arm Capture an image of the target area;

The memory is arranged on the main body of the robot to store program data; the processor is arranged on the main body of the robot and is connected to the mechanical arm, the camera assembly and the memory, and is used to execute the program data to realize the method provided by the above technical solution.

In order to solve the above problems, another technical solution adopted by the present application is to provide a computer-readable storage medium, in which program data is stored, and when the program data is executed by the processor, it is used to realize the above-mentioned The method provided by the technical solution.

The beneficial effects of the present application are: different from the situation of the prior art, the present application provides a robot, its object grasping method, and a computer-readable storage medium. This method uses visual images to locate and identify the target items in the target area, and controls the robotic arm of the robot to grab the target items and place them in the corresponding sub-area, which can quickly classify and place the target items , compared with manual sorting, it can improve the efficiency of sorting, reduce labor costs, and improve user experience.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort. in:

Fig. 1 is a schematic flow chart of an embodiment of a method for grabbing objects by a robot provided by the present application;

Fig. 2 is a schematic diagram of an embodiment of the first image of the target area provided by the present application;

Fig. 3 is a schematic diagram of another embodiment of the first image of the target area provided by the present application;

Fig. 4 is a schematic flow chart of another embodiment of the object grabbing method of the robot provided by the present application;

FIG. 5 is a schematic flow diagram of an embodiment of step 43 provided by the present application;

Fig. 6 is a schematic flow chart of another embodiment of the object grabbing method of the robot provided by the present application;

FIG. 7 is a schematic flowchart of an embodiment of step 63 provided by the present application;

FIG. 8 is a schematic flowchart of an embodiment of step 64 provided by the present application;

FIG. 9 is a schematic flowchart of an embodiment of step 643 provided by the present application;

FIG. 10 is a schematic flowchart of an embodiment before step 6432 provided by the present application;

Fig. 11 is a schematic flowchart of another embodiment of the method for grabbing objects by a robot provided in the present application;

Fig. 12 is a schematic diagram of another embodiment of the first image of the target area provided by the present application;

Fig. 13 is a schematic flowchart of another embodiment of the method for grabbing objects by a robot provided in the present application;

Fig. 14 is a schematic structural diagram of an embodiment of a robot provided by the present application;

Fig. 15 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided by the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific embodiments described here are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only some structures related to the present application are shown in the drawings but not all structures. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

The terms "first", "second", etc. in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "include" and "have", as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally further includes For other steps or units inherent in these processes, methods, products or apparatuses.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrences of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described herein can be combined with other embodiments.

Referring to FIG. 1 , FIG. 1 is a schematic flowchart of an embodiment of a method for grabbing objects by a robot provided in the present application. The method includes:

Step 11: Acquire the first image of the target area.

In some embodiments, acquiring the first image of the target area may be acquired using an image acquisition component. For example, the image acquisition component is arranged on the head of the robot, which can acquire images of the entire target area. In another example, the image acquisition component is arranged at the end of the robot's mechanical arm, and by controlling the mechanical arm, the image acquisition component at the end of the mechanical arm can acquire images of the entire target area. Wherein, the image collection component may be a camera capable of collecting depth-of-field images.

In some embodiments, the target area may be a tabletop, a cleaning or sorting surface, and the like.

Step 12: Recognize the first image to determine a first position and a first type of the first target item in the first image.

In some embodiments, a target detection algorithm may be used to identify the first image, and then determine the first position and the first type of the first target item in the first image.

It can be understood that if there are multiple first target items in the first image, each first target item will be identified to obtain the first position and the first type corresponding to each first target item.

In other embodiments, the image segmentation algorithm can be used to identify the first image first, and then determine the first position of the first target item in the first image, perform image segmentation based on the first position, and then input the segmented image to A learning model is used to identify the first type thereof.

Step 13: According to the first position and the first type of the first target item, control the mechanical arm of the robot to grab the first target item and place it in the sub-area corresponding to the first type of the first target item in the target area .

In some embodiments, according to the first position and the first type of the first target item, a grasping trajectory can be set for the mechanical arm of the robot, so that the mechanical arm grasps the first target item at the first position according to the grasping trajectory.

Among them, the first position is used to determine the grabbing start position of the robot arm, and the first type is used to determine the grabbing end position of the robot arm.

In an application scenario, it will be described with reference to Figure 2:

Fig. 2 is a schematic diagram of an embodiment of a first image of a target area. In FIG. 2 , the first position and the first type of the first target item c are identified through the above steps. Wherein, the target area includes sub-area A and sub-area B. If the first type of the first target item c corresponds to the sub-area B, the robotic arm is controlled to grab the first target item c and place it in the sub-area B.

In another application scenario, it will be described in conjunction with Figure 3:

Fig. 3 is a schematic diagram of another embodiment of the first image of the target area. In FIG. 3 , the first position and first type of the first target item c and the first position and first type of the first target item d are identified through the above steps. Wherein, the target area includes sub-area A and sub-area B. The first type of the first target item c corresponds to sub-area B, and the first type of the first target item d corresponds to sub-area a, then the control robot first grabs the first target item c and places it in sub-area B, and then The robotic arm is controlled to grab the first target item d and place it in the sub-area A.

In this embodiment, the visual image is used to locate and identify the target item in the target area, and the robot's mechanical arm is controlled to grab the target item and place it in the corresponding sub-area, so that the target item can be quickly identified. Compared with manual sorting and sorting, sorting and placing items can improve the efficiency of sorting and sorting, reduce labor costs, and improve user experience.

Referring to FIG. 4 , FIG. 4 is a schematic flowchart of another embodiment of a method for grabbing objects by a robot provided in the present application. The method includes:

Step 41: Acquire the first image of the target area.

Step 42: Comparing the first image with the background image to obtain a first position of the first target item in the first image.

Wherein, the background image is collected when there is no first target item in the target area.

In some embodiments, the head camera of the robot is used to collect images of the target area to obtain the first image. Since the target area and the head camera are fixed, it is possible to first collect an image of the target area without the first target item as the background image, and then use the method of background subtraction to obtain the first target object in the target area, thus A first position of the first target object in the first image is obtained.

In other embodiments, the first image is compared with the background image to obtain the first position of the first target item in the first image, and then the first contour information of the first target item is extracted according to the first position to obtain The center point of the first contour information is taken as the center point of the object. And use a rectangular frame to surround the first target item, that is, the first target item can be represented within the rectangular frame. Record the row and column of the upper left pixel of the rectangular frame, as well as the length and width of the rectangular frame, that is, the rectangular frame can be expressed as (x, y, w, h), where x and y represent the upper left pixel point of the rectangular frame Coordinates, w represents the width of the rectangle, h represents the length of the rectangle. The rectangular frame can represent the first outline information of the first target item.

Step 43: Input the first image into the pre-trained first learning model to obtain the first type corresponding to the first target item.

The first learning model may be trained based on a deep learning model, such as a convolutional neural network model. such as support vector machines.

In some embodiments, an image corresponding to the first target item may be obtained based on the first contour information, and then the image formed based on the first contour information is input into the first learning model to obtain the first type corresponding to the contour information. In this way, using the image segmented based on the contour information for type recognition can reduce the amount of calculation of the learning model, so that the learning model does not need to perform image segmentation.

In other embodiments, referring to FIG. 5, step 43 may also be the following process:

Step 431: Convert the image formed based on the first contour information to HSV format to obtain first HSV data.

HSV is a color model, H means Hue (hue, hue), S means Saturation (saturation, color purity), V means Value (brightness).

It can be understood that the first image captured by the camera component is formed based on RGB. Therefore, processing of three channels is required when processing an RGB image, which will increase the amount of calculation.

Therefore, this application converts the RGB image into HSV format to make the image closer to people's perceptual experience of color. It is very intuitive to express the hue, vividness and lightness of the color, which is convenient for color comparison. In the HSV color space, it is easier to track objects of a certain color than RGB.

Among them, the method of converting RGB format to HSV format is:

V is the maximum value among R, G, and B.

S is the difference between the maximum and minimum values of R, G, and B divided by the maximum value.

H satisfies the following conditions: if R is the maximum value, then H=(G-B)/(max-min)*60;

G is the maximum value, H=120+(B-R)/(max-min)*60;

B is the maximum value, H=240+(R-G)/(max-min)*60.

If H<0, then H=H+360.

Step 432: Obtain the first tone histogram in the first HSV data.

In this embodiment, because type identification is required, the H channel among the three HSV channels can be extracted, that is, the hue channel. A hue histogram is formed based on the data of the hue channel to represent the first target item.

When calculating the tone histogram, the value of H needs to be evenly divided into several small intervals, and each small interval becomes an entry of the histogram. Then, the hue histogram can be obtained by calculating the number of pixels whose H value falls within each small interval. In the embodiment of the present application, H can be evenly divided into 32 cell intervals with a value of 8, corresponding to a range of Hue values from 0 to 255.

Step 433: Input the first tone histogram into the first learning model to obtain the first type corresponding to the contour information.

It can be understood that when using the tone histogram to identify the first target item type, the first learning model is also trained using the tone histogram as a training sample.

The type is determined by means of a single channel of the tone histogram, which can reduce the calculation amount of the learning model and improve the efficiency of type determination.

Step 44: According to the first position and the first type of the first target item, control the mechanical arm of the robot to grab the first target item and place it in the sub-area corresponding to the first type of the first target item in the target area .

In this embodiment, the visual image is used to locate the target item in the target area, the learning model is used to identify the type of the target item, and the mechanical arm of the robot is controlled to grab the target item and place it in the corresponding sub-section. In the area, the target items can be quickly sorted and placed. Compared with manual sorting, it can improve the efficiency of sorting, reduce labor costs, and improve user experience.

Referring to FIG. 6 , FIG. 6 is a schematic flowchart of another embodiment of a method for grabbing objects by a robot provided in the present application. The method includes:

Step 61: Acquire the first image of the target area.

Step 62: Recognize the first image to determine a first position and a first type of the first target item in the first image.

Steps 61-62 have the same or similar technical solutions as those in the above-mentioned embodiments, which will not be repeated here.

Step 63: Generate a grasping track according to the first position and the first type.

In this embodiment, the center point of the first target item can be determined based on the first position, and the sub-area can be determined according to the first type, then the grasping trajectory can be determined according to the position information of the center point and the position information of the sub-area.

In some embodiments, referring to FIG. 7, step 63 may be the following process:

Step 631: Determine the sub-area corresponding to the first target item according to the first type.

Combined with Figure 2 for illustration:

After the first type of the first target item c is determined, it can be determined that the first type corresponds to the sub-area B according to the correspondence between the types and the sub-areas.

Step 632: Determine a first distance between the first target item and the sub-area according to the first position.

A center point of the first target item is determined based on the first position, and a first distance between the first target item and the sub-area is determined by using position information of the center point.

Step 633: Generate a grasping track based on the first distance.

When there is only one first target item in the target area, the grasping trajectory can be set as a linear trajectory to improve the grasping efficiency.

When there are multiple first target items in the target area, the grasping trajectory may be set according to the first position of each first target item.

Step 64: Control the mechanical arm to move to the first position according to the grasping trajectory, so as to grasp the first target item.

In some embodiments, if the robot is a rigid robot, the robotic arm of the robot has high positioning accuracy, and can directly grasp the first target item according to the first position of the first target item.

In some embodiments, if the robot is a non-rigid robot, the positioning accuracy of the robotic arm of the robot is poor, and the object grasping can be performed referring to FIG. 8 . Specifically, step 64 may be the following process:

Step 641: Control the mechanical arm to move to the first position according to the grabbing track.

The robot used to execute the process of FIG. 8 includes a plurality of camera components, wherein a camera component is arranged on the main body of the robot to collect the first image of the target area. A camera assembly is arranged at the end of the mechanical arm, and is used to collect a second image of the object by using the camera when the mechanical arm is controlled to grasp the object.

Therefore, when the mechanical arm moves to the first position, because the robot is a non-rigid robot, the mechanical arm does not actually move to the first position accurately, and there is a large error between the first position and the first position. The robotic arm is not very good at grabbing items. At this time, the camera assembly at the end of the robotic arm can be used to collect images of the target area. Since the robotic arm is near the first target item, the image can capture the first target item. Then the robot can execute step 642 to process the image collected by the camera assembly on the end of the robot arm.

Step 642: Obtain a second image captured by the camera assembly at the end of the robotic arm.

Step 643: Recognize the second image to determine a second position and a second type of the second target item in the second image.

In some embodiments, step 643 may be processed in the manner of identifying the first image in any of the foregoing embodiments.

In some embodiments, referring to FIG. 9, step 643 may be the following process:

Step 6431: Identify the second target item in the second image, and obtain the second position of the second target item in the second image.

Step 6432: Input the second image into the pre-trained second learning model to obtain the second type of the second target item in the second image.

In some embodiments, referring to FIG. 10, the following process may be performed before step 6432:

Step 101: Obtain second contour information of a second target item based on a second position.

Step 102: Convert the image formed based on the second contour information to HSV format to obtain second HSV data.

Step 103: Obtain a second tone histogram in the second HSV data.

After obtaining the second tone histogram, step 6432 may be to input the second tone histogram into the pre-trained second learning model to obtain the second type corresponding to the second target item in the second image.

It can be understood that due to the mechanical arm, step 642-step 643 is a cyclic process. After the first second image fails to meet the subsequent conditions, the mechanical arm will be controlled to continue to move and continue to collect the second image. Execute step 642-step 643.

The second learning model here and the first learning model in the above embodiment are trained using different training samples. The first learning model is trained using the first image collected by the first camera or based on the first tone histogram. The second learning model is trained using the second image collected by the second camera or based on the second tone histogram.

Step 644: If the first type is the same as the second type, determine that the first target item and the second target item are the same item, and control the mechanical arm to grab the second target item.

In some embodiments, when it is determined that the first type is the same as the second type, the Camshift algorithm may be used to converge the area to be tracked during the movement of the robotic arm. Wherein, the area to be tracked is used to represent the size of the second target item in the second image. It can be understood that when the area to be tracked satisfies the conditions, it means that the robotic arm has moved to the optimal grasping position, and grasping can be performed.

Through the above method, the position of the first target item can be tracked in real time. Make the arm always move towards the center of the first target item. When it is detected that the size of the first target item in the second image reaches a preset value, the grabbing operation can be performed. In this way, the combination of visual images solves the problem of low repeat positioning accuracy of non-rigid robots, and can also accurately grasp target items with the help of visual images.

Step 65: Place in the sub-area corresponding to the first type of the first target item in the target area.

In this embodiment, the visual image is used to locate the target item in the target area, and the learning model is used to identify the type of the target item, and the non-rigid robot is re-used to image recognition to realize the non-rigid robot. Accurate grabbing of target items can quickly classify and place target items. Compared with manual sorting, it can improve sorting efficiency, reduce labor costs, and improve user experience.

Referring to FIG. 11 , FIG. 11 is a schematic flow chart of another embodiment of the object grabbing method of the robot provided in the present application. The method includes:

Step 111: Acquire a first image of the target area.

This embodiment is applied to a scene where there are multiple first target items in the target area.

Step 112: Recognize the first image to determine a first position and a first type of each first target item in the first image.

In this embodiment, step 112 can identify a plurality of first target items in the manner of identifying a single first target item in any of the above-mentioned embodiments, then the first position and the first position of each first target item can be obtained respectively. first type.

Step 113: Determine corresponding sub-regions according to the first type of each first target item.

Step 114: Determine the first distance to the corresponding sub-area according to the first position of each first target item.

Step 115: Sorting the first distances of the first target items of the same first type in descending order, so as to obtain the grabbing order corresponding to each first type.

Step 113-step 115 is described in conjunction with Fig. 12:

In FIG. 12, subregions A, B and C exist. The detected first target items include d, e, f, g, h and i. By identifying the first type of d, e, f, g, h and i, it is determined that d and g correspond to subregion B, e and f correspond to subregion A, and h and i correspond to subregion C. Then a grab sequence can be generated for d and g, a grab sequence can be generated for e and f, and a grab sequence can be generated for h and i.

Among them, in d and g, if the distance between d and sub-region B is smaller than the distance between g and sub-region B, it can be sorted from small to large based on the distance, and then the corresponding grabbing order can be to grab d first and then grab g. In other embodiments, the grabbing may be performed in a descending order, first grabbing g and then grabbing d. This grasping method can reduce the movement of the mechanical arm, reduce the loss of the mechanical arm, and improve the service life of the mechanical arm.

The capture sequence corresponding to e and f, and the capture sequence corresponding to h and i can be performed in the above manner, and will not be described here.

Step 116: Generate a grasping trajectory based on the grasping sequence.

In step 116, based on the grasping order and the first position of the first target item of the same first type, the grasping trajectory may be generated by using the RRT algorithm.

RRT (Rapidly-Exploring Random Trees) is a basic global path search algorithm, which is simple and fast.

The first image may be a depth image, and the RGB-D information of the target area may be collected through the first image. Through this information, the spatial information in the target area may be restored and represented by a spatial point cloud. The spatial point cloud can complete the construction of the configuration space of the target area, and combined with the RRT algorithm, the path that can avoid obstacles can be planned. The path is the above-mentioned grabbing track.

Step 117: Control the robot arm to move to the first position according to the grabbing trajectory, so as to grab the first target item and place it in the sub-area corresponding to the first type of the first target item in the target area.

In step 117, if the robot is a non-rigid robot, it can grasp the object in the manner of the above-mentioned non-rigid robot.

In this embodiment, visual images are used to locate multiple target items in the target area, the learning model is used to identify the type of target items, and the grasping trajectory is reasonably planned to quickly classify and place the target items. Compared with manual sorting and sorting, it can improve the efficiency of sorting and sorting, reduce labor costs, and improve user experience.

Referring to FIG. 13 , FIG. 13 is a schematic flow chart of another embodiment of the object grabbing method of the robot provided in the present application. The method includes:

Step 131: Acquire a first image of the target area.

Step 132: Recognize the first image to determine a first position and a first type of each first target item in the first image.

Step 131-step 132 have the same or similar technical solutions as those of the above-mentioned embodiments, which will not be repeated here.

In this embodiment, the robot at least includes a first robotic arm and a second robotic arm.

Step 133: When there are multiple first target items in the target area, determine whether there are at least two first types of the multiple first target items.

When it is determined that the types of the multiple first target items are not unique, step 134 may be performed.

If the multiple first target items are of the same first type, only one of the first mechanical arm and the second mechanical arm may be controlled to grab the item.

Step 134: Determine the first type to be grasped and the first target item corresponding to the first type for the first robot arm and the second robot arm respectively.

It will be described in conjunction with FIG. 12 . In FIG. 12 , d and g correspond to subregion B, e and f correspond to subregion A, and h and i correspond to subregion C. If the first robotic arm is close to subarea B and the second robotic arm is close to subareas A and C, it can be determined that the first robotic arm grabs d and g, and the second robotic arm grabs e and f and h and i.

Step 135: Control the first robot arm and the second robot arm to grab the corresponding first target item respectively, and place them in the sub-area corresponding to the first type of the first target item.

In step 135, the grasping trajectories of the first robotic arm and the second robotic arm may be generated in the manner of any of the above-mentioned embodiments. If the robot is a non-rigid robot, it can grasp objects in the manner of the above-mentioned non-rigid robot. That is, a camera is respectively arranged at the ends of the first mechanical arm and the second mechanical arm.

In other embodiments, a first image of the target area is acquired, and the first image is identified to determine a first position and a first type of each first target item in the first image. Wherein, if the first target item in the sub-area does not correspond to the sub-area, it also needs to be grabbed to place it in the corresponding sub-area.

When determining the distance of each first target object, if any distance value is smaller than the set threshold, collision detection is performed. If there is a collision, the robotic arm is controlled to move it to a temporary position without objects on the target area. If there is no collision, it will be grabbed in order according to the sorting.

In an application scenario, the target area is a desktop. Such as the desktop of the elderly, the desktop of children, the desktop of office workers. Then the robot can be controlled to organize the desktop. Take the desktop of the elderly as an example: many elderly people live alone, and it is not easy for them to organize the desktop. For example, the desktop sub-area of the elderly can be divided into food area, daily necessities area and medicine area. The robot is arranged on one side of the desktop, and the camera assembly on the main body of the robot can collect panoramic images on the desktop. Through the method of any of the above-mentioned embodiments, if the items on the desktop are identified, two items belonging to food, two items belonging to daily necessities, and two items belonging to medicines are identified, and these items are respectively determined. location, as well as the location of the food area, daily necessities area and medicine area. Grab trajectories are then generated for these items. The robotic arm that controls the robot grabs the item according to the grabbing track and places it in the corresponding area.

In some embodiments, when applied to daily life scenes such as desktops, the robot adopts a non-rigid robot. Compared with a rigid robot, the joints of the mechanical arm of a non-rigid robot are composed of elastic materials. During the running/non-running phase, All have high security.

In this embodiment, the dual-arm robot can simultaneously grasp and classify different types of items, and has a larger grasping range, which can improve the efficiency of sorting and sorting, reduce labor costs, and improve user experience.

Referring to FIG. 14 , the robot 140 includes a robot body 141 , a robot arm 142 , a camera assembly 143 , a memory 144 and a processor 145 . Wherein, the mechanical arm 142 is arranged on the robot main body 141; the camera assembly 143 is arranged on the robot main body 141 and/or the mechanical arm 142, and is used to collect images of the target area; the memory 144 is arranged on the robot main body 141, and is used to store program data; the processor 145 is arranged on the robot main body 141, and connects the mechanical arm 142, the camera assembly 143 and the memory 144, and is used to execute the program data, so as to realize the following method:

Acquiring a first image of the target area; identifying the first image to determine a first position and a first type of a first target item in the first image; controlling the robot according to the first position and first type of the first target item The robotic arm grabs the first target item and places it in a sub-area corresponding to the first type of the first target item in the target area.

It can be understood that the processor 145 in this embodiment may also implement the method in any of the foregoing embodiments, which will not be repeated here.

In some embodiments, the camera assembly 143 includes a first camera (not shown in the figure) and a second camera (not shown in the figure), the first camera is arranged on the robot body 141, and is used to collect the first image of the target area, and can collect the global the image of the object; the second camera is set at the end of the mechanical arm 142 for collecting the image data of the first target item. It can be understood that the accuracy of the target dynamic object captured by the second camera is higher than that of the first camera. Wherein, the number of the second cameras corresponds to the number of the mechanical arms 142 , that is, a second camera is arranged at the end of each mechanical arm 142 .

In some embodiments, the first camera is installed on the head of the robot body 141 to observe the overall situation of the target area, and may be a depth camera, and the second camera is installed at the end of the robot arm, and may be an RGB camera. First, system calibration is required. The calibration of the second camera can be directly obtained from the product description, so only the first camera needs to be calibrated. This type of calibration belongs to eye-to-hand in hand-eye calibration, that is, the first camera is relatively fixed to the robot main body 141 and separated from the mechanical arm. In the eye-to-hand problem, the quantity to be sought is the fixed transformation matrix ^base T _camera from the first camera to the base coordinate system of the robot body 141, which is a 4*4 matrix, representing the camera of the first camera The conversion relationship from the coordinate system to the base coordinate system of the robot main body 141 . The checkerboard calibration plate can be fixed on the first mechanical arm or the second mechanical arm, the first mechanical arm or the second mechanical arm drives the calibration plate to move to different positions and angles under the first camera, and the first camera takes multiple shots of the calibration plate Identify the corner points to calculate.

When the first robot arm or the second robot arm drives the calibration plate to move any two poses, the poses are used to obtain a fixed transformation matrix from the first camera to the base coordinate system of the robot body 141 .

Referring to FIG. 15 , program data 151 is stored in a computer-readable storage medium 150, and when the program data 151 is executed by a processor, it is used to implement the following method:

Acquiring a first image of the target area; identifying the first image to determine a first position and a first type of a first target item in the first image; controlling the robot according to the first position and first type of the first target item The robotic arm grabs the first target item and places it in a sub-area corresponding to the first type of the first target item in the target area.

It can be understood that the computer-readable storage medium 150 in this embodiment can also implement the method in any of the above-mentioned embodiments, which will not be repeated here.

In some embodiments, the first learning model and/or the second learning model mentioned in any of the above embodiments may adopt a SVM (Support Vector Machine, support vector machine) model. The basic idea of SVM is to solve the separation hyperplane that can correctly divide the training data set and has the largest geometric interval, and its advantages over other classifiers such as KNN (K-NearestNeighbor, k-nearest neighbor classification algorithm) are as follows: After the SVM training is completed, Most of the training samples do not need to be kept, and the final model is only related to the support vectors.

In some embodiments, the SVM method in the machine learning class in the image processing library OpenCV is used to implement. The parameters that need to be set include the SVM type, the SVM kernel type, the parameters of the kernel, the number of model iterations and the precision, and the specific values are in order. : C_SVC, indicating that the SVM type is a C-class support vector classification machine, which allows incomplete classification with the outlier penalty factor C; the linear kernel SVM::LINEAR, which has the fastest calculation speed; the kernel parameter gamma=0.1, and the optimization parameter C is set to 0; the number of iterations is 3000, and the iteration precision is 0.001.

In the several implementation manners provided in this application, it should be understood that the disclosed methods and devices may be implemented in other ways. For example, the device implementation described above is only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated units in the above other embodiments are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc and other media that can store program codes.

The above is only the implementation of the application, and does not limit the patent scope of the application. Any equivalent structure or equivalent process conversion made by using the specification and drawings of the application, or directly or indirectly used in other related technologies fields, are all included in the scope of patent protection of this application in the same way.

Claims (8)

1. A method of article grasping by a robot, the method comprising:

acquiring a first image of a target area;

comparing the first image with a background image to obtain first contour information of a first target object in the first image; wherein the background image is acquired when the target area is free of a first target item;

performing HSV format conversion on the image formed based on the first contour information to obtain first HSV data;

acquiring a first color tone histogram in the first HSV data;

inputting the first color tone histogram into a first learning model to obtain a first type corresponding to the first contour information;

when a plurality of first target items exist in the target area, identifying the first image to determine a first position and a first type of each first target item in the first image;

determining a sub-area corresponding to the first target object according to the first type;

determining a corresponding sub-area according to the first type of each first target object;

determining a first distance between each first target object and the corresponding sub-area according to the first position of the first target object;

sequencing the first distances of the first target objects of the same first type from small to large to obtain a grabbing sequence corresponding to each first type;

generating a grabbing track based on the grabbing sequence;

and controlling the mechanical arm to move to the first position according to the grabbing track so as to grab the first target object, and placing the first target object in a sub-area corresponding to the first type of the first target object in the target area.

2. The method of claim 1,

the controlling the mechanical arm to move to the first position according to the grabbing track so as to grab the first target object comprises:

controlling the mechanical arm to move to the first position according to the grabbing track;

acquiring a second image acquired by a camera assembly at the tail end of the mechanical arm;

identifying the second image to determine a second location and a second type of a second target item in the second image;

and if the first type is the same as the second type, determining that the first target object and the second target object are the same, and controlling the mechanical arm to grab the second target object.

3. The method of claim 2,

the identifying the second image to determine a second location and a second type of a second target item in the second image comprises:

identifying the second target object in the second image to obtain a second position of the second target object in the second image;

and inputting the second image into a pre-trained second learning model to obtain a second type of the second target object in the second image.

4. The method of claim 3,

before inputting the second image into a second learning model trained in advance to obtain the second type of the second target item in the second image, the method includes:

obtaining second contour information of the second target object based on the second position;

performing HSV format conversion on the image formed based on the second contour information to obtain second HSV data;

acquiring a second hue histogram in the second HSV data;

the inputting the second image into a second learning model trained in advance to obtain a second type of the second target item in the second image includes:

and inputting the second color tone histogram into a pre-trained second learning model to obtain a second type corresponding to the second target object in the second image.

5. The method of claim 1,

the generating of the grabbing track based on the grabbing sequence comprises:

generating a grabbing track by using an RRT algorithm based on the grabbing sequence and the first position of the first target object of the same first type.

6. The method of claim 1,

the robot comprises at least a first mechanical arm and a second mechanical arm;

when a plurality of first target items are present in the target area, the method further comprises:

determining whether at least two first types of the plurality of first target items are present;

if yes, respectively determining the first type of the grabbed first target object and the first target object corresponding to the first type for the first mechanical arm and the second mechanical arm;

the controlling a robot arm of the robot to grasp the first target item and place the first target item in a sub-region of the target region corresponding to the first type of the first target item according to the first position and the first type of the first target item comprises:

and controlling the first mechanical arm and the second mechanical arm to respectively grab the corresponding first target object and place the first target object in the sub-area corresponding to the first type of the first target object.

7. A robot, characterized in that the robot comprises:

a robot main body;

a robot arm provided to the robot main body;

the camera assembly is arranged on the robot main body and/or the mechanical arm and is used for acquiring images of a target area;

a memory provided in the robot main body for storing program data;

a processor disposed in the robot body and connected to the robot arm, the camera assembly and the memory for executing the program data to implement the method of any one of claims 1-6.

8. A computer-readable storage medium, in which program data are stored which, when being executed by a processor, are adapted to carry out the method of any one of claims 1-6.

Images