CN102688823B - Atomizing positioning device and method based on hand-eye atomizing mechanical arm - Google Patents

Abstract

The invention discloses an atomizing positioning device and a method based on a hand-eye atomizing mechanical arm and belongs to the technical field of machine vision and image processing. The atomizing positioning device comprises a mechanical arm, a camera and a sprayer, wherein the mechanical arm is fixed on a mechanical arm base, and the camera and the sprayer are respectively arranged at the tail end of the mechanical arm. The atomizing positioning device and the method provided by the invention can extract and position the three-dimensional information outline of a target with accurate positioning and moderate calculation amount, the accurate automatic atomization is realized, the environmental pollution is reduced, and the pesticide utilization ratio is improved.

Description

A spray positioning device and method based on a hand-eye spray robot arm

technical field

The invention belongs to the technical field of machine vision and image processing, and in particular relates to a spray positioning device and method based on a hand-eye spray mechanical arm.

Background technique

In the process of agricultural production, in order to prevent pests and diseases, it is often necessary to spray pesticides many times. The process of applying pesticides will cause certain harm to the environment and the health of operators, especially in greenhouse conditions, where the space is relatively closed and the frequency of pesticide application is high, so the hazards are more obvious. Through the development of an automated precision pesticide application system, the liquid medicine can be sprayed directly on the surface of the crops, avoiding waste of the liquid medicine, improving the use efficiency of the medicine liquid, reducing environmental pollution, ensuring the health of workers, and reducing labor intensity. The development of automatic pesticide application system has important practical significance and social value.

In the automated pesticide application system, the main problems currently are:

1. The positioning of spraying is not accurate enough, and the waste of liquid medicine is serious. According to the 2010 International Conference on Plant Protection Machinery and Pesticide Application Technology, the average utilization rate of pesticides in my country is extremely low, only about 20%. Most of the pesticides have not been fully and effectively used. The root cause is that the methods and methods of pesticide application are not scientific and reasonable. In addition, extensive spraying is often used in the use of pesticides, and the technology and conditions for precise pesticide application are lacking.

2. The spraying of pesticides is not uniform enough, and the residual pesticide solution on the surface of crops exceeds the standard, especially for crops produced in greenhouses. The atomization effect and spray operation mode of the chemical liquid spray have a great influence on the uniformity of the spray. According to the data, the use of electrostatic spraying can form tiny droplet particles with good adhesion, which is beneficial to reduce re-spraying and missed spraying, and improve the uniformity of spraying. Anti-drift and other technologies can also improve the spray effect to a certain extent, but fundamentally speaking, the accuracy of spray positioning will directly affect the spray quality.

3. The use adaptability of spray agricultural machinery is limited. For example, sprayers used in foreign orchards use ultrasonic spray positioning. This method requires fruit trees to be cultivated at a specific distance and arrangement. As long as there is an object within the ultrasonic detection range, it will be sprayed. It works the same way regardless of the crop form when spraying. Therefore, it is difficult to work effectively when the environment and crops change.

4. The robot positioning detection effect for automatic and precise spraying is not ideal, and the real-time performance is poor. For example, for a robot that uses visual detection technology to spray a specific area of pests and diseases, its positioning detection algorithm is often complicated and requires a certain amount of computing time. At the same time, there is also a certain error rate in the detection of target crops that need to be sprayed.

5. The two-dimensional positioning system is basically used for the spraying and application of crops. During the working process, it is generally first to detect and obtain the two-dimensional information of the spray object through a specific sensor or camera, to move the nozzle to the specified position or to control the opening and closing of multiple nozzles, and the distance between the nozzle and the target crop It is often set in advance and is not adjusted during the work process. Therefore, when there is a certain difference in the shape and size of the crops, different spray effects will be caused.

6. The cost performance of the pesticide spraying automation system is also a problem that restricts its wide application. However, with the continuous development of the number and technology of facility agriculture, and the rising labor costs associated with the arrival of an aging society, automated spraying operations will have broad application prospects.

To sum up, the most important problem at present is to solve the problem of spray target identification and positioning, and to develop a spray positioning system with good adaptability, accurate positioning, good real-time performance and high cost performance.

At present, the methods for obtaining three-dimensional information in object space mainly include laser, ultrasonic, radar, infrared and binocular vision. When the first four work, the distance information is usually calculated through the reflected wave time or phase difference. The binocular vision mainly uses the triangular ranging principle to achieve positioning information acquisition through left and right image matching. The advantage of the binocular vision positioning system is that it has a wide range of applications, and it can directly realize the recognition and positioning of the target through a certain algorithm; its disadvantage is that the recognition and positioning algorithms are often complicated, and the real-time performance and robustness are poor. It is more difficult to detect objects with irregular shapes, complex environments, and poor lighting conditions.

Contents of the invention

Aiming at the disadvantages mentioned in the above-mentioned background technology that the method of extracting and calculating the target feature size is complex and easily affected by factors such as target shape and illumination, the present invention proposes a spray positioning device and method based on a hand-eye spray robot arm.

The technical solution of the present invention is a spray positioning device based on a hand-eye spray robot arm, which is characterized in that the device includes a robot arm base, a robot arm, a camera and a nozzle;

The mechanical arm is fixed on the base of the mechanical arm; the camera and the spray head are respectively installed at the end of the mechanical arm.

The manipulator is a four-degree-of-freedom spray manipulator.

A spray positioning method based on a hand-eye spray mechanical arm, which locates a target through a mechanical arm and a camera, is characterized in that the method includes the following steps:

Step 1: Determine the conversion formula of camera coordinates and nozzle coordinates;

Step 2: Move the robotic arm to collect images from three directions: main view, top view and left view;

Step 3: Perform morphological operations on the image, eliminate noise interference areas, and determine the circumscribed contour and centroid of the target;

Step 4: Move the robotic arm so that the optical axis of the camera is aligned with the centroid, acquire the image at the first set position of the target, and calculate the feature size of the image at the first set position;

Step 5: Continue to move the robotic arm to the set position, acquire the image of the second set position of the target, calculate the feature size of the image at the second set position, and pass the image of the first set position and the image of the second set position The image calculates the distance between the camera and the target;

Step 6: On the basis of step 5, obtain the position and spatial size of the target in the coordinate system where the camera is located;

Step 7: Calculate the coordinate position of the target in the coordinate system of the manipulator base, and then obtain the three-dimensional size of the target outline and the position information of the target relative to the manipulator base.

The method for determining the circumscribed contour and centroid of the target is: scan the image from four directions, up, down, left and right, determine the contour of the target, and use the center of the rectangular frame of the target's contour as the centroid of the target.

The invention provides a crop automatic spray positioning device and method based on a hand-eye mechanical arm. It can realize the three-dimensional information contour extraction and positioning of the target. The method is accurate in positioning and moderate in calculation, which can meet the real-time requirements of spraying, and has high flexibility and adaptability in the working process. It provides an effective method for realizing precise automatic spraying, reducing environmental pollution, and improving the utilization rate of pesticides. implementation method.

Description of drawings

Fig. 1 is a schematic representation of reconstructed plant outline space;

Figure 2 is a schematic diagram of target distance detection;

Fig. 3 is a schematic diagram of the image processing process of main-view plant contour detection; Fig. 3a is an image of the first position along the optical axis; Fig. 3b is an image of the second position along the optical axis;

Figure 4 is an image processing and control flow chart taking the main view as an example;

Figure 5 is an application example of a four-degree-of-freedom spray robot arm.

Detailed ways

The preferred embodiments will be described in detail below in conjunction with the accompanying drawings. It should be emphasized that the following description is only exemplary and not intended to limit the scope of the invention and its application.

The technical problem to be solved by the present invention is to provide a crop contour detection and positioning method based on the hand-eye spray arm, which can quickly and accurately calculate the approximate contour information of the target crop and the relative positional relationship from the spray arm, and the algorithm has good adaptability and real-time. This method overcomes the limited field of view of the fixed camera binocular vision system, the high requirement for camera calibration accuracy, and the error matching that is easy to occur when occluded.

To solve the above problems, the present invention provides a spray positioning method based on a hand-eye spray robot. Including the base of the robotic arm, the robotic arm, the camera and the nozzle; the robotic arm is fixed on the base of the robotic arm; the camera and the nozzle are respectively installed at the end of the robotic arm. The robotic arm is a four-degree-of-freedom spray robotic arm.

This method uses the control system of the manipulator to move the camera fixed at the end of the manipulator in a specified way according to the specified requirements, and the camera can obtain images at different positions. According to the principle of the camera pinhole imaging model, as long as the corresponding target is found at different positions The corresponding feature size in the image can be combined with the camera calibration parameters to obtain the linear distance between the target crop and the camera lens, that is, the depth information between the target crop and the camera can be obtained, and the distance between the target crop and the camera can be calculated. actual size range. Control the manipulator to change the imaging angle of the camera, move the manipulator along the three directions of the main view, top view and left view, which are orthogonal to the center of the target crop, and take pictures. Finally, the actual circumscribed rectangle outline size of the target crop in the three directions can be obtained, and according to The three-dimensional rectangular block outline information of crops is reconstructed from the projection correspondence (as shown in Figure 1).

According to the principle of camera imaging, the relationship between the coordinate value of the spatial point P on the image plane and the coordinate value of the camera coordinate system is:

$\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; x</span>}\\ \mathrm{<span\; class="notranslate">\; the\; y</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; f</span>}}{{\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}}\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}\end{array}\right]<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Among them: c is the camera coordinate system, the coordinates of point P in the camera coordinate system are (x _c , y _c , z _c ), x and y are the abscissa and ordinate of point P on the image plane; f is the camera lens focal length.

From formula (1) can get:

x z _c =f x _c

In the formula, the focal length f of the camera is a fixed value, if the distance of moving the camera along its optical axis is s, the coordinates x _c and y _c of point P in the camera coordinate system remain unchanged. Available: xz _c = constant.

When working, firstly, an image of the target is obtained at the first position, and then the camera moves a certain distance along its optical axis to obtain the second image (Figure 2). Thus, it can be obtained from the above formula:

x ₁ z _c1 =x ₂ z _c2 ; x ₁ (z _c2 +s)=x ₂ z _c2 ;

Among them, x ₁ and x ₂ are the abscissa coordinates of the target feature in the image coordinate system at the first and second positions respectively; z _c1 and z _c2 are the coordinates of the target point in the z-axis direction of the camera coordinate system at the two positions respectively. coordinate.

The depth information can be calculated by substituting the outer dimension value of the plant canopy with two position changes into the above formula.

Based on the above calculation principles, the image processing process when obtaining the relevant depth information calculation parameters is shown in Figure 3. In the figure, the crop main view image is taken as an example, which reflects the crop image processing and its circumscribed rectangle contour extraction process.

The image processing process and method are as follows:

1. Image preprocessing to filter out noise and segment the image by the color features of the crops to extract the target crops.

2. Identify and initially locate target crops. Determine whether the crop is detected by calculating the ratio of the crop area to the entire image, and use this to judge whether the robotic arm has reached a relatively suitable position for the crop. (The priority order of camera movement is translation first, then vertical).

3. Morphological operation to filter out small interference areas.

4. Scan the image on four sides, determine the circumscribed contour of the target crop, and use the center of the rectangular frame as the centroid of the target crop.

5. After moving the robotic arm to align the camera to the centroid, calculate the feature size of the first position image.

6. After moving the arm along the optical axis, calculate the feature size of the second position image.

7. Substitute in the corresponding length and width, and calculate the distance from formula 2.

8. Substitute the known distance and image coordinates into Equation 1 to find the position and size of the target point under the camera coordinates.

9. Change the position and repeat the above process. Obtain the target crop position and spatial size under the camera coordinates.

10. Calculate the coordinate position of the target relative to the base of the manipulator based on the hand-eye calibration information of the manipulator and the position information of the control system. Finally, the contour and the position information of the target crop relative to the spray base are obtained.

Control the robotic arm to move to the other two positions, calculate the length and width of the outline in the same way, and finally obtain the three-dimensional length, width, height and position information of the crop.

The image processing and control process is shown in Figure 4:

Step 1: Determine the conversion formula of camera coordinates and nozzle coordinates;

Install the camera on the four-degree-of-freedom spray manipulator shown in Figure 5. The robotic arm control system can realize accurate positioning and movement control of the camera installed at the end of the robotic arm. The coordinate position and motion trajectory of the end effector position (spray head) relative to the spray arm base can be controlled in real time.

Hand-eye calibration of the camera. That is to determine the relative conversion relationship between the camera coordinates and the nozzle coordinates. The calibration method is as follows:

1. Fix the black and white grid plane calibration template within the field of view of the camera, move the robotic arm to collect multiple calibration template images, and calculate the internal and external parameters of the camera according to Zhang Zhengyou's calibration method.

2. Find the relative positional relationship between the coordinates of the two cameras at different positions relative to the calibration template

Assuming that the camera moves from the spatial position a to point b, the respective extrinsic parameter matrices A and B relative to the fixed calibration template can be obtained through the previous calibration process, which are known parameter matrices. Therefore, the rotation and translation matrix C of the camera from a to b can be obtained by the formula C=AB ^-1 .

3. Corresponding to the previous step, the relative spatial relationship matrix D of the two corresponding positions of the nozzle (end effector) can be obtained by using the known control system information of the manipulator. Combined with the camera relative position relationship matrix C obtained in the previous step, the hand-eye relationship matrix X can be further obtained.

According to the conversion relationship between the coordinate systems, the above parameter matrix can be obtained to satisfy the relationship CX=XD. Rewrite the above formula as a relationship expressed by the rotation and translation parameters:

$\left[\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}& {\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}\\ {\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]\left[\begin{array}{cc}\mathrm{<span\; class="notranslate">\; R</span>}& \mathrm{<span\; class="notranslate">\; t</span>}\\ {\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\left[\begin{array}{cc}\mathrm{<span\; class="notranslate">\; R</span>}& \mathrm{<span\; class="notranslate">\; t</span>}\\ {\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]\left[\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}}& {\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}\\ {\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

Expand the above formula to get:

R _c R = RR _d (4)

R _c t + t _c = Rt _d + t

What are known in the above formula are R _c , R _d , t _c , and t _d , and R, R _c , and R _d are all orthogonal identity matrices, and R and t need to be solved. Since a set of relational expressions cannot obtain a definite solution, the manipulator is controlled to move to the next position in the same way. Therefore, new equations can be obtained. Applying the least squares method to solve, the X matrix can be finally obtained.

With the hand-eye relationship matrix X, the relationship matrix of the camera at any position relative to the base of the manipulator can be obtained according to the relationship matrix of the nozzle (end effector of the manipulator) relative to the base of the manipulator. When the current positional relationship of the target crop relative to the camera is obtained, the specified motion of the camera relative to the crop can be realized according to the coordinate relationship of the target crop relative to the base of the manipulator. Similarly, according to the position information of the control signal of the manipulator, the relative position relationship between the cameras at any different positions can also be obtained. Calibrate the known system in advance, and use a fixed relative position to collect images during the work process, which can avoid solving a large number of complex calculations in real time, directly use the known calibration information, and use the corresponding relationship of feature points according to the detected crop images to solve the three-dimensional spatial information. Since the relative positions of the cameras at each image acquisition position are known, matching constraints such as epipolar constraints and uniqueness constraints can be used to improve the calculation speed and accuracy of positioning matching. The calculation results can be used to further improve the accuracy and robustness of the localization algorithm.

Step 2: Move the robotic arm to collect images from three directions: main view, top view and left view;

The visual positioning system of the spray manipulator takes the main viewing direction as the initial position, moves the walking platform along the crop row, and collects images. Crop detection by color features. The ultra-green image segmentation algorithm is used to calculate the proportion of the green target area to the entire image area in real time. When the specified value is reached, it means that the target crop appears. Slow down the speed of the mobile platform, and the ratio value obtained from several consecutive pictures will no longer increase. When the value is the same within the error range, it can be considered that a single crop has been found. Stop and move the robotic arm in a small range to make the initial position of the camera relative to the crop reach the specified value. s position. The basis for judging is whether the ratio of the target crop image area to the total image area reaches the specified range.

Step 3: Perform morphological operations on the image to eliminate noise interference areas. Scan the image on four sides, determine the outline of the rectangle circumscribing the target crop, and use the center of the rectangle as the centroid of the target crop. When scanning to the position of the first target pixel, the number of pixels in the calculated connected domain must be greater than the set value to be considered valid, so as to avoid misjudgment of the contour boundary due to interference information.

Step 4: Move the robotic arm so that the optical axis of the camera is aligned with the centroid, acquire the image at the first set position of the target, and calculate the feature size of the image at the first set position;

Step 5: Move the robotic arm to the second set position, acquire the image of the target at the second set position, calculate the feature size of the image at the second set position, and pass the image at the first set position and the second set position The image calculates the distance between the camera and the target;

Step 6: On the basis of step 5, obtain the position and spatial size of the target in the coordinate system where the camera is located;

Step 7: Calculate the coordinate position of the target in the coordinate system of the manipulator base, and then obtain the three-dimensional size of the target outline and the position information of the target relative to the spray base.

According to the calibration information of the hand-eye of the manipulator and the position information of the control system, the coordinate position of the target crop relative to the base of the manipulator is calculated. Finally, the three-dimensional size of the crop outline and the position information of the target crop relative to the base of the robotic arm are obtained.

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of changes or modifications within the technical scope disclosed in the present invention. Replacement should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (2)

1. a method of carrying out spray location based on trick spraying mechanical arm, positions target by mechanical arm and camera, it is characterized in that the method comprises the following steps:

Step 1: the conversion formula of determining camera coordinates and shower nozzle coordinate;

Step 2: mobile mechanical arm, respectively from main depending on, overlook and a left side is looked three directions and gathered images;

Step 3: image is carried out to morphology operations, eliminate noise jamming region, determine external profile and the centre of form of target;

Step 4: mobile mechanical arm, make the optical axis alignment centre of form of video camera, obtain the image of the first desired location of target, calculate the characteristic size of the image of the first desired location;

Step 5: mobile mechanical arm to the second desired location, obtain the image of the second desired location of target, calculate the characteristic size of the image of the second desired location, by the image of the first desired location and the image calculation video camera of the second desired location and the distance of target;

Step 6: on the basis of step 5, try to achieve position and the bulk of target in the coordinate system of video camera place;

Step 7: calculate target in the coordinate system of mechanical arm pedestal place coordinate position, and then obtain the three-dimensional dimension of objective contour and the target positional information with respect to mechanical arm pedestal.

2. the method for spray location according to claim 1, it is characterized in that the external profile of described definite target and the method for the centre of form are: from upper and lower, left and right four direction scan image, determine the profile of target, and with the center of the rectangle frame of the profile of target the centre of form as target.

Images