CN110209184A - A kind of unmanned plane barrier-avoiding method based on binocular vision system - Google Patents

Abstract

The invention discloses a UAV obstacle avoidance method based on a binocular vision system. The method obtains depth map information after obtaining image information through the binocular vision system; uses an ecological method combined with an exponentially weighted moving average algorithm to carry out the depth map Denoising operation; the threshold segmentation algorithm based on region growth extracts the obstacle contour, and at the same time uses a rectangular frame to fit the disparity value; calculates the obstacle distance through the similar triangle theorem; transfers the obtained distance to the flight controller, and presets After comparing the threshold distances, different commands are sent out, and the UAV completes obstacle avoidance at an accurate time; the invention combines the morphological method with the exponentially weighted moving average algorithm, which improves the denoising degree of the depth map, and improves the accuracy of the UAV and UAV. The calculation accuracy of the distance between obstacles improves the UAV's rapid visual obstacle avoidance ability, and its accuracy is greatly improved compared with existing algorithms.

Description

A UAV obstacle avoidance method based on binocular vision system

technical field

The invention relates to the field of UAV vision and digital image processing, in particular to an UAV obstacle avoidance method based on a binocular vision system.

Background technique

With the continuous deepening of science and technology, unmanned machines have become the current trend, and the market continues to expand. UAVs often perform some special tasks. higher requirement. The vision-based control system is characterized by simple structure, low cost, high cost performance, etc. At the same time, in comparison, the binocular vision system obtains more information than the monocular vision system, and because the binocular vision system can obtain The images are coplanarly processed to obtain distortion-free and line-aligned images. At this time, the measurement accuracy is higher than that of the monocular system, which is conducive to the accurate obstacle avoidance of the UAV. The binocular vision system is currently used in machine navigation, target pursuit and other related fields. Widely used.

The widely used noise reduction method generally uses morphological opening and closing operations to denoise the depth map. Each opening operation is erosion-expansion, and the purpose is to eliminate isolated outliers in the image that are larger than the points in the field. The closing operation is dilation-corrosion, and its purpose is to eliminate isolated outliers in the image that are smaller than the points in the domain. However, the opening and closing operations are independent of each other, and the image effect after the operation is not very ideal, and the pixels with relatively small gray value differences between the noise area and the non-noise area have not been completely eliminated, so there is still room for improvement.

Exponentially weighted moving average algorithm, analyzed from the perspective of transfer function, it has good robustness, and at the same time shows good low-pass filtering performance. For image pixels, it can be regarded as further erosion of the image. However, if it is used alone, the difference between the gray value of each pixel in the noise area of the depth image to be processed and the non-noise area must be within a certain range to have a better performance.

Therefore, the current noise reduction processing effect on the depth image is not ideal, which makes the ranging results of the UAV inaccurate, which in turn leads to errors in the judgment of the UAV when it is in an obstacle, causing it to make a control error.

Contents of the invention

The present invention overcomes the deficiencies in the prior art. On the basis of depth image processing, a UAV obstacle avoidance method based on a binocular vision system is proposed. The distance result is inaccurate, leading to the problem of inaccurate control of the UAV.

The present invention is achieved through the following technical solutions.

A method for avoiding obstacles of an unmanned aerial vehicle based on a binocular vision system, specifically comprising the following steps:

1) Use the binocular camera to obtain the image directly in front of the UAV, and obtain the left image and the right image.

2) Grayscale the image to obtain the feature information of each pixel of the image, and perform stereo matching on this information to obtain the depth map information.

3) Obtain the depth map, and then use the morphological method and the exponentially weighted moving average algorithm to denoise the depth map; the morphological method is to convert the depth map into a binary depth map, and three morphological opening operations are performed on the binary depth map , the three-time pattern closing operation is in the process image; the specific calculation formula of the exponentially weighted moving average algorithm is as follows.

EWMA(t)=aY(t)+(1-a)EWMA(t-1), t=1, 2,..., n.

Among them, EWMA(t) is the estimated value at time t, Y(t) is the measured value at time t, n is the total observation time, a(0<a<1) is the weight coefficient of historical measured values; take multiple a values , and determine the a value corresponding to the maximum signal-to-noise ratio through the signal-to-noise ratio curve.

4) Use the threshold segmentation algorithm based on region growth to extract the obstacle contour, and at the same time use the rectangular frame to fit it.

5) Get the parallax value, so as to get the distance between the obstacle and the drone.

6) Set the obstacle avoidance threshold distance of the UAV, compare the obtained distance with the threshold, and transmit the data information to the flight controller, and the flight controller will make judgments based on the information and make corresponding actions.

Further preferably, in step 1, using the binocular camera to obtain the image of the UAV is specifically: installing the binocular camera directly in front of the UAV, and calibrating the binocular camera, so as to obtain the internal parameter matrix, external parameter matrix, Distortion parameters, use the image obtained by the camera, correct the camera according to the obtained internal and external parameter matrix and distortion parameters, and obtain an undistorted and line-aligned image.

Further, the step 2 is: grayscale the obtained undistorted and line-aligned image, the grayscaled image is filtered by Gaussian filter method, the Laplacian operator sharpens and improves the edge of the image, and obtains the pixel information.

Further preferably, the stereo matching described in step 2 is: use the block matching algorithm to perform stereo matching on the processed image to obtain a depth map, and in the block matching algorithm, use the SAD window to calculate the grayscale of the pixels in the left and right images. The sum of degree differences, the specific calculation formula is:

in, is the gray level of the left image pixel, is the gray level of the right image pixel, is the neighborhood window, and d is the disparity value.

Further preferably, the specific process of obtaining the depth map in step 3 is: using the disparity map and correction parameters to reproject the image points of the left and right image planes in the three-dimensional space, and the calculation formula of the reprojection matrix Q is:

Q=

in,( ) is the principal point of the left image, f is the focal length of the left lens, for right image top and left image middle the corresponding location.

The specific formula for converting from two-dimensional coordinates to three-dimensional coordinates is:

Q

Among them, d is the disparity value.

Therefore, the depth information z can be obtained by using the reprojection matrix Q, and the specific calculation formula is in the form:

z=Z/W.

Further preferably, as described in step 6, the obstacle avoidance threshold distance of the UAV is set, and comparing the obtained distance with the threshold is to obtain the theoretical distance through the similarity theorem of triangles, and compare it with the set threshold distance to obtain different flight modes .

Furthermore, the flight modes include: natural flight mode, alarm mode and obstacle avoidance mode; in the natural flight mode, when the theoretical distance is greater than the threshold distance, the drone transmits information to the flight controller, and the flight controller issues Continue to keep moving forward; the alarm mode is that when the theoretical distance is equal to the threshold distance, the unmanned aerial vehicle transmits information to the flight controller, and the flight controller issues an instruction that the LED light flashes red; the obstacle avoidance mode is that the theoretical distance is less than the threshold distance When , the UAV transmits the information to the flight controller, and the flight controller issues an obstacle avoidance command.

During the flight of the UAV, the actual image is obtained through the binocular vision sensor, the calibration parameters are obtained, the depth map is obtained through stereo matching, the image preprocessing is performed on the depth map, the parallax value is obtained, and the ranging value is obtained, and the threshold distance is set. The UAV flight mode is divided into three types: natural flight mode, alarm mode and obstacle avoidance mode, which are transmitted to the flight controller. The flight controller sends different instructions according to different modes, and the UAV makes corresponding actions.

Compared with the prior art, the present invention has the following beneficial effects.

The present invention combines the noise reduction method combining the morphological opening and closing operation algorithm and the exponentially weighted moving average algorithm. The continuous noise reduction processing of the depth image by the moving average algorithm is the perfection of the morphological operation, and further eliminates the isolated outliers of the points in the field in the depth image. Therefore, the combination of the two may have a more ideal noise reduction effect, so that the UAV The ranging result is more accurate, and the UAV control is more accurate.

The present invention uses morphological opening and closing operations combined with an exponentially weighted moving average algorithm during depth image processing to improve the accuracy of drone control. Compared with the existing single ecological method for processing depth images, the accuracy of the parallax value measured by the present invention is higher, the depth image denoising degree is higher, and the accuracy of UAV control is improved. To some extent, it can be regarded as The flexibility of the drone is improved, the accident rate of the drone is reduced, and the efficiency of the drone's mission is improved.

Description of drawings

FIG. 1 is a calibration diagram of a calibration camera in the present invention.

Fig. 2 is a hardware structural diagram of the drone control method based on the binocular vision system of the present invention.

Fig. 3 is an algorithm flow chart of the UAV control method based on the binocular vision system of the present invention.

Fig. 4 is a graph of signal-to-noise ratio of an image processed by an exponentially weighted moving average algorithm in the present invention.

Fig. 5 is the original left image of the obstacle in the embodiment of the present invention.

Fig. 6 is the original right image of the obstacle in the embodiment of the present invention.

Fig. 7 is an effect diagram of denoising the obstacle depth map through the existing morphological opening and closing operation in the embodiment.

Fig. 8 is a diagram of the denoising effect of the morphological opening and closing operation fusion exponential weighted moving average algorithm on the obstacle depth map according to the present invention.

Detailed ways

In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer, the present invention will be further described in detail in combination with the embodiments and accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. The technical solutions of the present invention will be described in detail below in conjunction with the embodiments and accompanying drawings, but the scope of protection is not limited thereto.

As shown in Fig. 3, the present invention provides a control system for UAV obstacle avoidance, including a positioning module, an inertial measurement module, an image acquisition module and an embedded image processing module.

Among them, the image acquisition module includes a binocular vision system, which is fixed in front of the drone and installed horizontally.

The image acquisition module transmits the left and right images to the embedded image processing module synchronously: the CPU module is responsible for speed correction, the GPU module is responsible for calculating the camera calibration parameters; the positioning module and the inertial measurement module are responsible for the actual position and attitude measurement of the drone, in addition , the measurement module is also responsible for transmitting the measurement data to the embedded image processing module, so as to adjust the calculated speed information; the flight controller receives the information generated by the embedded image processing module, judges the current mode, and issues corresponding instructions. The drone makes corresponding actions.

In this example, the image acquisition module uses a binocular camera with a resolution of 680*240 and a baseline distance of 5cm. Both the baseline distance and the focal length of the lens can be adjusted, and the image information is transmitted to the embedded image processing module through the USB interface.

As shown in Figure 3, the specific process of the algorithm flow of the UAV obstacle avoidance control method based on the binocular vision system of the present invention is as follows: first, the binocular camera is calibrated using the calibration diagram shown in Figure 1, and the internal parameter matrix of the lens is obtained, The distortion parameter matrix and the external parameter matrix between the two cameras store the obtained data information in the embedded image processing module.

The object is photographed by a binocular camera, and the obtained image is grayscaled and image segmented to obtain the grayscaled original left image as shown in Figure 5 and the grayscaled original right image as shown in Figure 6, Use the previously obtained internal and external parameter matrix and distortion parameter matrix to correct the left and right images to generate two undistorted line-aligned images; grayscale the two images, and filter the grayscaled image by Gaussian filter method , the Laplacian operator sharpens and improves the edge of the image to obtain pixel information.

Use the block stereo matching algorithm to perform stereo matching on the processed image to obtain the depth map. In the block matching algorithm, use the SAD window to calculate the sum of the gray difference of the pixels in the left and right images. The specific calculation formula is:

in, is the gray level of the left image pixel, is the gray level of the right image pixel, is the neighborhood window, and d is the disparity value.

In the process of obtaining the depth map, the disparity map and correction parameters are used to reproject the image points of the left and right image planes in the three-dimensional space. The calculation formula of the reprojection matrix Q is:

Q=

in,( ) is the principal point of the left image, f is the focal length of the left lens, for right image top and left image middle the corresponding location.

The specific formula for converting from two-dimensional coordinates to three-dimensional coordinates is:

Q

Among them, d is the disparity value.

Therefore, the depth image z can be obtained by using the reprojection matrix Q, and the specific calculation formula is in the form:

z=Z/W.

Perform denoising operations on the obtained depth map, convert the depth map into a binary depth map, perform three morphological opening operations on the binary depth map, and three morphological closing operations on the process image. The effect of the depth map after the morphological opening and closing operation is shown in Figure 7 As shown, and combined with the exponentially weighted moving average algorithm for denoising, the specific calculation formula of the exponentially weighted moving average algorithm (EWMA) is:

EWMA(t)=aY(t)+(1-a)EWMA(t-1),t=1,2,...,n;

Among them, EWMA(t) is the estimated value at time t, Y(t) is the measured value at time t, n is the total observation time, and a(0<a<1) is the weight coefficient of historical measured values.

In this implementation plan, 0.75 is selected as the weight coefficient of historical measurement values. Referring to the signal-to-noise ratio curve in Figure 4, it can be seen that the signal-to-noise ratio is the largest at this time, and the denoising effect is the best. The denoising effect is shown in Figure 8.

Set the threshold distance, get the theoretical distance through the similarity theorem of triangles, compare it with the set threshold distance, and divide the flight mode into three types: when the theoretical distance is greater than the threshold distance, the drone will send the information to the flight controller, The flight controller issues an instruction to keep moving forward, which is called the natural flight mode; when the theoretical distance is equal to the threshold distance, the UAV sends the information to the flight controller, and the flight controller issues an instruction to flash the LED light red, which is called the alarm mode; the theoretical distance is less than When the threshold distance is reached, the UAV transmits the information to the flight controller, and the flight controller issues an obstacle avoidance command, which is called the obstacle avoidance mode.

Different from the prior art, the present invention provides a UAV control method based on a binocular vision system. In the process of performing missions, UAVs have extremely high requirements on the control accuracy of the fuselage. Using morphological methods to combine The exponentially weighted moving average algorithm can effectively improve the denoising effect of the depth map, and promote a more accurate obstacle distance, so that the accuracy of UAV control is higher. The control method of the drone based on the binocular vision system of the present invention improves the working efficiency of the drone in the operation process and reduces the accident risk of the drone.

The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments. It cannot be determined that the specific embodiments of the present invention are limited thereto. Under the circumstances, some simple deduction or replacement can also be made, all of which should be regarded as belonging to the scope of patent protection determined by the submitted claims of the present invention.

Claims (7)

1. A method for avoiding obstacles of unmanned aerial vehicle based on binocular vision system, is characterized in that, specifically comprises the following steps: 1) Use the binocular camera to obtain the image directly in front of the UAV, and obtain the left image and the right image; 2) Grayscale the image to obtain the feature information of each pixel of the image, and perform stereo matching on this information to obtain the depth map information; 3) Obtain the depth map, and then use the morphological method and the exponentially weighted moving average algorithm to denoise the depth map; the morphological method is to convert the depth map into a binary depth map, and three morphological opening operations are performed on the binary depth map , the three-time pattern closing operation is in the process image; the specific calculation formula of the exponentially weighted moving average algorithm is: EWMA(t)=aY(t)+(1-a)EWMA(t-1), t=1, 2,..., n; Among them, EWMA(t) is the estimated value at time t, Y(t) is the measured value at time t, n is the total observation time, a(0<a<1) is the weight coefficient of historical measured values; take multiple a values , and determine the a value corresponding to the maximum signal-to-noise ratio through the signal-to-noise ratio curve; 4) Use the threshold segmentation algorithm based on region growth to extract the obstacle contour, and at the same time use the rectangular frame to fit it; 5) Get the parallax value, so as to get the distance between the obstacle and the UAV; 6) Set the obstacle avoidance threshold distance of the UAV, compare the obtained distance with the threshold, and transmit the data information to the flight controller, and the flight controller will make judgments based on the information and make corresponding actions. 2. according to claim 1, a kind of UAV obstacle avoidance method based on binocular vision system, it is characterized in that, the image of using binocular camera to obtain UAV described in step 1 is specifically: the binocular camera is installed on In front of the UAV, the binocular camera is calibrated to obtain the internal reference matrix, external reference matrix, and distortion parameters. Using the image obtained by the camera, the camera is corrected according to the obtained internal and external reference matrix and distortion parameters to obtain a distortion-free and line Align the image. 3. A kind of UAV obstacle avoidance method based on binocular vision system according to claim 2, it is characterized in that, described step 2 is: the undistorted and row alignment image that obtains is grayscaled, grayscaled The image is filtered by the Gaussian filter method, and the Laplacian operator sharpens and improves the edge of the image to obtain pixel information. 4. a kind of UAV obstacle avoidance method based on binocular vision system according to claim 1, it is characterized in that, the stereo matching described in step 2 is: utilize block matching algorithm to carry out stereo matching to the image after processing, obtain In the depth map, in the block matching algorithm, the SAD window is used to calculate the sum of the gray difference of pixels in the left and right images. The specific calculation formula is: in, is the gray level of the left image pixel, is the gray level of the right image pixel, is the neighborhood window, and d is the disparity value. 5. A method for avoiding obstacles based on a binocular vision system for UAVs according to claim 1, wherein the specific process of obtaining the depth map in step 3 is: using the parallax map and correction parameters to re-project in three-dimensional space The image points of the left and right image planes, the calculation formula of the reprojection matrix Q is: Q= in,( ) is the principal point of the left image, f is the focal length of the left lens, for right image top and left image middle the corresponding location; The specific formula for converting from two-dimensional coordinates to three-dimensional coordinates is: Q Among them, d is the disparity value; Therefore, the depth information z can be obtained by using the reprojection matrix Q, and the specific calculation formula is in the form: z=Z/W. 6. A kind of UAV obstacle avoidance method based on binocular vision system according to claim 1, it is characterized in that, step 6 described setting UAV obstacle avoidance threshold distance, comparing the obtained distance with the threshold is through triangle According to the similarity theorem, the theoretical distance is obtained, and compared with the set threshold distance, different flight modes are obtained. 7. a kind of UAV obstacle avoidance method based on binocular vision system according to claim 6, is characterized in that, described flight mode comprises: natural flight mode, alarm mode and obstacle avoidance mode; Described natural flight mode When the theoretical distance is greater than the threshold distance, the UAV transmits information to the flight controller, and the flight controller issues instructions to keep moving forward; the alarm mode is that when the theoretical distance is equal to the threshold distance, the UAV transmits information to the flight controller. The flight controller issues an instruction to flash the LED light red; in the obstacle avoidance mode, when the theoretical distance is less than the threshold distance, the UAV transmits the information to the flight controller, and the flight controller issues an obstacle avoidance instruction.