CN102538782B - Helicopter landing guide device and method based on computer vision - Google Patents

Abstract

The invention discloses a computer vision-based helicopter landing guidance device and method. The device includes a ground three-camera visual sensor system, a foreground search module, a tracking and matching module, a pose calculation module and a wireless data transmission module. The ground three The camera visual sensor system, the foreground search module, the tracking and matching module, the pose calculation module and the wireless data transmission module are connected in sequence, and the ground three-camera visual sensor system is connected to the tracking and matching module; compared with the previous vision-guided landing scheme, the present invention The background of the image is the monotonous sky, which effectively reduces background interference and noise; at the same time, the image processing task is completed by the high-performance computer on the ground, which effectively improves the real-time performance and calculation accuracy compared with the airborne computer.

Description

A Helicopter Landing Guidance Method Based on Computer Vision

technical field

The invention relates to the technical field of aircraft navigation, in particular to a visual guidance method for unmanned helicopter landing and a device for realizing the method. the

Background technique

The autonomous landing of micro unmanned helicopters is a very complicated flight phase, which requires accurate altitude, speed and heading and other self-motion information at low speeds and low altitudes, and feeds back to the airborne flight control system to control The position and attitude of the aircraft, landing to the specified position. the

The traditional UAV landing guidance method is to fuse the data of the airborne Global Positioning System (GPS), Inertial Navigation System (INS) and electronic compass to obtain the position and attitude information of the helicopter. However, due to the limitation of the accuracy of the airborne GPS system, as well as landing noise interference, occlusion and other reasons, the GPS system cannot provide sufficiently accurate pose information. However, INS and electronic compass often have large integral errors, and cannot independently and accurately provide positioning information. the

In response to this problem, in recent years, a landing guidance scheme that uses visual sensors to capture ground marks to obtain helicopter position and attitude information has emerged. However, the vision method needs to perform complex calculation processing on the collected images to extract the above pose information. However, limited by the size and load capacity of the helicopter, the performance of the onboard computer system is limited, which means that there are big problems in the real-time and accuracy of the calculation. In addition, when the airborne visual sensor acquires ground images, there are problems such as image shaking and smoke occlusion to varying degrees, so it is still necessary to find a new and more complete helicopter landing scheme. the

Contents of the invention

In order to improve the real-time performance and accuracy of aircraft pose measurement, the present invention provides a method that uses the plastic buffer ball at the end of the helicopter landing gear as a characteristic corner point, uses the visual sensor to track the corner point in real time, and obtains the helicopter pose information through an image processing algorithm. A device and method for guiding a helicopter to land at a fixed point autonomously. the

The object of the present invention is achieved through the following technical solutions: a helicopter landing guidance device, which includes: a ground three-camera visual sensor system, a prospect search module, a tracking and matching module, a pose calculation module and a wireless data transmission module, etc., The ground three-camera visual sensor system, foreground search module, tracking and matching module, pose calculation module and wireless data transmission module are sequentially connected, and the ground three-camera visual sensor system is connected to the tracking and matching module. the

A method for visual guidance of unmanned helicopter landing, comprising the steps of:

(1) The ground three-camera vision sensor system collects images;

(2) Search for the foreground of the image in the image collected in step 1, that is, the four airborne artificial landmarks whose colors are red, yellow, blue, and green;

(3) Use particle filter algorithm to track the above artificial landmarks in real time;

(4) Calculate the position and attitude of the helicopter relative to the ground coordinate system by the pose calculation module;

(5) The ground station computer transmits the pose information obtained in step 4 to the receiver connected to the flight control computer through the wireless data transmission module, and then transmits it to the flight control computer through the RS-232 interface;

(6) The flight control computer guides the helicopter to land at the designated position according to the above position and attitude information.

The beneficial effect of the present invention is, the present invention has adopted the scheme that artificial landmarks are set on the helicopter body, and the aerial landmarks are photographed by the ground multi-camera visual sensor system. The monotonous sky effectively reduces background interference and noise; at the same time, the image processing task is completed by the high-performance computer on the ground, which effectively improves the real-time performance and calculation accuracy compared with the airborne computer. In addition, this solution uses a multi-camera vision system to collect image information from three angles for fusion, which can effectively solve the occlusion problem between landmarks and increase the measurement space range of the system. The invention only adds four artificial landmarks and three cameras on the basis of the original hardware, fully utilizes the original ground station computer and wireless data transmission module, and effectively controls the cost. the

Description of drawings

Below in conjunction with accompanying drawing and embodiment the present invention is further described:

Fig. 1 is the functional block schematic diagram of the helicopter landing device of the present invention;

Fig. 2 is a structural diagram of the ground three-camera vision sensor system;

Fig. 3 is the prospect search module software flowchart;

Fig. 4 is a flow chart of tracking and matching module software;

Fig. 5 is a software flowchart of the pose calculation module;

Fig. 6 is a schematic flow chart of the helicopter landing method of the present invention.

Detailed ways

FIG. 1 is an example of a functional block diagram of a helicopter landing guide device according to the present invention. the

As shown in Figure 1, the helicopter landing guidance device of the present invention comprises: ground three-camera visual sensor system 1, foreground search module 2, tracking matching module 3, pose calculation module 4 and wireless data transmission module 5, ground three-camera visual sensor System 1, foreground search module 2, tracking and matching module 3, pose calculation module 4 and wireless data transmission module 5 are connected in sequence, and ground three-camera visual sensor system 1 is connected to tracking and matching module 3. the

As shown in Figure 2, the ground three-camera vision sensor system includes three optical cameras, and their connection relationship is shown in Figure 2. The optical camera can adopt Logitech's C160 product, but is not limited thereto. the

As shown in Figure 3, the working process of the foreground search module is as follows: the image output by the visual sensor is converted from the RGB domain to the HSV domain; after screening the required color areas for detection, that is, the red, yellow and blue corresponding to the four artificial landmarks , green area; respectively flip the above-mentioned color areas into inverse colors, that is, green, blue, yellow, and red colors; convert the above-mentioned two images into Gray images to remove noise, and perform difference operations to eliminate the background; then the above-mentioned The image uses the Canny operator to find the edge of the sphere; finally, the Hough transform is used to solve the radii of the above four circles and the coordinates of the center of the circle in the image. This module can be implemented by calling OpenCV library functions in C++ language, but not limited thereto. the

The aforementioned artificial landmarks are four hard plastic buffer balls fixed at the end of the landing gear of the helicopter, and their positions relative to the center of mass of the helicopter have been calibrated. The above-mentioned artificial landmarks can use the landing gear attached to the T-REX 600 Nitro Super Pro helicopter produced by Futaba Corporation of Japan, but it is not limited to this. the

As shown in Figure 4, the working process of the tracking and matching module is as follows: set the number of particles, determine the motion model as a first-order adaptive model; set the four landmarks detected in S2 as tracking targets, and use the weighted color histogram Establishing the target model: For each of the above artificial landmarks, let the coordinates of the center of the target, that is, the projection circle, be y, the quantization level of the histogram be m, and the pixel coordinates in the tracking area be {x _i } _i=1,2,…,N , and establish Weighted color histogram target model p={p _u (y)} _u=1,2,..,m where:

;

;

in, is the kernel function bandwidth; is the profile function corresponding to the kernel function; function b divides the pixels into the response bins of the histogram according to their color values; is the Kronecker function; u is the histogram quantization series;

And establish the initial state particle set with a total of N particles:

;

in, Represents the center coordinates and radius information of the landmark projection circle in the i-th particle at frame 0, and the initial weight value of each particle is 1/N. In the subsequent image frames, the system state transition is performed to randomly propagate the particles, and the particle weights are calculated; after the particles are weighted, the coordinate positions of the above-mentioned targets in the image plane are obtained; the coordinate position values are output; and then resampling is performed to redistribute the particles. Location. This module can be implemented in C++ language, but not limited thereto.

As shown in Figure 5, the design and implementation process of the pose calculation module includes the following steps: The conversion relationship between the known helicopter coordinate system and the camera coordinate system is as follows:

;

Among them, (x _c , y _c , z _c ) (x _h , y _h , z _h ) represent the homogeneous coordinates of the landmark center in the camera coordinate system and the homogeneous coordinates in the helicopter coordinate system respectively; The multiplication algorithm is used to solve the rotation matrix R and displacement matrix t of the helicopter coordinate system relative to the origin of the camera coordinate system; the yaw angle, roll angle and pitch angle of the helicopter can be obtained by solving the rotation matrix; the helicopter’s yaw angle can be obtained by solving the displacement matrix Altitude information and distance information relative to the landing point.

The wireless data transmission module can use the APC200A-43 module produced by AMT Technology Co., Ltd., but it is not limited thereto. the

As shown in Figure 6, the unmanned helicopter landing visual guidance method of the present invention comprises the following steps:

In the initial stage, it is assumed that the helicopter has been guided by the global positioning system and has flown to the effective field of view of the ground visual sensor system. The flight control computer switches to the helicopter landing guidance device involved in the present invention to provide high-precision position and attitude information required for landing.

1. The ground three-camera vision sensor system collects images. the

The ground three-camera vision sensor system collects images in real time, and transmits the collected images to the foreground search module. the

2. Search for the foreground of the image in the image collected in step 1, that is, the four airborne artificial landmarks whose colors are red, yellow, blue, and green.

The specific operation is as follows: the foreground search module converts the image output by the ground three-camera visual sensor system from the RGB domain to the HSV domain; after screening the required color areas for detection, that is, the red, yellow, blue, and green areas corresponding to the four artificial landmarks ;Respectively flip the above-mentioned color areas into inverse colors, that is, green, blue, yellow, and red colors; convert the above-mentioned two images into Gray images to remove noise, and perform a difference operation to eliminate the background; then apply Canny to the above-mentioned images operator to find the edge of the sphere; finally, the Hough transform is applied to solve the radii of the above four circles and the coordinates of the center of the circle in the image. the

3. Use the particle filter algorithm to track the above artificial landmarks in real time

The specific operation is as follows: In the tracking and matching module, set the number of particles and determine the motion model as a first-order adaptive model; set the four landmarks detected in step 2 as tracking targets, and use the weighted color histogram to establish the target model : For each of the above artificial landmarks, let the center coordinate of the target, that is, the projection circle, be y, the histogram quantization series be m, and the pixel coordinates in the tracking area be {x _i } _i=1,2,…,N to establish a weighted color histogram Graph target model p={p _u (y)} _u=1,2,..,m where:

;

;

in, is the kernel function bandwidth; is the profile function corresponding to the kernel function; function b divides the pixels into the response bins of the histogram according to their color values; is the Kronecker function; u is the histogram quantization series;

And establish the initial state particle set with a total of N particles

;

in Represents the center coordinates and radius information of the landmark projection circle in the i-th particle at frame 0, and the initial weight value of each particle is 1/N. In the subsequent image frames, the system state transition is performed to randomly propagate the particles, and the particle weights are calculated; after the particles are weighted, the coordinate positions of the above-mentioned targets in the image plane are obtained; the coordinate position values are output; and then resampling is performed to redistribute the particles. Location.

4. Calculate the position and attitude of the helicopter relative to the ground coordinate system by the pose calculation module. the

The specific operation is as follows: The conversion relationship between the known helicopter coordinate system and the camera coordinate system is as follows:

;

Among them, (x _c , y _c , z _c ) (x _h , y _h , z _h ) represent the homogeneous coordinates of the landmark center in the camera coordinate system and the homogeneous coordinates in the helicopter coordinate system respectively; The multiplication algorithm is used to solve the rotation matrix R and displacement matrix t of the helicopter coordinate system relative to the origin of the camera coordinate system; the yaw angle, roll angle and pitch angle of the helicopter can be obtained by solving the rotation matrix; the helicopter’s yaw angle can be obtained by solving the displacement matrix Altitude information and distance information relative to the landing point.

5. The ground station computer transmits the above position and attitude information to the receiver connected to the flight control computer through the wireless data transmission module, and then transmits it to the flight control computer through the RS-232 interface. the

6. The flight control computer guides the helicopter to land at the designated position according to the above position and attitude information. the

According to the above steps, the unmanned helicopter can realize safe fixed-point landing. the

Claims (1)

1. A helicopter landing guidance method based on computer vision, the method has applied the helicopter landing guidance device, and the helicopter landing guidance device comprises: ground three-camera visual sensor system, prospect search module, tracking matching module, pose calculation module With the wireless data transmission module, the three-camera visual sensor system on the ground, the prospect search module, the tracking and matching module, the pose calculation module and the wireless data transmission module are connected in sequence, and the three-camera visual sensor system on the ground is connected to the tracking and matching module; It is characterized in that the method comprises the steps of: (1) The ground three-camera vision sensor system collects images; (2) Search for the foreground of the image in the image collected in step (1), that is, the four airborne artificial landmarks whose colors are red, yellow, blue, and green; (3) Use particle filter algorithm to track the above artificial landmarks in real time; (4) Calculate the position and attitude of the helicopter relative to the ground coordinate system by the pose calculation module; (5) The ground station computer transmits the pose information obtained in step (4) to the receiver connected to the flight control computer through the wireless data transmission module, and then transmits it to the flight control computer through the RS-232 interface; (6) The flight control computer guides the helicopter to land at the designated position according to the above position and attitude information; The step (2) is specifically: converting the image output by the ground three-camera visual sensor system from the RGB domain to the HSV domain; after screening and detecting the required color areas, that is, the red, yellow, blue, and red corresponding to the four artificial landmarks. Green area; respectively flip the above-mentioned color areas into inverse colors, that is, green, blue, yellow, and red colors; convert the above-mentioned two images into Gray images to remove noise, and perform difference operations to eliminate the background; then the above-mentioned images Use the Canny operator to find the edge of the sphere; finally apply the Hough transform to solve the radius of the above four circles and the coordinates of the center of the circle in the image; The step (3) specifically includes: setting the number of particles, determining the motion model as a first-order adaptive model; setting the four landmarks detected in step (2) as tracking targets, and using the weighted color histogram to establish the target Model: For each of the above artificial landmarks, set the center coordinate of the target, that is, the projection circle, as y, the histogram quantization series as m, and the pixel coordinates in the tracking area as {x _i } _i=1,2,…,N to establish a weighted color Histogram target model: p={p _u (y)} _u=1,2,..,m where: ${\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; f</span>}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\frac{\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$ $\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\frac{\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ;</span>$ Among them, h is the kernel function bandwidth; k(||x|| ² ) is the profile function corresponding to the kernel function; function b divides the pixels into the response bins of the histogram according to their color values; δ is the Kronecker function; u Quantize the series for the histogram; And establish the initial state particle set with a total of N particles: ${<span\; class="notranslate">\; \{</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; \}</span>}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; ;</span>$ in Indicates the center coordinates and radius information of the landmark projection circle in the i-th particle at frame 0, and the initial weight value of each particle is 1/N; the system state transition is performed in subsequent image frames to randomly propagate particles, And solve the particle weight; obtain the coordinate position of the above-mentioned targets in the image plane after particle weighting; output the coordinate position value; then resample and redistribute the position of the particle; The step (5) is specifically: the conversion relationship between the known helicopter coordinate system and the camera coordinate system is as follows: $\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}\end{array}\right]<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; R</span>}\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}\\ {\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}\end{array}\right]<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ;</span>$ Among them, (x _c , y _c , z _c ) (x _h , y _h , z _h ) represent the homogeneous coordinates of the landmark center in the camera coordinate system and the homogeneous coordinates in the helicopter coordinate system respectively; The multiplication algorithm is used to solve the rotation matrix R and displacement matrix t of the helicopter coordinate system relative to the origin of the camera coordinate system; the yaw angle, roll angle and pitch angle of the helicopter can be obtained by solving the rotation matrix; the helicopter’s yaw angle can be obtained by solving the displacement matrix Altitude information and distance information relative to the landing point.