CN106097304A - A kind of unmanned plane real-time online ground drawing generating method - Google Patents

Abstract

The present invention proposes a real-time online map generation method for unmanned aerial vehicles, which realizes real-time online reconstruction of the map on the unmanned aerial vehicle, then transmits the reconstructed map back to the ground through a high-bandwidth data transmission link, and applies it to the unmanned aerial vehicle to obtain its real-time A three-dimensional map of the surrounding environment and the realization of its own positioning. Through the present invention, the UAV can fly in the unknown airspace, and under the condition of uncertain location, obtain its own positioning and carry out three-dimensional map by detecting and comparing the environmental information, extracting and matching the characteristic information of the environment build. The present invention can be applied to battlefield requirements (such as autonomous flight and attack of UAVs); emergency rescue of disasters such as fire/earthquake/flood; UAV public security management and monitoring, etc. Compared with the existing methods, this method is at the upper-middle level in terms of position drift, angle drift and absolute error. This method obtains more information about the environment, and compared with the full-dense method, the experimental accuracy of this method is still the same. It can meet the use standard, and can run directly on the CPU, with good real-time performance.

Description

A real-time online map generation method for UAV

technical field

The invention relates to the fields of computer image processing and map surveying, and specifically relates to a method for generating a real-time online map of an unmanned aerial vehicle. By changing the transmission mode of the unmanned aerial vehicle and a ground station, the real-time online map generation of the unmanned aerial vehicle is realized.

Background technique

Traditional surveying and mapping technology uses the feature points and boundaries of the ground to obtain graphics and position information reflecting the current status of the ground through measurement means (such as remote sensing, laser, ultrasonic, etc.). Traditional surveying and mapping technology has high requirements for sensors. Although its surveying and mapping accuracy is high, the high cost and the long time required from information collection to results have restricted the application of traditional surveying and mapping in some occasions (such as relatively high timeliness requirements). .

SLAM (Simultaneous Localization and Mapping) based on computer vision, that is, simultaneous positioning and map construction, has always been a research hotspot for scholars due to its important theoretical and application value. However, the current 3D reconstruction method is that the images captured by the drone's onboard camera are transmitted to the ground computer through high-definition images, and the ground computer performs 3D reconstruction of the captured images. This method has great disadvantages:

1. A large number of images are repeated and redundant;

2. The transmission speed is very high, resulting in a small transmission distance, and it is difficult to transmit the image back to the ground station;

3. The transmission signal is susceptible to interference.

4. The aircraft cannot be made to make decisions and fly automatically.

Contents of the invention

In order to solve the problems existing in the prior art, the present invention proposes a real-time online map generation method for drones, which realizes real-time online reconstruction of the map on the drone, and then transmits the reconstructed map back to the ground through a high-bandwidth data transmission link .

The hardware implementation of the present invention is mainly divided into two parts, the first part is the sky terminal, the main device is to place the microcomputer on the drone, and the drone is directly equipped with a USB camera. During the flight of the UAV, the image information is collected through the USB camera, and the data is directly sent to the microcomputer, and the microcomputer reconstructs the map of the collected image information. The image processing software of the computer provides GPS and transmits it to the DDL video transmission; the second part is the ground terminal, and the information transmission between the air terminal and the ground terminal is mainly realized through the DDL video transmission.

In the whole process, the microcomputer and the flight control system use the serial port to transmit information. Mainly use the USB to UART to transfer the GPS information of the flight control to the microcomputer. The information of the flight control system and the microcomputer will be transmitted to the DDL image transmission on the ground in real time through the DDL image transmission.

On the sky side, the UAV needs to process the collected data information to obtain its own attitude and a map of the three-dimensional environment in real time, which is mainly realized by the following steps:

Step 1: Acquire Images

The drone's onboard camera collects a series of images and transmits the images to the drone's onboard microcomputer;

Step 2: The microcomputer onboard the UAV processes the first frame of image acquired by the camera to obtain an initialization map:

Step 2.1: Perform dedistortion processing on the first frame image to obtain the first frame image after dedistortion;

Step 2.2: Depth initialization of the first frame of image after dedistortion: according to the set gray gradient threshold, screen out the pixels whose gray gradient is greater than the gray gradient threshold in the distorted first frame image, and assign Random depth value of selected pixels;

Step 2.3: Back-project the pixels assigned depth values in step 2.2 back to the three-dimensional environment according to the parameters of the drone’s onboard camera, and obtain the initialized map;

Step 2.4: Set the first frame image after de-distortion as a key frame;

Step 3: Perform the following processing on the i-th frame image obtained by the drone's onboard camera in real time, i=2,3,4...:

Step 3.1: Perform dedistortion processing on the i-th frame image to obtain the i-th frame image after de-distortion;

Step 3.2: Taking the current key frame as the benchmark, perform the operation of aligning the i-th frame image after dedistortion with the reference image, and obtain the pose change from the i-th frame to the current key frame;

Step 3.3: According to the camera pose corresponding to the current key frame, and the pose change from the i-th frame to the current key frame obtained in step 3.2, obtain the camera pose corresponding to the i-th frame and the position of the camera in local coordinates;

Step 3.4: According to the set gray-scale gradient threshold, filter out the pixels whose gray-scale gradient is greater than the gray-scale gradient threshold in the distorted i-th frame image, and obtain the camera pose corresponding to the i-th frame according to the camera parameters and step 3.3 , back-project the screened pixels back to the 3D environment to obtain the screened pixel depth values; and add the screened pixel points with depth values to the map;

Step 3.5: If the pose change from the i-th frame to the current keyframe obtained in step 3.2 is greater than the set pose change threshold, replace the current keyframe with the i-th frame as a new keyframe.

Step 4: After the real-time image processing of the drone's onboard camera with a preset number of frames is completed, the drone's onboard microcomputer transmits the generated map to the ground terminal for display through the drone's onboard DDL image.

Further preferred scheme, described a kind of unmanned aerial vehicle real-time online map generating method is characterized in that:

Extract and store the feature points in each key frame image;

If the image alignment operation in step 3.2 cannot be realized, perform failed reconstruction:

Extract the feature points of the de-distorted current frame image, match the feature points of the current frame with the feature points in each stored key frame image, and find the key frame with the largest number of successfully matched feature points. If the number of successfully matched feature points accounts for no more than 40% of the total number of feature points in the key frame, then take the current frame as the first frame and return to step 2; The frame image is aligned with the benchmark image to obtain the pose change from the current frame to the current key frame;

According to the camera pose corresponding to the benchmark, and the pose change from the current frame to the current key frame, the camera pose corresponding to the current frame is obtained;

According to the set gray gradient threshold, filter out the pixels whose gray gradient is greater than the gray gradient threshold in the distorted current frame image, and back-project the filtered pixels according to the camera parameters and the camera pose corresponding to the current frame Go back to the 3D environment to get the depth value of the selected pixels; add the screened pixels with depth values to the map; then proceed to step 3.

In a further preferred solution, the method for generating a real-time online map of an unmanned aerial vehicle is characterized in that: a fast corner point detection method is used to extract feature points.

A further preferred solution, the method for generating a real-time online map of an unmanned aerial vehicle, is characterized in that: in step 3.4, in the process of adding the screened pixel points with depth values to the map, if a certain pixel point reverses After projection, if there is already a map 3D point in the neighborhood of the corresponding 3D point in the map, then the corresponding 3D point in the map and the existing map 3D point in the neighborhood of the 3D point are removed after the pixel is back-projected , and add the corresponding 3D point in the map after back-projection of the pixel point, and the weighted average point of the existing map 3D points in the neighborhood of the 3D point to the map.

Further preferred scheme, described a kind of unmanned aerial vehicle real-time online map generation method is characterized in that: in step 3.5, if step 3.2 obtains the pose change of i frame to current key frame greater than the set pose change threshold , and the frame number difference between the i-th frame and the current key frame is not less than 15 frames, then use the i-th frame to replace the current key frame as a new key frame.

Further preferred scheme, described a kind of unmanned aerial vehicle real-time online map generation method is characterized in that: the image alignment operation in step 3.2 adopts the following process:

First set the initial value of the pose change from the i-th frame to the current key frame; according to the pose change from the i-th frame to the current key frame, filter out the pixels whose gray gradient is greater than the gray gradient threshold in the current key frame Back-project to the 3D environment, and then project from the 3D environment onto the i-th frame image after dedistortion to obtain the projection point; and find it on the i-th frame image after de-distortion, the gray gradient selected in the current key frame is greater than The corresponding point of the pixel point of the gray gradient threshold; calculate the residual sum of the photometric value of the projected point and the corresponding point; iteratively change the pose change from the i-th frame to the current key frame to minimize the residual sum of the photometric value.

A further preferred solution, the method for generating a real-time online map of a UAV, is characterized in that: the pose change from the i-1th frame to the current key frame is used as the initial change in pose from the i-th frame to the current key frame value.

Further preferred scheme, described a kind of unmanned aerial vehicle real-time online map generating method, it is characterized in that: in step 3.4, after obtaining the pixel point depth value that is screened out; Adopt graph optimization method to i-th frame corresponding camera in local The position under the coordinates and the selected pixel position with depth value are optimized, and the optimized pixel point with depth value is added to the map.

A further preferred solution, the method for generating a real-time online map of an unmanned aerial vehicle, is characterized in that: the reconstructed map is converted to the world coordinate system:

In the UAV real-time map reconstruction process, the trajectory information X _n of the UAV in the world coordinate system is obtained at each frame time through the satellite positioning signal, and n represents the nth frame; and in the UAV real-time map During the reconstruction process, the position x _n of the camera corresponding to each frame in the local coordinate system is obtained; through the optimization function

$\underset{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}{\mathrm{<span\; class="notranslate">\; argmin</span>}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Get the transformation matrix δ corresponding to the minimum value of the optimization function, where N is the total number of frames in the UAV real-time map reconstruction process, T(x _n , δ) represents the projection conversion function from the local coordinate system to the world coordinate system, δ is the transformation matrix from the local coordinate system to the world coordinate system; according to the projection transformation function corresponding to the obtained transformation matrix δ, transform the reconstructed map into the world coordinate system.

In a further preferred solution, the method for generating a real-time online map of a UAV is characterized in that: when the satellite positioning signal frequency is lower than the frame frequency, for each satellite positioning signal collection time t _n , the world coordinates of the UAV are obtained The trajectory information X _n under the system; and use the position interpolation of the camera in the local coordinate system corresponding to a frame before and after the acquisition time t _n to obtain the position x _n of the camera under the local coordinate system at the acquisition time t _n ; through the optimization function

$\underset{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}{\mathrm{<span\; class="notranslate">\; argmin</span>}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Obtain the transformation matrix δ corresponding to the minimum value of the optimization function, where N is the total collection points of satellite positioning signals, T(x _n , δ) represents the projection conversion function from the local coordinate system to the world coordinate system, and δ is the transformation from the local coordinate system Transformation matrix to the world coordinate system; transform the reconstructed map into the world coordinate system according to the projection transformation function corresponding to the obtained transformation matrix δ.

Beneficial effect

Compared with many existing methods, the method proposed by the present invention is at the upper-middle level in terms of position drift, angle drift and absolute error. This method obtains more information about the environment, and compared with the full dense method, this method The accuracy of the experiment can still meet the use standard, and it can run directly on the CPU without requiring the GPU, and the real-time performance is good.

Moreover, the present invention adopts a real-time online map generation method on the UAV, which makes image information transfer easier to realize, reduces the storage of data volume, and the amount of information transmitted, specifically: 1. There are no large amounts of repeated and redundant pictures 2. The amount of transmitted information is small, so that the transmission rate is small and the transmission distance is long; 3. Real-time reconstruction of the map when the drone is flying can allow the aircraft to make decisions and fly automatically, which can be used for barriers and path planning Wait.

Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and comprehensible from the description of the embodiments in conjunction with the following drawings, wherein:

Figure 1: Schematic diagram of determining pixel depth by triangulation projection;

Figure 2: Schematic diagram of the direct image alignment method;

Figure 3: Schematic diagram of failed reconstruction;

Figure 4: Schematic diagram of graph optimization method;

Figure 5: Schematic diagram of fitting satellite navigation data;

Figure 6: A frame of original image captured by the camera;

Figure 7: 3D point cloud image of real-time reconstruction;

Figure 8: Maps reconstructed in real-time online using drones;

Figure 9: The stitched map;

Figure 10: Schematic diagram of the air end of the UAV online real-time map reconstruction system;

Figure 11: Schematic diagram of the ground terminal of the UAV online real-time map reconstruction system.

detailed description

Embodiments of the present invention are described in detail below, and the embodiments are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

The hardware implementation of this embodiment is mainly divided into two parts. The first part is performed on the air terminal, as shown in FIG. 10 , and the second part is performed on the ground terminal, as shown in FIG. 11 .

During the whole process, the onboard microcomputer and the flight control system use USB-UART for information transmission. Mainly use the USB-UART to transmit the GPS information of the flight control to the airborne microcomputer. The information of the flight control system and the onboard microcomputer will be transmitted to the ground terminal for display through the onboard DDL image in real time.

The specific implementation steps of this embodiment are as follows:

The task in the sky is to reconstruct the real-time map during the flight:

Step 1: Acquire Images

The drone's onboard camera collects a series of images, and transmits the images to the drone's microcomputer via USB2.0 or USB3.0; this transmission method is much faster than remote transmission.

Step 2: The microcomputer onboard the UAV processes the first frame of image acquired by the camera to obtain an initialization map:

Step 2.1: Use the pre-acquired calibration data to perform de-distortion processing on the first frame of image to obtain the de-distorted first frame of image for subsequent processing.

Step 2.2: Depth initialization of the first frame of image after dedistortion: according to the set gray gradient threshold, screen out the pixels whose gray gradient is greater than the gray gradient threshold in the distorted first frame image, and assign The random depth value of the selected pixels; such random processing will not affect the subsequent reconstruction accuracy, because after dozens of frames of processing, the depth map will gradually approach an accurate model.

Step 2.3: According to the camera parameters, back-project the pixels assigned depth values in step 2.2 back to the 3D environment to obtain the initialized map.

Step 2.4: Set the first frame of image after dedistortion as a key frame; extract and store the feature points in the key frame. The methods for feature point detection are: 1. SIFT, 2. SURF, 3. fast-corner (fast corner detection: through Gaussian filtering, then corner detection), etc., because the present invention is intended to be applied to real-time generation of three-dimensional maps of the environment, the program pays attention to real-time, Although the first two methods have high precision and good effects, they take a long time and are not suitable for real-time operation. Therefore, fast-corner is used to detect feature points in this step and subsequent steps.

Step 3: Perform the following processing on the i-th frame image obtained by the camera in real time, i=2,3,4...:

Step 3.1: Perform de-distortion processing on the i-th frame image to obtain the i-th frame image after de-distortion.

Step 3.2: After the i-th frame image data is loaded in, the system takes the current key frame as the reference (i.e., the tracking basis), and performs the alignment operation between the de-distorted i-th frame image and the reference image, and obtains the i-th frame to the current key frame pose changes.

Due to the high frame rate, the difference (time, space) between adjacent frames is not very large. It is assumed that the luminosity value of the same pixel does not change in such a small time interval between two frames (the change is small and negligible) . This article directly compares the photometric values of the two images, and obtains the pose change between the two frames by minimizing the photometric error, that is, completes the image alignment operation:

First set the initial value of the pose change from the i-th frame to the current key frame.

According to the pose change from the i-th frame to the current key frame, back-project the pixels with a gray gradient greater than the gray gradient threshold selected in the current key frame to the 3D environment, and then project from the 3D environment to the i-th image after dedistortion On the frame image, the projection point is obtained; and on the i-th frame image after dedistortion, the corresponding point of the pixel whose gray gradient is greater than the gray gradient threshold screened out in the current key frame is found; the projection point and the corresponding point are calculated The residual sum of the photometric value; iteratively change the pose change from the i-th frame to the current key frame to minimize the residual sum of the photometric value.

Due to the high frame rate, the pose change between two frames is approximately constant in a small area, so the pose change from the i-1th frame to the current keyframe can be used as the i-th frame to the current keyframe The initial value of the pose change of .

Using the parameter matrix of the camera to back-project the pixels into the three-dimensional environment is to calculate the pose change between the current frame (i.e. frame i) and the current key frame through triangulation, that is, SE(3) transformation: SE(3) is a A 4×4 matrix representing position and attitude changes (also called an extrinsic matrix in the camera projection equation):

This matrix is mainly divided into two parts, where from a ₀₀ to a ₂₂ is SO(3), which represents the change of attitude (angle) in three-dimensional space, and from T ₀ to T ₂ represents the change of position, namely (x, y, z) the amount of change. SIM3 consists of SE(3) plus Scale parameters, SO(3)*s can turn SE(3) into SIM3:

s represents the Scale parameter, which is used for affine transformation.

Step 3.3: According to the camera pose corresponding to the current key frame and the pose change from the i-th frame to the current key frame obtained in step 3.2, obtain the camera pose corresponding to the i-th frame and the position of the camera in local coordinates.

Step 3.4: According to the set gray-scale gradient threshold, filter out the pixels whose gray-scale gradient is greater than the gray-scale gradient threshold in the distorted i-th frame image, and obtain the camera pose corresponding to the i-th frame according to the camera parameters and step 3.3 , and back-project the screened pixels back to the 3D environment to obtain the screened pixel depth values; and add the screened pixel points with depth values to the map.

Due to the strong front-to-back dependence of the current system, in order to reduce the influence of the error from the sensor on the final result and correct the pose of each step, the graph optimization method is used here to determine the position of the camera corresponding to the i-th frame under the local coordinates, and the corresponding The selected pixel points with depth values are optimized, and the optimized pixels with depth values are added to the map. The well-known graph optimization method formula is:

$\underset{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}}{\mathrm{<span\; class="notranslate">\; argmin</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;\; e</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; \&Omega;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>$

x _k represents the previously defined nodes (also can be understood as states), Z _k represents edges (also can be understood as constraints), e _k represents the state of these nodes satisfying the constraints (if there is no noise and e _k = 0), Ω _k represents The introduced information matrix and the confidence degree of the constraint, if the error is large, its corresponding confidence degree will be small. After defining the variables, what we have to do is to minimize the entire error function to achieve the global optimum. In this method, we define the three-dimensional position of the map point and the pose of the UAV as nodes, and define the projection relationship from the image to the map point and the change of SE(3) between two adjacent frames as edges, and the information The matrix contains two aspects: the number of key frame observations of map points and the gray gradient of points in the image.

In addition, in step 3.4, during the process of adding the screened pixels with depth values to the map, if a certain pixel is back-projected, the neighborhood of the corresponding three-dimensional point in the map (a set small value), if there is already a 3D point on the map, the corresponding 3D point in the map will be back-projected after the pixel point, and the existing map 3D point in the neighborhood of the 3D point will be removed, and the pixel point will be back-projected on the The weighted average point of the corresponding 3D point in the map and the existing map 3D point in the neighborhood of the 3D point is added to the map.

Step 3.5: If the pose change from the i-th frame to the current keyframe obtained in step 3.2 is greater than the set pose change threshold, replace the current keyframe with the i-th frame as a new keyframe. In this embodiment, in order to improve the calculation rate and reduce the amount of stored data, it is required that if the pose change from the i-th frame to the current key frame obtained in step 3.2 is greater than the set pose change threshold, and the i-th frame and the current key frame If the frame difference is not less than 15 frames, the i-th frame is used to replace the current key frame as a new key frame.

The establishment of the key frame is because its pose has a large change compared with the previous key frame, and the three-dimensional environment information detected by it is quite different from the previous key frame, so it is set as a ruler for Expand the global map and detect whether subsequent frames have large pose changes.

During the tracking process, if there is a "frame loss" phenomenon (possible reasons are: the camera moves too fast, resulting in too large a "gap" between the current frame and the current key frame, and it is impossible to track on the current key frame, if this If no processing is performed, there will be no close "connection" between the two maps created before and after, making all the previous work meaningless), so it is judged that if the image alignment operation in step 3.2 cannot be realized, the reconstruction will fail:

Extract the feature points of the de-distorted current frame image, match the feature points of the current frame with the feature points in each stored key frame image, and find the key frame with the largest number of successfully matched feature points. If the number of successfully matched feature points accounts for no more than 40% of the total number of feature points in the key frame, then take the current frame as the first frame and return to step 2; The frame image is aligned with the benchmark image to obtain the pose change from the current frame to the current key frame;

According to the camera pose corresponding to the benchmark, and the pose change from the current frame to the current key frame, the camera pose corresponding to the current frame is obtained;

According to the set gray gradient threshold, filter out the pixels whose gray gradient is greater than the gray gradient threshold in the distorted current frame image, and back-project the filtered pixels according to the camera parameters and the camera pose corresponding to the current frame Go back to the 3D environment to get the depth value of the selected pixels; add the screened pixels with depth values to the map; then proceed to step 3.

Since the above-mentioned reconstructed 3D environment is based on the local coordinate system and does not match the real 3D environment on a unified scale, in order to better apply the reconstructed map, the following is based on satellite positioning signals for fitting, and the reconstructed The 3D environment is matched to the real 3D environment at a unified scale.

In the UAV real-time map reconstruction process, the trajectory information X _n of the UAV in the world coordinate system is obtained at each frame time through the satellite positioning signal, and n represents the nth frame; and in the UAV real-time map During the reconstruction process, the position x _n of the camera corresponding to each frame in the local coordinate system is obtained; through the optimization function

$\underset{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}{\mathrm{<span\; class="notranslate">\; argmin</span>}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Get the transformation matrix δ corresponding to the minimum value of the optimization function, where N is the total number of frames in the UAV real-time map reconstruction process, T(x _n , δ) represents the projection conversion function from the local coordinate system to the world coordinate system, δ is the transformation matrix from the local coordinate system to the world coordinate system; according to the projection transformation function corresponding to the obtained transformation matrix δ, transform the reconstructed map into the world coordinate system.

In addition, the frequency of satellite positioning signals is often smaller than the frame frequency, and the acquisition time of satellite positioning signals is not strictly aligned with the image acquisition time. At this time, for each satellite positioning signal acquisition time t _n , the UAV in the world coordinate system is obtained Trajectory information X _n ; and interpolate the position of the camera in the local coordinate system corresponding to a frame before and after the acquisition time t _n to obtain the position x _n of the camera in the local coordinate system at the acquisition time t _n ; through the optimization function

$\underset{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}{\mathrm{<span\; class="notranslate">\; argmin</span>}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Obtain the transformation matrix δ corresponding to the minimum value of the optimization function, where N is the total collection points of satellite positioning signals, T(x _n , δ) represents the projection conversion function from the local coordinate system to the world coordinate system, and δ is the transformation from the local coordinate system Transformation matrix to the world coordinate system; transform the reconstructed map into the world coordinate system according to the projection transformation function corresponding to the obtained transformation matrix δ.

Step 4: After the real-time image processing of the drone's onboard camera with a preset number of frames is completed, the drone's onboard microcomputer transmits the generated map to the ground terminal for display through the drone's onboard DDL image. For example, it can be set that after the onboard microcomputer processes the 10th frame, the 20th frame... and other pre-set frame images, the real-time generated map will be transmitted to the ground terminal for display through the DDL image carried by the drone .

The reconstruction results are given below, including real-time and offline data test results, and compared with some current mainstream visual SLAM methods to evaluate system performance. All the following tests are performed on mint17 (based on Ubuntu14.04, 64bit), 8G RAM, without GPU, and the main frequency is 3.50GHz.

Table 1: Comparison of RGB-D benchmark results

Table 2: Comparison of absolute tracking error test results (unit cm)

This patented method [3] [4] [5] [6] fr2-xyz 1.46 3.78 24.29 1.21 2.7 Sim 0.36 2.31 - 0.14 - Sim-desk 0.05 1.54 - 0.26 - fr2-desk 4.53 13.51 x 1.78 9.6

Note: [1] The method is: Dense RGB-D Odometry; [2] The method is: PTAM (keypoint-based); [3] The method is: semi-densemono-VO; [4] The method is: keypoint-based mono- SLAM; [5] the method is: DirectRGB-D SLAM; [6] the method is: keypoint-based RGB-D SLAM; - indicates that the data of this method is not found, and X indicates that tracking failure (Tracking Lost); used for comparison The data comes from Computer Vision Group; through comparison, it can be found that the method of this patent is at the upper-middle level in terms of position drift, angle drift and absolute error. Compared with other methods, our method obtains more information about the environment, and at the same time, compared With the full-dense method, the accuracy of our experiment can still meet the usage standard, and it can run directly on the CPU without requiring a GPU.

In addition, this method verifies real-time map construction on a UAV, as shown in Figures 8 and 9. In this system, the UAV is equipped with a high-definition camera and a microcomputer. Using the microcomputer to run this method can calculate the sparse point cloud of the earth in real time. Using the calculated aircraft pose, point cloud, and feature points, the captured The images are spliced in real time to obtain a two-dimensional map. This experiment can prove that the method of the present invention can realize real-time reconstruction of maps.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be construed as limitations to the present invention. Variations, modifications, substitutions, and modifications to the above-described embodiments are possible within the scope of the present invention.

Claims (10)

1. A real-time online map generation method for unmanned aerial vehicle, is characterized in that: comprise the following steps: Step 1: Acquire Images The drone's onboard camera collects a series of images and transmits the images to the drone's onboard microcomputer; Step 2: The microcomputer onboard the UAV processes the first frame of image acquired by the camera to obtain an initialization map: Step 2.1: Perform dedistortion processing on the first frame image to obtain the first frame image after dedistortion; Step 2.2: Depth initialization of the first frame of image after dedistortion: according to the set gray gradient threshold, screen out the pixels whose gray gradient is greater than the gray gradient threshold in the distorted first frame image, and assign Random depth value of selected pixels; Step 2.3: Back-project the pixels assigned depth values in step 2.2 back to the three-dimensional environment according to the parameters of the drone’s onboard camera, and obtain the initialized map; Step 2.4: Set the first frame image after de-distortion as a key frame; Step 3: Perform the following processing on the i-th frame image obtained by the drone's onboard camera in real time, i=2,3,4...: Step 3.1: Perform dedistortion processing on the i-th frame image to obtain the i-th frame image after de-distortion; Step 3.2: Taking the current key frame as the benchmark, perform the operation of aligning the i-th frame image after dedistortion with the reference image, and obtain the pose change from the i-th frame to the current key frame; Step 3.3: According to the camera pose corresponding to the current key frame, and the pose change from the i-th frame to the current key frame obtained in step 3.2, obtain the camera pose corresponding to the i-th frame and the position of the camera in local coordinates; Step 3.4: According to the set gray-scale gradient threshold, filter out the pixels whose gray-scale gradient is greater than the gray-scale gradient threshold in the distorted i-th frame image, and obtain the camera pose corresponding to the i-th frame according to the camera parameters and step 3.3 , back-project the screened pixels back to the 3D environment to obtain the screened pixel depth values; and add the screened pixel points with depth values to the map; Step 3.5: If the pose change from the i-th frame obtained in step 3.2 to the current key frame is greater than the set pose change threshold, replace the current key frame with the i-th frame as a new key frame; Step 4: After the real-time image processing of the drone's onboard camera with a preset number of frames is completed, the drone's onboard microcomputer transmits the generated map to the ground terminal for display through the drone's onboard DDL image. 2. a kind of unmanned aerial vehicle real-time online map generation method according to claim 1, is characterized in that: Extract and store the feature points in each key frame image; If the image alignment operation in step 3.2 cannot be realized, perform failed reconstruction: Extract the feature points of the de-distorted current frame image, match the feature points of the current frame with the feature points in each stored key frame image, and find the key frame with the largest number of successfully matched feature points. If the number of successfully matched feature points accounts for no more than 40% of the total number of feature points in the key frame, then take the current frame as the first frame and return to step 2; The frame image is aligned with the benchmark image to obtain the pose change from the current frame to the current key frame; According to the camera pose corresponding to the benchmark, and the pose change from the current frame to the current key frame, the camera pose corresponding to the current frame is obtained; According to the set gray gradient threshold, filter out the pixels whose gray gradient is greater than the gray gradient threshold in the distorted current frame image, and back-project the filtered pixels according to the camera parameters and the camera pose corresponding to the current frame Go back to the 3D environment to get the depth value of the selected pixels; add the screened pixels with depth values to the map; then proceed to step 3. 3. a kind of unmanned aerial vehicle real-time online map generation method according to claim 2, is characterized in that: adopt fast corner point detection method to extract feature point. 4. A kind of unmanned aerial vehicle real-time online map generation method according to claim 1, it is characterized in that: in step 3.4, in the process of adding the pixel point with the depth value screened out to the map, if a certain pixel point After back-projection, if there is already a 3D point on the map in the neighborhood of the corresponding 3D point in the map, then the corresponding 3D point in the map and the existing 3D point on the map in the neighborhood of the 3D point will be back-projected. Remove, and add the corresponding three-dimensional point in the map after the back projection of the pixel point, and the weighted average point of the existing map three-dimensional point in the neighborhood of the three-dimensional point to the map. 5. a kind of unmanned aerial vehicle real-time online map generation method according to claim 1, is characterized in that: in step 3.5, if step 3.2 obtains the pose change of i frame to current key frame greater than the pose change of setting threshold, and the frame number difference between the i-th frame and the current key frame is not less than 15 frames, then use the i-th frame instead of the current key frame as a new key frame. 6. a kind of unmanned aerial vehicle real-time online map generation method according to claim 1, is characterized in that: the image alignment operation in the step 3.2 adopts the following process: First set the initial value of the pose change from the i-th frame to the current key frame; according to the pose change from the i-th frame to the current key frame, filter out the pixels whose gray gradient is greater than the gray gradient threshold in the current key frame Back-project to the 3D environment, and then project from the 3D environment onto the i-th frame image after dedistortion to obtain the projection point; and find it on the i-th frame image after de-distortion, the gray gradient selected in the current key frame is greater than The corresponding point of the pixel point of the gray gradient threshold; calculate the residual sum of the photometric value of the projected point and the corresponding point; iteratively change the pose change from the i-th frame to the current key frame to minimize the residual sum of the photometric value. 7. a kind of unmanned aerial vehicle real-time online map generation method according to claim 6, is characterized in that: adopt i-1th frame to the pose change of current key frame as the pose change of i frame to current key frame initial value. 8. A kind of unmanned aerial vehicle real-time online map generating method according to claim 1, it is characterized in that: in step 3.4, after obtaining the pixel point depth value that is screened out; Adopt graph optimization method to the i-th frame corresponding camera in The position under the local coordinates and the selected pixel point with depth value are optimized, and the optimized pixel point with depth value is added to the map. 9. according to claim 1 or 8 described a kind of unmanned aerial vehicle real-time online map generation method, it is characterized in that: the map after reconstruction is converted under the world coordinate system: In the UAV real-time map reconstruction process, the trajectory information X _n of the UAV in the world coordinate system is obtained at each frame time through the satellite positioning signal, and n represents the nth frame; and in the UAV real-time map During the reconstruction process, the position x _n of the camera corresponding to each frame in the local coordinate system is obtained; through the optimization function $\underset{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}{\mathrm{<span\; class="notranslate">\; argmin</span>}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$ Get the transformation matrix δ corresponding to the minimum value of the optimization function, where N is the total number of frames in the UAV real-time map reconstruction process, T(x _n , δ) represents the projection conversion function from the local coordinate system to the world coordinate system, δ is the transformation matrix from the local coordinate system to the world coordinate system; according to the projection transformation function corresponding to the obtained transformation matrix δ, transform the reconstructed map into the world coordinate system. 10. according to claim 1 or 8 described a kind of UAV real-time online map generating method, it is characterized in that: when satellite positioning signal frequency is less than frame frequency, for each satellite positioning signal collection time t _n , get UAV Trajectory information X _n in the world coordinate system; and use the position interpolation of the camera in the local coordinate system corresponding to a frame before and after the acquisition time t _n to obtain the position x _n of the camera in the local coordinate system at the acquisition time t _n ; through optimization function $\underset{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}{\mathrm{<span\; class="notranslate">\; argmin</span>}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; \&delta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$ Obtain the transformation matrix δ corresponding to the minimum value of the optimization function, where N is the total collection points of satellite positioning signals, T(x _n , δ) represents the projection conversion function from the local coordinate system to the world coordinate system, and δ is the transformation from the local coordinate system Transformation matrix to the world coordinate system; transform the reconstructed map into the world coordinate system according to the projection transformation function corresponding to the obtained transformation matrix δ.