CN110047142A - No-manned plane three-dimensional map constructing method, device, computer equipment and storage medium - Google Patents

Abstract

The present application relates to a method for constructing a three-dimensional map of an unmanned aerial vehicle. The method includes: acquiring a video frame image captured by a camera, extracting feature points in each video frame image, and using a color histogram and a scale-invariant feature transformation mixing and matching algorithm Match the feature points to obtain feature point matching pairs, calculate the pose transformation matrix according to the feature point matching pairs, determine the three-dimensional coordinates corresponding to each video frame image according to the pose transformation matrix, and convert the three-dimensional coordinates of the feature points in the video frame image. Convert to the world coordinate system to obtain a 3D point cloud map, use the video frame image as the input of the target detection model, obtain the target object information, combine the 3D point cloud map with the target object information, and obtain a 3D point cloud containing the target object information. map. This method improves the real-time and accuracy of 3D point cloud map construction, and contains rich information. In addition, a UAV 3D map construction device, computer equipment and storage medium are also proposed.

Description

UAV 3D map construction method, device, computer equipment and storage medium

technical field

The invention relates to the field of computer technology, and in particular, to a method, device, computer equipment and storage medium for constructing a three-dimensional map of an unmanned aerial vehicle.

Background technique

With the development of science and technology, drones have become increasingly miniaturized and intelligent, and their flight space has expanded to jungles, cities and even buildings. Environmental perception is the basis for UAVs to understand, navigate, plan, and act on the environment during their work. Among them, the most important goal of environmental perception is to build a complete three-dimensional map, and then plan and navigate the path based on the three-dimensional map. The construction of traditional three-dimensional maps is either low in accuracy or low in real-time, and the amount of information in the constructed map is small, which affects the planning and navigation of subsequent paths.

SUMMARY OF THE INVENTION

Therefore, it is necessary to provide a method, device, computer equipment and storage medium for UAV 3D map construction that can meet the requirements of good real-time performance under the premise of ensuring high precision and contain a large amount of information in response to the above problems.

In a first aspect, an embodiment of the present invention provides a method for constructing a three-dimensional map of a UAV, the method comprising:

Obtain the video frame images captured by the camera, and extract the feature points in each video frame image;

Match the feature points between the video frame images by using the color histogram and the scale-invariant feature transformation mixed matching algorithm to obtain the feature point matching pairs between the video frame images;

Calculate the pose transformation matrix between the video frame images according to the feature point matching pairs between the video frame images;

Determine the three-dimensional coordinates corresponding to each video frame image according to the pose transformation matrix;

Convert the three-dimensional coordinates of the feature points in the video frame image to the world coordinate system according to the three-dimensional coordinates corresponding to the video frame image and the corresponding pose transformation matrix to obtain a three-dimensional point cloud map;

The video frame image is used as the input of the target detection model, and the target object information in the video frame image detected by the target detection model is obtained;

The three-dimensional point cloud map is combined with the target object information to obtain a three-dimensional point cloud map containing the target object information.

In a second aspect, an embodiment of the present invention provides a device for constructing a three-dimensional map of an unmanned aerial vehicle, the device comprising:

The extraction module is used to obtain the video frame images captured by the camera, and extract the feature points in each video frame image;

The matching module is used to match the feature points between the video frame images by using the color histogram and the scale-invariant feature transformation mixed matching algorithm to obtain the feature point matching pairs between the video frame images;

A calculation module, for calculating the pose transformation matrix between the video frame images according to the feature point matching pairs between the video frame images;

a determining module, configured to determine the three-dimensional coordinates corresponding to each video frame image according to the pose transformation matrix;

The conversion module is used to convert the three-dimensional coordinates of the feature points in the video frame image to the world coordinate system according to the three-dimensional coordinates corresponding to the video frame image and the corresponding pose transformation matrix to obtain a three-dimensional point cloud map;

a detection module, configured to use the video frame image as an input of a target detection model, and obtain target object information in the video frame image detected by the target detection model;

The combining module is used for combining the three-dimensional point cloud map with the target object information to obtain a three-dimensional point cloud map containing the target object information.

In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the following steps:

Obtain the video frame images captured by the camera, and extract the feature points in each video frame image;

Match the feature points between the video frame images by using the color histogram and the scale-invariant feature transformation mixed matching algorithm to obtain the feature point matching pairs between the video frame images;

Calculate the pose transformation matrix between the video frame images according to the feature point matching pairs between the video frame images;

Determine the three-dimensional coordinates corresponding to each video frame image according to the pose transformation matrix;

Convert the three-dimensional coordinates of the feature points in the video frame image to the world coordinate system according to the three-dimensional coordinates corresponding to the video frame image and the corresponding pose transformation matrix to obtain a three-dimensional point cloud map;

The video frame image is used as the input of the target detection model, and the target object information in the video frame image detected by the target detection model is obtained;

The three-dimensional point cloud map is combined with the target object information to obtain a three-dimensional point cloud map containing the target object information.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, causes the processor to perform the following steps:

Obtain the video frame images captured by the camera, and extract the feature points in each video frame image;

Match the feature points between the video frame images by using the color histogram and the scale-invariant feature transformation mixed matching algorithm to obtain the feature point matching pairs between the video frame images;

Calculate the pose transformation matrix between the video frame images according to the feature point matching pairs between the video frame images;

Determine the three-dimensional coordinates corresponding to each video frame image according to the pose transformation matrix;

Convert the three-dimensional coordinates of the feature points in the video frame image to the world coordinate system according to the three-dimensional coordinates corresponding to the video frame image and the corresponding pose transformation matrix to obtain a three-dimensional point cloud map;

The video frame image is used as the input of the target detection model, and the target object information in the video frame image detected by the target detection model is obtained;

The three-dimensional point cloud map is combined with the target object information to obtain a three-dimensional point cloud map containing the target object information.

The above-mentioned method, device, computer equipment and storage medium for constructing a three-dimensional map of an unmanned aerial vehicle can improve the accuracy of feature point matching by using a color histogram and a scale-invariant feature transformation mixed matching algorithm to match feature points between video frame images. degree and real-time. In addition, the target detection model is used to identify and detect the target in the video frame image, and the target information is combined with the three-dimensional point cloud map to obtain a three-dimensional point cloud map containing the object information, so that the established three-dimensional point cloud map contains There is richer information. That is to say, the accuracy of 3D map construction is improved by mixing and matching of color histogram and scale-invariant feature transformation, and combined with the target information recognized by the target detection model, the 3D point cloud map contains richer content, which is for the follow-up. It provides support for optimal path planning.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained according to the structures shown in these drawings without creative efforts.

1 is a flowchart of a method for constructing a three-dimensional map of an unmanned aerial vehicle in one embodiment;

2 is a schematic diagram of a method for constructing a three-dimensional map of an unmanned aerial vehicle in one embodiment;

3 is a schematic diagram of the combination of color histogram and SIFT feature matching in one embodiment;

4 is a schematic diagram of training and prediction of a UAV target detection model based on deep learning in one embodiment;

5 is a structural block diagram of an apparatus for constructing a three-dimensional map of an unmanned aerial vehicle in one embodiment;

6 is a structural block diagram of an apparatus for constructing a three-dimensional map of an unmanned aerial vehicle in another embodiment;

7 is a structural block diagram of a device for constructing a three-dimensional map of an unmanned aerial vehicle in another embodiment;

FIG. 8 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

As shown in FIG. 1, a method for constructing a three-dimensional map of a UAV is proposed. The method for constructing a three-dimensional map of an UAV is applied to a UAV or a terminal or server connected to the UAV. Taking man-machine as an example, it includes the following steps:

Step 102 , acquiring video frame images captured by the camera, and extracting feature points in each video frame image.

Among them, feature points can be simply understood as more prominent points in the image, such as contour points, bright spots in darker areas, and dark spots in brighter areas. In one embodiment, the camera of the UAV may use an RGB-D camera to acquire the color image and the depth image obtained by shooting, align the acquired color image and the depth image in time, and then extract the features in the color image The feature extraction of feature points can use color histogram and scale-invariant feature transformation for feature extraction.

Step 104 , using a color histogram and a scale-invariant feature transformation mixing and matching algorithm to match the feature points between the video frame images to obtain feature point matching pairs between the video frame images.

Among them, the color histogram matching algorithm focuses on the matching of color features, and the scale invariant feature transform (SIFT) focuses on the matching of shape features. Therefore, the color histogram matching algorithm and the scale transformation feature transformation are mixed, that is, the "color" of the color histogram is combined with the "shape" of the SIFT algorithm, thereby improving the accuracy of feature recognition and improving the accuracy of feature point matching. At the same time, it is also beneficial to improve the real-time performance of recognition, thereby improving the real-time performance and accuracy of subsequent 3D point cloud map generation.

After the feature points in each video frame image are extracted, feature matching is performed according to the features of the feature points to obtain feature point matching pairs between the video frame images. Since the drone is constantly flying, the position of the same point in the real space is different in different video frame images. By acquiring the features of the feature points in the video frames before and after, and then matching according to the features, the real space is obtained. The position of the same point in different video frames.

In one embodiment, two adjacent video frame images are acquired, the features of multiple feature points are extracted from the previous video frame image and the next video frame image, and then the features of the feature points are matched to obtain the previous video frame image. The matched feature points in the video frame image and the next video frame image constitute a feature point matching pair. For example, the feature points in the previous video frame image are respectively P1, P2, P3..., Pn, and the corresponding matching feature points in the next video frame image are respectively Q1, Q2, Q3..., Qn. Among them, P1 and Q1 are feature point matching pairs, P2 and Q2 are feature point matching pairs, P3 and Q3 are feature point matching pairs, and so on. The matching of feature points can use Brute Force or Fast Approximate Nearest Neighbor (FLANN) algorithm for feature matching. The Fast Approximate Nearest Neighbor algorithm is to judge whether the ratio of the nearest matching distance and the second nearest matching distance exceeds the set threshold. If it exceeds the preset threshold, it is determined that the matching is successful, so as to reduce the mismatched point pairs.

Step 106: Calculate the pose transformation matrix between the video frame images according to the feature point matching pairs between the video frame images.

Wherein, after the positions of the feature points in the video frame images are determined, the pose transformation matrix between the video frame images can be calculated and obtained according to the correspondence between the positions.

Step 108: Determine the three-dimensional coordinates corresponding to each video frame image according to the pose transformation matrix.

The three-dimensional coordinates corresponding to the video frame images refer to the three-dimensional coordinates corresponding to the camera in the drone. After the pose transformation matrix between the video frame images is known, the three-dimensional coordinates of any video frame image can be calculated according to the transformation relationship. The three-dimensional coordinates corresponding to the video frame image actually refer to the camera when the video frame image is captured. The 3D point coordinates corresponding to the position.

Step 110: Convert the three-dimensional coordinates of the feature points in the video frame image to the world coordinate system according to the three-dimensional coordinates corresponding to the video frame image and the corresponding pose transformation matrix to obtain a three-dimensional point cloud map.

Among them, since the three-dimensional coordinates corresponding to each video frame image are the three-dimensional coordinates in the corresponding camera coordinate system, the coordinates of the feature points in the video frame image are also in the camera coordinate system. In order to convert the coordinates of the feature points to In the world coordinate system, the transformation is performed according to the pose transformation matrix to obtain the three-dimensional coordinates of the feature points in the world coordinates, thereby obtaining the three-dimensional point cloud map.

Step 112 , taking the video frame image as the input of the target detection model, and acquiring target object information in the video frame image detected by the target detection model.

Among them, the target detection model is obtained by pre-training, and the target detection model is used to detect the target object appearing in the video frame image, such as a car. Since the video frame image may contain multiple objects, if the category of each object needs to be identified, it is necessary to train multiple target detection models accordingly. After training the target detection model, using the video frame image as the input of the target detection model, the target object in the video frame image and the position of the target object can be detected.

Step 114 , combine the three-dimensional point cloud map with the target object information to obtain a three-dimensional point cloud map including the target object information.

Among them, after the information of the target object in the obtained video frame image is matched with the feature points on the three-dimensional point cloud map, the feature point corresponding to the target object can be determined, and the target object information corresponding to the feature point can be marked in the 3D point cloud map, so that the established 3D point cloud map has more abundant information. The target detection model is used for local perception, and the construction of 3D point cloud map is based on global perception, which combines global perception and local perception, thereby improving the richness of 3D point cloud map.

The above-mentioned UAV 3D map construction method can improve the accuracy and real-time performance of feature point matching by using a color histogram and a scale-invariant feature transformation mixed matching algorithm to match feature points between video frame images. In addition, the target detection model is used to identify and detect the target in the video frame image, and the target information is combined with the three-dimensional point cloud map to obtain a three-dimensional point cloud map containing the object information, so that the established three-dimensional point cloud map contains There is richer information. That is to say, the accuracy of 3D map construction is improved by mixing and matching of color histogram and scale-invariant feature transformation, and combined with the target information recognized by the target detection model, the 3D point cloud map contains richer content, which is for the follow-up. The optimal path planning provides support and improves the intelligence level of UAV environmental perception.

As shown in FIG. 2 , in one embodiment, a schematic diagram of a method for constructing a three-dimensional map of a UAV includes two parts: global perception and local perception. In the global perception, the color histogram and the SIFT feature are used to match the structural framework, and then the localization and the construction of the 3D point cloud map are carried out. The local perception uses the target detection model to identify the target in the video frame image. Finally, the two are combined to obtain a 3D point cloud map containing the target information.

In one embodiment, using a color histogram and a scale-invariant feature transformation mixing and matching algorithm to match feature points between video frame images to obtain feature point matching pairs between video frame images, including: using color histograms The graph feature matching algorithm matches the feature points between the video frame images to obtain the first matching pair set; the scale-invariant feature transformation matching algorithm is used to further match the matching points in the first matching pair set to obtain the target feature point matching pairs.

Among them, the color histogram is used to perform preliminary feature point matching to obtain the first matching pair set, and then the scale-invariant feature transformation matching algorithm is used to further match the matching points in the first matching pair set to obtain the target feature point matching pair. . In one embodiment, the matching of the color histogram is calculated using the Bhattacharyya distance, or the Correlation distance. As shown in FIG. 3 , it is a schematic diagram of the combination of color histogram and SIFT feature matching in one embodiment, and the two are in a cascade relationship.

In one embodiment, calculating the pose transformation matrix between the video frame images according to the feature point matching pairs between the video frame images includes: acquiring the three-dimensional coordinates of each feature point in the feature point matching pairs; Calculate the converted three-dimensional coordinates obtained by converting the three-dimensional coordinates of the feature points in one video frame image to another video frame image; obtain the target three-dimensional coordinates corresponding to the corresponding matching feature points in the another video frame image; According to the converted three-dimensional coordinates The coordinates and the three-dimensional coordinates of the target are calculated to obtain a pose transformation matrix.

Among them, after the matching pair of feature points is determined, the three-dimensional coordinates of each feature point are obtained. The three-dimensional coordinates are obtained according to the color image and depth image captured by the RGB-D camera. The color image is used to identify the x and y value, the depth image is used to get the corresponding z value. For two video frame images, the feature point matching pairs are taken as two sets respectively, the set of feature points in the first video frame image is {P|P _i ∈ R ³ , i=1, 2KN}, the second video frame image The set of feature points in the image is {Q|Q _i ∈ R ³ , i=1, 2KN}, the error between the two point sets is used as the cost function, and the corresponding rotation matrix R is obtained by minimizing the cost function and translation vector t. It can be expressed by the following formula:

where R and t are the rotation matrix and translation vector, respectively. The steps of the iterative closest point algorithm are:

1) The nearest point corresponding to each point in Q in P _i is denoted as Q _i ;

2) Find the minimum transformation matrix R and t according to the above formula;

3) Use R and t to perform rigid body transformation on the point set P to obtain a new point set Compute the error distance between the new point set and the point set Q:

In practice, the constrained rotation matrix and translation vector can be represented by unconstrained Lie algebra, and the number of feature points whose error distance is less than the set threshold, that is, the number of interior points, can be recorded. If the error distance _Ed calculated in step 3) is less than the threshold and the inner point is greater than the set threshold, or whether the number of iterations reaches the set threshold, the iteration ends; if not, go to step 1) for the next iteration.

In one embodiment, the target detection model is obtained by training based on a deep learning model; before the target object is obtained by using the video frame image as the input of the target detection model to obtain the output of the target detection model, It also includes: acquiring training video image samples, the training video image samples include positive samples and negative samples, and the positive samples include a target and a position mark of the target in the video image; according to the training video The image samples are used to train the target detection model to obtain a trained target detection model.

Among them, the target detection model is obtained by training a deep learning model. In order to train the target detection model, first obtain training video image samples, and set positive samples and negative samples. A positive sample is a video image containing the target object and the position mark of the target object in the video image. Object detection model for objects. As shown in FIG. 4 , in one embodiment, a schematic diagram of training and prediction of a UAV target detection model based on deep learning is divided into two parts: preprocessing and real-time detection. To detect targets in real time, first preprocess the data collected by the drone, divide the collected video stream into video frame images, mark the targets in the images, and divide them into training and test data sets. The learning framework trains the model, and then applies the saved model to the video stream returned by the platform to complete the real-time detection of the target.

Use a small drone carrier, equipped with an industrial camera, and widely sample a large amount of video data for the scene from the perspective of the drone, determine the target that the drone needs to identify, and mark the target to be identified in the acquired video data. Use the preprocessed data to train the neural network model, adjust the model parameters until the training results meet the convergence conditions, save the training model for subsequent target detection, load the trained model to the UAV, and use the UAV to detect the target Experiment, and constantly adjust the optimization model.

In a specific embodiment, the deep learning model adopts the YOLOv3 network structure (also called Darknet-53) and adopts a fully convolutional network, including: the introduction of a residual (residual) structure, that is, the ResNet skip layer connection method, and a large number of residual Poor network characteristics. Downsampling is performed using convolution with a stride of 2, while upsampling and route operations are used to perform 3 detections in one network structure. The bounding boxes are predicted using dimensional clustering as anchor boxes, and the object score for each bounding box is predicted by logistic regression using the sum of the squared error losses during training. If the previous bounding box is not the best, and after overlapping the object under test by more than a certain threshold, we ignore the prediction and move on. We use a threshold of 0.5 to systematically assign only one bounding box to each object under test. If the previous bounding box is not assigned to the object under test, there is no penalty for coordinate or class prediction. Each box uses multi-label classification to predict the class that the bounding box is likely to contain. During training, a binary cross-entropy loss is used for class prediction. The YOLOv3 lightweight target detection neural network structure is applied to the UAV platform, which improves the ability of real-time target recognition under the limited computing power of the UAV.

In one embodiment, the combining the three-dimensional point cloud map with the target object information to obtain a three-dimensional point cloud map containing the target object information includes: acquiring the detected target of the target object in the video frame image position; determine the matching feature point according to the target position; mark the target object information on the three-dimensional point cloud map according to the feature point.

Among them, according to the position of the detected target in the video frame image and the position of the feature point in the video frame image, the target object information matching the feature point is determined, and the target object information is marked on the three-dimensional point cloud map, so as to Get a more informative 3D point cloud map.

In one embodiment, the method further includes: acquiring measurement data measured by an inertial measurement unit; calculating an initial pose transformation matrix between video frames according to the measurement data; The feature point matching pair is calculated to obtain the pose transformation matrix between the video frame images, including: calculating the target position between the video frames according to the initial pose transformation matrix and the feature point matching pair between the video frame images. Pose transformation matrix.

Among them, an inertial measurement unit (Inertial measurement unit, IMU) is a device for measuring the three-axis attitude angle (or angular rate) and acceleration of an object. The inertial measurement unit is used as the inertial parameter measurement device of the UAV, which includes a three-axis gyroscope, a three-axis acceleration and a three-axis magnetometer. The UAV can directly read the measurement data measured by the inertial measurement unit, including: angular velocity, acceleration and magnetometer data. After the measurement data measured by the inertial measurement unit is obtained, the pose transformation matrix of the UAV can be directly calculated according to the measurement data. Since the inertial measurement unit will have accumulated errors, the obtained pose transformation matrix of the UAV is not enough. precise. In order to distinguish it from the subsequent optimized pose transformation matrix, the pose transformation matrix directly calculated according to the measurement data is called the "initial pose transformation matrix". The pose transformation matrix includes a rotation matrix R and a translation vector t. In one embodiment, the initial pose transformation matrix corresponding to the measurement data is obtained by calculating by using a complementary filtering algorithm. After the initial pose transformation matrix is obtained, the initial pose transformation matrix is used as the initial matrix, and the iterative closest point (Iterative Closest Point, ICP) algorithm is used to calculate the target between the video frames according to the feature point matching pairs between the video frame images. Pose transformation matrix. Using the initial pose transformation matrix obtained by the inertial measurement unit as the initial matrix is beneficial to improve the calculation speed.

In an embodiment, after the calculating the pose transformation matrix between the video frame images according to the feature point matching pair between the video frame images, the method further includes: calculating the distance between the current video frame and the previous key frame If the motion amount is greater than the preset threshold, the current video frame is used as a key frame; when the current video frame is a key frame, the current video frame is matched with the key frame in the previous key frame library, if the There is a key frame matching the current video frame in the key frame library, then the current video frame is used as the loopback frame; the corresponding pose transformation matrix is optimized and updated according to the loopback frame, and the updated pose transformation matrix is obtained; Determining the three-dimensional coordinates corresponding to each video frame image by the pose transformation matrix includes: determining the three-dimensional coordinates corresponding to each video frame image according to the updated pose transformation matrix.

Among them, in order to reduce the complexity of subsequent optimization, the computational complexity can be reduced by extracting key frames. Because the collected video frames are relatively dense, for example, 30 frames can be collected in one second. It can be seen that the similarity between frames is very high, or even exactly the same, so calculating each frame will undoubtedly increase the calculation. the complexity. So the complexity can be reduced by extracting keyframes. Specifically, the first video frame is first used as a key frame, and then by calculating the amount of motion between the current video frame and the previous key frame, if the amount of motion is within a certain threshold range, it is selected as a key frame, where the calculation formula for the amount of motion is:

Among them, Em represents the measure of the amount of motion, t _x , _ty , _{t z} represent the three translation distances of the translation vector t, Represents the Euler angle of rotation between frames, which can be converted from the rotation matrix. ω ₁ , ω ₂ are the balance weights of translation and rotation, respectively. For the visual field captured by the camera, rotation is easier than translation to bring greater scene changes , so the value of ω ₂ is larger than that of ω ₁ , and the specific value should be adjusted according to the specific situation.

After the key frames are extracted, the loop closure detection method is used to optimize and update the obtained pose transformation matrix. In one embodiment, loop closure detection is performed using a closed loop detection algorithm. After loop closure detection, the target pose transformation matrix is updated and optimized according to the loop closure detection result, and a more accurate pose transformation matrix is obtained, which is called "updated pose transformation matrix" in order to distinguish. The three-dimensional coordinates corresponding to each video frame image are determined according to the updated pose transformation matrix.

As shown in Figure 5, a device for constructing a three-dimensional map of an unmanned aerial vehicle is proposed, which includes:

The extraction module 502 is used to obtain the video frame images captured by the camera, and extract the feature points in each video frame image;

The matching module 504 is used to match the feature points between the video frame images by adopting the color histogram and the scale-invariant feature transformation mixing and matching algorithm to obtain the feature point matching pairs between the video frame images;

The calculation module 506 is used to calculate the pose transformation matrix between the video frame images according to the feature point matching pairs between the video frame images;

A determination module 508, configured to determine the three-dimensional coordinates corresponding to each video frame image according to the pose transformation matrix;

The conversion module 510 is used to convert the three-dimensional coordinates of the feature points in the video frame image to the world coordinate system according to the three-dimensional coordinates corresponding to the video frame image and the corresponding pose transformation matrix, so as to obtain a three-dimensional point cloud map;

The detection module 512 is configured to use the video frame image as the input of the target detection model, and obtain the target object information in the video frame image detected by the target detection model;

The combining module 514 is configured to combine the three-dimensional point cloud map with the target object information to obtain a three-dimensional point cloud map containing the target object information.

In one embodiment, the matching module 504 is further configured to use a color histogram feature matching algorithm to match the feature points between the video frame images to obtain a first set of matching pairs; The matching points in the first matching pair set are further matched to obtain target feature point matching pairs.

In one embodiment, the calculation module 506 is further configured to obtain the three-dimensional coordinates of each feature point in the feature point matching pair; calculate the conversion obtained by converting the three-dimensional coordinates of the feature points in one video frame image to another video frame image three-dimensional coordinates; acquiring the three-dimensional coordinates of the target corresponding to the corresponding matching feature points in the other video frame image; and calculating the pose transformation matrix according to the converted three-dimensional coordinates and the three-dimensional coordinates of the target.

In one embodiment, the target detection model is obtained by training based on a deep learning model; the above-mentioned apparatus for constructing a three-dimensional map of an unmanned aerial vehicle further includes: a training module for acquiring training video image samples, wherein the training video image samples include positive sample and negative sample, the positive sample includes the target object and the position mark of the target object in the video image; the target detection model is trained according to the training video image sample, and the trained target detection model is obtained Model.

In one embodiment, the combining module 514 is further configured to acquire the target position of the detected target in the video frame image; determine the matching feature point according to the target position; Object category information is marked on the three-dimensional point cloud map.

As shown in Figure 6, in one embodiment, the above-mentioned device for constructing a three-dimensional map of an unmanned aerial vehicle further includes:

The initial calculation module 505 is used to obtain the measurement data obtained by the inertial measurement unit measurement, and calculate the initial pose transformation matrix between the video frames according to the measurement data;

The computing module is further configured to include: calculating and obtaining a target pose transformation matrix between video frames according to the initial pose transformation matrix and the feature point matching pair between the video frame images.

As shown in FIG. 7 , in one embodiment, the above-mentioned device for constructing a three-dimensional map of an unmanned aerial vehicle further includes:

The key frame determination module 516 is configured to calculate the motion amount between the current video frame and the previous key frame, and if the motion amount is greater than a preset threshold, the current video frame is used as the key frame.

The loopback frame determination module 518 is used to match the current video frame with the key frame in the previous key frame library when the current video frame is a key frame, if there is a matching frame with the current video frame in the key frame library. If the key frame is selected, the current video frame is used as the loopback frame.

The optimization module 520 is configured to optimize and update the corresponding pose transformation matrix according to the loop closure frame to obtain an updated pose transformation matrix.

The determining module 508 is further configured to determine the three-dimensional coordinates corresponding to each video frame image according to the updated pose transformation matrix.

Figure 8 shows an internal structure diagram of a computer device in one embodiment. The computer equipment can be a drone, or a terminal or server connected to the drone. As shown in Figure 8, the computer device includes a processor, memory, and a network interface connected by a system bus. Wherein, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system, and also stores a computer program. When the computer program is executed by the processor, the processor can implement a method for constructing a three-dimensional map of an unmanned aerial vehicle. A computer program may also be stored in the internal memory, and when the computer program is executed by the processor, the processor may execute the method for constructing a three-dimensional map of an unmanned aerial vehicle. The network interface is used to communicate with external devices. Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, the method for constructing a three-dimensional map of an unmanned aerial vehicle provided by the present application may be implemented in the form of a computer program, and the computer program may be executed on a computer device as shown in FIG. 8 . The memory of the computer equipment can store each program template that constitutes the three-dimensional map construction device of the unmanned aerial vehicle. For example, extraction module 502 , matching module 504 , calculation module 506 , determination module 508 , conversion module 510 , detection module 512 and combination module 514 .

A computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor is made to perform the following steps: acquiring a video frame image captured by a camera, extracting feature points in each video frame image; use a color histogram and a scale-invariant feature transformation hybrid matching algorithm to match the feature points between the video frame images to obtain feature point matching pairs between the video frame images; according to the described The feature point matching pairs between the video frame images are calculated to obtain the pose transformation matrix between the video frame images; the three-dimensional coordinates corresponding to each video frame image are determined according to the pose transformation matrix; the three-dimensional coordinates corresponding to the video frame images and The corresponding pose transformation matrix converts the three-dimensional coordinates of the feature points in the video frame image to the world coordinate system to obtain a three-dimensional point cloud map; the video frame image is used as the input of the target detection model to obtain the target detection model detection. obtaining target object information in the video frame image; combining the three-dimensional point cloud map with the target object information to obtain a three-dimensional point cloud map containing the target object information.

In one embodiment, using a color histogram and a scale-invariant feature transformation mixing and matching algorithm to match feature points between video frame images to obtain feature point matching pairs between video frame images, including: using color histograms The graph feature matching algorithm matches the feature points between the video frame images to obtain the first matching pair set; the scale-invariant feature transformation matching algorithm is used to further match the matching points in the first matching pair set to obtain the target feature point matching pairs.

In one embodiment, calculating the pose transformation matrix between the video frame images according to the feature point matching pairs between the video frame images includes: acquiring the three-dimensional coordinates of each feature point in the feature point matching pairs; Calculate the converted three-dimensional coordinates obtained by converting the three-dimensional coordinates of the feature points in one video frame image to another video frame image; obtain the target three-dimensional coordinates corresponding to the corresponding matching feature points in the another video frame image; According to the converted three-dimensional coordinates The coordinates and the three-dimensional coordinates of the target are calculated to obtain a pose transformation matrix.

In one embodiment, the target detection model is obtained by training based on a deep learning model; before the target object is obtained by using the video frame image as the input of the target detection model to obtain the output of the target detection model, It also includes: acquiring training video image samples, the training video image samples include positive samples and negative samples, and the positive samples include a target and a position mark of the target in the video image; according to the training video The image samples are used to train the target detection model to obtain a trained target detection model.

In one embodiment, the combining the three-dimensional point cloud map with the target object information to obtain a three-dimensional point cloud map containing the target object information includes: acquiring the detected target of the target object in the video frame image position; determine the matching feature point according to the target position; mark the object category information on the three-dimensional point cloud map according to the feature point.

In one embodiment, when the computer program is processed by the processor, the computer program is further configured to perform the following steps: acquiring measurement data measured by an inertial measurement unit; calculating an initial pose between video frames according to the measurement data transformation matrix; the calculating and obtaining the pose transformation matrix between the video frame images according to the feature point matching pair between the video frame images, including: according to the initial pose transformation matrix and the video frame image The feature point matching pairs are calculated to obtain the target pose transformation matrix between video frames.

In one embodiment, after the pose transformation matrix between the video frame images is calculated according to the feature point matching pair between the video frame images, when the computer program is processed by the processor, the computer program further uses In performing the following steps: calculating the motion amount between the current video frame and the previous key frame, if the motion amount is greater than a preset threshold, the current video frame is used as a key frame; when the current video frame is a key frame, the current video frame Matching with the key frames in the previous key frame library, if there is a key frame matching the current video frame in the key frame library, the current video frame is used as a loopback frame; according to the loopback frame, the corresponding pose transformation is performed The matrix is optimized and updated to obtain an updated pose transformation matrix; the determining the three-dimensional coordinates corresponding to each video frame image according to the pose transformation matrix includes: determining the corresponding three-dimensional coordinates of each video frame image according to the updated pose transformation matrix. three-dimensional coordinates.

A computer-readable storage medium storing a computer program, when the computer program is executed by a processor, the processor is made to perform the following steps: acquiring a video frame image captured by a camera, extracting features in each video frame image The feature points between the video frame images are matched by the color histogram and the scale-invariant feature transformation mixed matching algorithm to obtain the feature point matching pairs between the video frame images; according to the feature points between the video frame images The matching pairs are calculated to obtain the pose transformation matrix between the video frame images; the three-dimensional coordinates corresponding to each video frame image are determined according to the pose transformation matrix; the three-dimensional coordinates corresponding to the video frame images and the corresponding pose transformation matrix The three-dimensional coordinates of the feature points in the frame image are converted into the world coordinate system to obtain a three-dimensional point cloud map; the video frame image is used as the input of the target detection model, and the target in the video frame image detected by the target detection model is obtained. object information; combine the three-dimensional point cloud map with the target object information to obtain a three-dimensional point cloud map containing the target object information.

In one embodiment, the matching of feature points between video frame images by using a color histogram and a scale-invariant feature transformation hybrid matching algorithm to obtain feature point matching pairs between video frame images includes: using a color histogram The graph feature matching algorithm matches the feature points between the video frame images to obtain the first matching pair set; the scale-invariant feature transformation matching algorithm is used to further match the matching points in the first matching pair set to obtain the target feature point matching pairs.

In one embodiment, calculating the pose transformation matrix between the video frame images according to the feature point matching pairs between the video frame images includes: acquiring the three-dimensional coordinates of each feature point in the feature point matching pairs; Calculate the converted three-dimensional coordinates obtained by converting the three-dimensional coordinates of the feature points in one video frame image to another video frame image; obtain the target three-dimensional coordinates corresponding to the corresponding matching feature points in the another video frame image; According to the converted three-dimensional coordinates The coordinates and the three-dimensional coordinates of the target are calculated to obtain a pose transformation matrix.

In one embodiment, the target detection model is obtained by training based on a deep learning model; before the target object is obtained by using the video frame image as the input of the target detection model to obtain the output of the target detection model, It also includes: acquiring training video image samples, the training video image samples include positive samples and negative samples, and the positive samples include a target and a position mark of the target in the video image; according to the training video The image samples are used to train the target detection model to obtain a trained target detection model.

In one embodiment, the combining the three-dimensional point cloud map with the target object information to obtain a three-dimensional point cloud map containing the target object information includes: acquiring the detected target of the target object in the video frame image position; determine the matching feature point according to the target position; mark the object category information on the three-dimensional point cloud map according to the feature point.

In one embodiment, when the computer program is processed by the processor, the computer program is further configured to perform the following steps: acquiring measurement data measured by an inertial measurement unit; calculating an initial pose between video frames according to the measurement data transformation matrix; the calculating and obtaining the pose transformation matrix between the video frame images according to the feature point matching pair between the video frame images, including: according to the initial pose transformation matrix and the video frame image The feature point matching pairs are calculated to obtain the target pose transformation matrix between video frames.

In one embodiment, after the pose transformation matrix between the video frame images is calculated according to the feature point matching pair between the video frame images, when the computer program is processed by the processor, the computer program further uses In performing the following steps: calculating the motion amount between the current video frame and the previous key frame, if the motion amount is greater than a preset threshold, the current video frame is used as a key frame; when the current video frame is a key frame, the current video frame Matching with the key frames in the previous key frame library, if there is a key frame matching the current video frame in the key frame library, the current video frame is used as a loopback frame; according to the loopback frame, the corresponding pose transformation is performed The matrix is optimized and updated to obtain an updated pose transformation matrix; the determining the three-dimensional coordinates corresponding to each video frame image according to the pose transformation matrix includes: determining the corresponding three-dimensional coordinates of each video frame image according to the updated pose transformation matrix. three-dimensional coordinates.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium , when the program is executed, it may include the flow of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided in this application may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (10)

1. a kind of no-manned plane three-dimensional map constructing method, which is characterized in that the described method includes:

The video frame images that camera is shot are obtained, the characteristic point in each video frame images is extracted；

The characteristic point between video frame images is carried out using color histogram and Scale invariant features transform mixing matching algorithm Matching, obtains the Feature Points Matching pair between video frame images；

According to the Feature Points Matching between the video frame images to the module and carriage transformation matrix being calculated between video frame images；

The corresponding three-dimensional coordinate of each video frame images is determined according to the module and carriage transformation matrix；

According to the corresponding three-dimensional coordinate of video frame images and corresponding module and carriage transformation matrix by the characteristic point in video frame images Three-dimensional coordinate is transformed under world coordinate system, obtains three-dimensional point cloud map；

Using the video frame images as the input of target detection model, the video that the target detection model inspection obtains is obtained Object information in frame image；

By the three-dimensional point cloud map in conjunction with the object information, obtain include object information three-dimensional point cloud Figure.

2. the method according to claim 1, wherein described use color histogram and Scale invariant features transform Mixing matching algorithm matches the characteristic point between video frame images, obtains the Feature Points Matching between video frame images It is right, comprising:

The characteristic point between video frame images is matched using color histogram Feature Correspondence Algorithm, obtains the first matching pair Set；

First matching further matches the match point in set using Scale invariant features transform matching algorithm Obtain target feature point matching pair.

3. the method according to claim 1, wherein according to the Feature Points Matching pair between the video frame images The module and carriage transformation matrix between video frame images is calculated, comprising:

Obtain the three-dimensional coordinate of each characteristic point of Feature Points Matching centering；

It calculates and the three-dimensional coordinate of characteristic point in a video frame images is transformed into the conversion three-dimensional that another video frame images obtain Coordinate；

Obtain in another video frame images the corresponding target three-dimensional coordinate of corresponding matched characteristic point；

Module and carriage transformation matrix is calculated according to the conversion three-dimensional coordinate and the target three-dimensional coordinate.

4. the method according to claim 1, wherein the target detection model is instructed based on deep learning model It gets；

Described using the video frame images as the input of target detection model, the inspection of the target detection model output is obtained Before measuring object, further includes:

Training video image pattern is obtained, the training video image pattern includes positive sample and negative sample, in the positive sample It include object and the object position mark in the video image；

The target detection model is trained according to the training video image pattern, obtains trained target detection mould Type.

5. the method according to claim 1, wherein described believe the three-dimensional point cloud map and the object Breath combine, obtain include object information three-dimensional point cloud map, comprising:

Obtain target position of the object for detecting and obtaining in video frame images；

Matching characteristic point is determined according to the target position；

According to the characteristic point by the object category information labeling to the three-dimensional point cloud map.

6. the method according to claim 1, wherein the method also includes:

Obtain the measurement data that Inertial Measurement Unit measurement obtains；

The initial module and carriage transformation matrix between video frame is calculated according to the measurement data；

The Feature Points Matching according between the video frame images converts the pose being calculated between video frame images Matrix, comprising:

According to the Feature Points Matching between the initial module and carriage transformation matrix and the video frame images to video frame is calculated Between object pose transformation matrix.

7. the method according to claim 1, wherein in the characteristic point according between the video frame images After matching is to the module and carriage transformation matrix being calculated between video frame images, further includes:

The amount of exercise between current video frame and previous key frame is calculated, if amount of exercise is greater than preset threshold, by current video Frame is as key frame；

When the current video frame is key frame, by current video frame and the key frame progress in key frame library before Match, if in the key frame library exist with the matched key frame of current video frame, using current video frame as winding frame；

Update is optimized to corresponding module and carriage transformation matrix according to the winding frame, obtains updating module and carriage transformation matrix；

It is described that the corresponding three-dimensional coordinate of each video frame images is determined according to the module and carriage transformation matrix, comprising: according to it is described more New module and carriage transformation matrix determines the corresponding three-dimensional coordinate of each video frame images.

8. a kind of no-manned plane three-dimensional map structuring device, which is characterized in that described device includes:

Extraction module, the video frame images shot for obtaining camera, extracts the characteristic point in each video frame images；

Matching module, for using color histogram and Scale invariant features transform mixing matching algorithm between video frame images Characteristic point matched, obtain the Feature Points Matching pair between video frame images；

Computing module, for according to the Feature Points Matching between the video frame images to being calculated between video frame images Module and carriage transformation matrix；

Determining module, for determining the corresponding three-dimensional coordinate of each video frame images according to the module and carriage transformation matrix；

Conversion module, for according to the corresponding three-dimensional coordinate of video frame images and corresponding module and carriage transformation matrix by video frame images In the three-dimensional coordinate of characteristic point be transformed under world coordinate system, obtain three-dimensional point cloud map；

Detection module, for obtaining the target detection model using the video frame images as the input of target detection model Detect the object information in obtained video frame images；

Binding modules, in conjunction with the object information, obtaining including object information the three-dimensional point cloud map Three-dimensional point cloud map.

9. a kind of computer equipment, including memory and processor, the memory is stored with computer program, the computer When program is executed by the processor, so that the processor executes the step such as any one of claims 1 to 7 the method Suddenly.

10. a kind of computer readable storage medium is stored with computer program, when the computer program is executed by processor, So that the processor is executed such as the step of any one of claims 1 to 7 the method.