CN107945265A - Real-time dense monocular SLAM method and systems based on on-line study depth prediction network - Google Patents

Abstract

The invention discloses a real-time dense monocular SLAM method based on the online learning depth prediction network: the camera pose of the key frame is obtained by minimizing the photometric error optimization of the high gradient point, and the depth of the high gradient point is predicted by the triangulation method to obtain the current The semi-dense map of the frame; select the online training picture pair, use the block-by-block stochastic gradient descent method to train and update the CNN network model online, and use the trained CNN network model to perform depth prediction on the current frame picture to obtain a dense map; according to the current frame. Performing depth scale regression on the semi-dense map and the predicted dense map to obtain the absolute scale factor of the depth information of the current frame; using the NCC score voting method to select the predicted depth values of each pixel of the current frame according to the two projection results to obtain a predicted depth map, And Gaussian fusion is performed on the predicted depth map to obtain a final depth map. The present invention also provides a corresponding real-time dense monocular SLAM system based on online learning depth prediction network.

Description

Real-time Dense Monocular SLAM Method and System Based on Online Learning Deep Prediction Network

technical field

The invention belongs to the technical field of computer vision three-dimensional reconstruction, and more specifically relates to a real-time dense monocular SLAM method and system based on an online learning depth prediction network.

Background technique

Simultaneous Localization And Mapping (SLAM) technology can predict the pose of the sensor in real time and reconstruct a 3D map of the surrounding environment, so it plays an important role in the fields of UAV obstacle avoidance and augmented reality. Among them, a SLAM system that only relies on a single camera as an input sensor is called a monocular SLAM system. Monocular SLAM has the characteristics of low power consumption, low hardware threshold and simple operation, and is widely used by researchers. However, the existing popular monocular SLAM systems, whether they are feature-based PTAM (Parallel Tracking And Mapping For Small AR Workspaces) and ORB-SLAM (Orb-slam: AVersatile AndAccurate Monocular Slam System), or LSD- SLAM (Lsd-slam: Large-scale Direct Monocular Slam) has two main problems: (1) It can only construct a sparse or semi-dense map of the scene, because only a few key points or the depth of high gradient points can be (2) It has scale uncertainty, and there is a phenomenon of scale drift.

In recent years, the deep convolutional neural network (Convolutional Neural Network, CNN) for monocular image depth estimation has made great progress. Its main principle is to learn the depth and shape, texture, and The intrinsic connection between scene semantics and scene context, etc., so as to accurately predict the depth information of the picture input into the network. The combination of CNN and monocular SLAM can not only improve the complete rate of mapping, but also obtain absolute scale information, thus making up for the defects and deficiencies of monocular SLAM. At present, the most successful system combining the two is called CNN-SLAM (Cnn-slam: Realtime Dense Monocular Slam With Learned Depth Prediction), which uses the result of CNN depth prediction as the initial depth value of the SLAM key frame, and then uses Pixel matching, triangulation, and graph optimization methods optimize the depth of high-gradient points in keyframes to obtain dense 3D reconstruction results and make scale information closer to the real scale. Although certain effects have been achieved, the system still has the following problems:

(1) The depth values of only a few high-gradient pixels are optimized, and the depth values of most of the low-gradient pixels remain unchanged, resulting in unsatisfactory reconstruction effects, especially for unknown scenes; (2) Using the high-gradient pixels in the CNN output The depth information is not accurate enough to predict the scale information, resulting in insufficient initialization, which will increase the error of SLAM system mapping and tracking.

Contents of the invention

In view of the above defects or improvement needs of the prior art, the present invention provides a method and system that combines online learning depth prediction network with monocular SLAM, the purpose of which is to make full use of the advantages of deep convolutional neural network to achieve The dense depth estimation of the key frames of the SLAM system, and the restoration of the real scale information of the scene based on the results, thereby solving the technical problems of the lack of scale information and the inability to achieve dense mapping in traditional monocular SLAM.

In order to achieve the above object, according to one aspect of the present invention, a real-time dense monocular SLAM method based on online learning depth prediction network is provided, including:

(1) Select the key frame from the image sequence collected by the monocular vision sensor through the rotation and translation motion, optimize the camera pose of the key frame by minimizing the photometric error of the high gradient point, and use the triangulation method to predict the depth of the high gradient point Get the semi-dense map of the current frame;

(2) Select an online training picture pair according to the key frame, use the block-by-block stochastic gradient descent method to train and update the CNN network model online according to the online training picture pair, and use the trained CNN network model to perform depth prediction on the current frame picture get a dense map;

(3) performing depth scale regression according to the semi-dense map of the current frame and the predicted dense map to obtain the absolute scale factor of the depth information of the current frame;

(4) Project the predicted dense map to the previous key frame through pose transformation according to the camera pose, and project the semi-dense map to the previous key frame according to the absolute scale factor, using the NCC score voting method Selecting the predicted depth values of each pixel in the current frame according to the two projection results to obtain a predicted depth map, and performing Gaussian fusion on the predicted depth map to obtain a final depth map.

In an embodiment of the present invention, the selection of the online training picture according to the key frame is specifically: using the following constraints to select the picture frame and the key frame to form a picture pair in the frame pictures before and after the key frame:

First, the camera motion constraint: the displacement in the horizontal direction between the two frames of pictures satisfies |t _x |>0.9*T, where T represents the baseline distance between the two frames of pictures;

Second, disparity constraints: For each pair of pictures, the optical flow method is used to calculate the average disparity Dis _avg in the vertical direction between the pictures, and only when Dis _avg is smaller than the preset threshold δ, the pair of pictures will be saved as candidate training pictures;

Third, diversity constraints: the same key frame can only generate a pair of training pictures;

Fourth, training pool capacity constraints: Whenever the number of training picture pairs reaches the set threshold V, the pictures in the training pool are sent to the network, the network is trained online, the network model obtained from the training is saved, and the training time is cleared. The pool continues to filter the training data.

In one embodiment of the present invention, a block-by-block stochastic gradient descent method is used to train and update the CNN network model online according to the online training pictures, specifically:

The convolutional layer in ResNet-50 is divided into 5 blocks, each of which is specifically represented as conv1, conv2_x, conv3_x, conv4_x, conv5_x; conv1 consists of a single 7X7 full convolutional layer; conv2_x consists of a 3X3 The convolution layer and 3 bottleneck building blocks are composed of 10 layers; conv3_x is composed of 4 bottleneck building blocks with 12 layers; conv4_x is composed of 6 bottleneck building blocks with 18 layers; conv5_x is composed of 3 bottleneck building blocks with 9 layers , the five parts together constitute the 50-layer structure of ResNet-50;

In each online learning and updating process, each iteration k, only update a part of the parameters W _i (i=1,2,3,4,5), keep the remaining 4 parts of the network layer parameters unchanged, And in the next iteration, update the i-th block parameter, where i=(i+1)%5; other layer parameters remain unchanged, and the entire iteration of online learning and updating continues until the preset stop condition is met.

In one embodiment of the present invention, the online training and updating CNN network model is a selective update, specifically:

Calculate the training loss function of each batch of pictures input into the CNN network model. Once the loss function of all pictures in a batch of pictures is greater than the preset threshold L _high , the process of online learning and updating will be started. Online learning and updating The process will continue until the loss function of the training image drops below the threshold L _low , or the number of iterations reaches the preset threshold.

In an embodiment of the present invention, the depth scale regression method is: RANSAC algorithm or least squares algorithm.

In an embodiment of the present invention, the predicted dense map is projected into the previous key frame through pose transformation, and the semi-dense map is projected into the previous key frame according to the absolute scale factor, using The NCC score voting method selects each pixel depth prediction value of the current frame according to the two kinds of projection results to obtain a predicted depth map, specifically:

For each pixel point p in the key frame i, according to the dense map D _cnn (p) predicted by CNN and the pose transformation Project the pixel point into the key frame i-1 closest to it, and the result of the projection is expressed as _p'cnn ;

Make another projection of the pixel point p in the key frame i, and map it to the key frame i-1 as p′ _sd , the projection is based on the result D _sp (p) of the semi-dense map and the absolute scale factor;

Select a small area near the projection points p′ _cnn and p′ _sd in the key frame i-1 respectively, and calculate the normalized _{cross-correlation coefficients NCC cnn and} The normalized cross-correlation coefficient NCC _sd between the region R(p) and R _sd (p′), if NCC _cnn is smaller than NCC _sd , it indicates that the depth prediction result of the semi-dense depth map is better than the result of CNN, choose D _sp ( p) as the final depth prediction value of pixel p, otherwise select R _cnn (p′), if some points only have CNN prediction results, use R _cnn (p′) as the final depth of pixel p.

In one embodiment of the present invention, Gaussian fusion is performed on the predicted depth map to obtain the final depth map, specifically:

The depth map obtained by the NCC score voting method is further processed, according to the context relationship between key frames, and combined with the uncertainty map of the key frame depth map for joint optimization, and the final depth map is obtained through joint optimization.

In one embodiment of the present invention, the dense map obtained by using the trained CNN network model to predict the depth of the current frame picture also includes:

Multiply the depth value of each pixel in the depth map by a scale factor

Among them, f _adapted is the focal length of the monocular camera used to obtain the training data online, B _adapted is the baseline of the binocular training picture, f _pre-train and B _pre-train are the focal length and the training picture used to train the original CNN network model, respectively. baseline.

In one embodiment of the present invention, the key frame is: define the whole image sequence or the first picture obtained by the camera in real time as a key frame, except for the first frame, a part of the following picture frames will also be defined as key frames, wherein The principle of defining a key frame is to monitor whether the translation and rotation between the current frame and the previous nearest key frame reach a preset threshold.

According to another aspect of the present invention, a real-time dense monocular SLAM system based on an online learning depth prediction network is also provided, including a direct method monocular SLAM module, an online adaptive CNN prediction module, an absolute scale regression module, and depth map fusion module, where:

The direct method monocular SLAM module is used to select key frames from the image sequence collected by the monocular vision sensor through rotation and translation motion, and optimize the camera pose of the key frame by minimizing the photometric error of high gradient points, and adopts triangulation The measurement method predicts the depth of high gradient points to obtain a semi-dense map of the current frame;

The online adaptive CNN prediction module is used to select an online training picture pair according to the key frame, and use block-by-block stochastic gradient descent method to train and update the CNN network model online according to the online training picture pair, and use the trained CNN network The model performs depth prediction on the current frame picture to obtain a dense map;

The absolute scale regression module is used to perform depth scale regression according to the semi-dense map of the current frame and the predicted dense map to obtain the absolute scale factor of the depth information of the current frame;

The depth map fusion module is used to project the predicted dense map to the previous key frame through pose transformation according to the camera pose, and project the semi-dense map to the previous key frame according to the absolute scale factor , using the NCC score voting method to select the predicted depth values of each pixel of the current frame according to the two projection results to obtain a predicted depth map, and performing Gaussian fusion on the predicted depth map to obtain a final depth map.

Generally speaking, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects: the present invention adopts monocular SLAM and adopts the direct method as the basis, and obtains the semi-dense map and camera of the scene in an optimized manner. Attitude; online adaptive CNN uses a weakly supervised depth prediction network, and performs online updates according to scene information, so that the network has a good effect in unknown scenes; depth scale regression can obtain scale information of depth values to improve 3D reconstruction accuracy; data fusion adopts regional voting and Gaussian fusion, which improves the accuracy of the results while ensuring the integrity rate.

Description of drawings

1 is a schematic diagram of the principle of a real-time dense monocular SLAM method based on an online learning depth prediction network in an embodiment of the present invention;

Fig. 2 is a schematic diagram of a triangulation model in an embodiment of the present invention;

Fig. 3 is the constraint relation of screening training pictures in the embodiment of the present invention; Wherein figure (a) is the image pair of the first kind of pixel correspondence, figure (b) is the image pair of the second kind of pixel correspondence;

Fig. 4 is a schematic diagram of scale coefficient adjustment in the embodiment of the present invention, wherein the upper part is the original network structure, and the lower part is the improvement of the network in the present invention;

Fig. 5 is a block-wise gradient descent method (block-wise SGD) schematic diagram in the embodiment of the present invention;

Fig. 6 is a scale regression and effect diagram in the embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a real-time dense monocular SLAM system based on an online learning depth prediction network in an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

The problem to be solved by the present invention is to realize a real-time monocular dense mapping SLAM system. The system adopts the combination of self-adaptive online CNN depth prediction network and monocular SLAM system based on direct method, which can not only significantly improve the performance of unknown scene The accuracy and robustness of depth prediction can also solve the problem of scale uncertainty of the monocular SLAM system.

In order to achieve the above purpose, the present invention adopts the method of combining CNN and SLAM, and proposes an algorithm with better accuracy and more robustness for the problems existing in monocular SLAM. The main innovations of the scheme include:

(1) An adaptive online CNN depth prediction network is adopted, which is the first time in the field to combine this type of network with a monocular SLAM system, which can greatly improve the accuracy of the system's depth prediction in unknown scenes;

(2) A "block-wise SGD" method and a selective update strategy are proposed, so that CNN can achieve better depth prediction results under limited training data conditions;

(3) An adaptive network-based absolute scale regression method is designed, which can greatly improve the accuracy of depth prediction and make the tracking and mapping of the whole system more accurate.

The system mainly consists of four components: direct method monocular SLAM, online adaptive CNN, depth-scale regression and data fusion. The block diagram of the method is shown in Figure 1. Monocular SLAM adopts the direct method, and based on the direct method, adopts an optimized method to obtain the semi-dense map and camera pose of the scene; the online adaptive CNN uses a weakly supervised depth prediction network, and performs online updates according to the scene information, so that The network has a good effect in unknown scenes; the depth scale regression can obtain the scale information of the depth value, which is used to improve the accuracy of 3D reconstruction; the data fusion adopts the method of regional voting and Gaussian fusion, and in the case of ensuring the integrity rate, Improved precision of results.

Specifically, the method includes the following processes:

(1) Direct method monocular SLAM: This part is based on the transformation of LSD-SLAM. By minimizing the photometric error of high gradient points, the camera pose of each frame is optimized, and the triangulation method is used to Predict the depth of high gradient points to obtain a semi-dense map;

Picture collection: This method is based on the monocular vision sensor. When collecting pictures, the monocular camera is required to have rotation and translation movements, and the translation range should be increased appropriately. There are two main reasons for this: first, if there are only static and pure rotation situations, it may cause the initialization failure of this part or the image tracking failure, which will cause the entire system to work abnormally; the second is to increase the translation appropriately. The amplitude helps the system to select suitable training pictures, so as to ensure the normal progress of the online training and updating CNN process.

Key frame definition: The monocular SLAM part defines the entire sequence or the first picture obtained by the camera in real time as a keyframe (key frame). In addition to the first frame, some of the subsequent picture frames will also be defined as key frames, where the key frame is defined The principle is to monitor whether the translation and rotation between the current frame and the previous most recent key frame have reached the preset threshold; the algorithm composition based on the key frame is the basis of the direct method monocular SLAM back-end optimization and also an important framework of the network part structure, and therefore require special introduction.

Camera pose tracking: There are six degrees of freedom in the movement of the camera in three-dimensional space, and the movement amount within Δt time can be represented by a six-dimensional array: ξ=[v ⁽¹⁾ ν ⁽²⁾ ν ⁽³⁾ ψ ⁽¹⁾ ψ ⁽²⁾ ψ ⁽³⁾ ] ^T . where [ν ⁽¹⁾ ν ⁽²⁾ ν ⁽³⁾ ] ^T represents the translation component of the motion of the rigid body along the three coordinate axes, and [ν ⁽¹⁾ ν ⁽²⁾ ν ⁽³⁾ ] ^T ∈ R ³ is the Euclidean space The vector in; [ψ ⁽¹⁾ ψ ⁽²⁾ ψ ⁽³⁾ ] ^T represents the rotation component of the rigid body motion along the three coordinate axes, and [ψ ⁽¹⁾ ψ ⁽²⁾ ψ ⁽³⁾ ] ^T ∈ SO(3 ) is a vector in the non-Euclidean three-dimensional rotation group SO(3). A vision-based camera is tracking the process of solving ξ through visual information. The monocular SLAM used in the present invention uses a direct method to track the camera pose, and projects all points with depth information in picture A to picture B to obtain a new picture B', by optimizing all points between B' and B The sum of the difference of the gray value of the position (photometric error) to get the position change of B relative to A. The direct method can better deal with changes in perspective, illumination changes, and sparse scene textures. It is a popular method at present. Therefore, this project uses the direct method to achieve camera pose tracking.

Specifically, the key idea of the direct method for camera pose tracking is to find an optimal camera pose between the current frame n and the nearest key frame k Minimize the photometric error between current frame n and key frame k. The existence of uniform regions may cause inaccurate pixel matching between frames due to different camera poses There may be similar photometric errors. In order to obtain highly reproducible tracking results and reduce the time overhead for optimization, the photometric error r is only calculated at the high gradient point {p} in the key frame k, as follows:

Among them, D(p) represents the depth value of the high-gradient pixel point p, π is the projection model that can project the 3D space point P ^c in the camera coordinate system to the 2D image plane pixel point p, and π is determined by the camera internal reference K.

Similarly, π ^-1 is a model of back projection, which can project the pixels of the 2D plane to the 3D space. An optimized camera pose It can be calculated by minimizing the photometric error r of all high gradient pixels, as follows:

Where w _p is the weight of the pixel p used to improve the robustness and minimize the influence of outliers.

The problem of (2) can be solved by a standard Gauss-Newton optimization algorithm. Of course, this kind of camera pose tracking through the above method will cause drift due to the accumulation of errors, but this drift can be eliminated by adding additional loopback detection. This project intends to use the loopback detection method based on the bag of words model to solve the cumulative error zone. Come the drift problem.

Semi-dense depth estimation: The thread used for mapping in the monocular direct method SLAM system estimates the depth value of high-gradient pixel points through the method of small baseline stereo comparison, which is the pixel matching method used in the triangulation method. Specifically, the model of feature matching and triangulation is shown in Figure 2 below. C and C' are the origin of the camera coordinate system of the key frame and reference frame respectively, X is the 3D point to calculate the depth, m and m' are the points respectively Projection of X on the camera projection planes of cameras C and C'.

Because the key frame and reference frame in monocular vision come from the same camera, their projection internal parameters are the same. If the rotation and translation [R,t] between the two camera coordinate systems have been obtained according to the visual method, then Has the following formula:

Among them, f _x , f _y , c _x , _cy , s are the internal parameters of the camera, R, t are a 3×3 and 3×1 matrix respectively, indicating the rotation of the camera C' coordinate system relative to the camera C coordinate system and translation, (x _c , y _c , z _c ) ^T , (x _c' , y _c' , z _c ') ^T represent the homogeneous coordinates of point X in the camera coordinate system C and C' respectively, (u, v) ^T , (u', v') ^T represent the pixel coordinates of point X on the projection planes of cameras C and C', respectively. Since the internal parameter matrix is a known value after camera calibration, [R,t] can be obtained from the previous positioning, so (m ₁₁ ...m ₃₄ ) are all known numbers, so the above formula can be simplified as:

Namely: A(x _c ,y _c ,z _c ) ^T ＝b, this system of equations contains 3 unknowns and 4 equations, it is an overdetermined system of equations, the least square solution is obtained make satisfied

Once a new keyframe k is created, its depth map D _pri and depth value prediction uncertainty U _pri are first initialized by adding the depth D _k-1 and uncertainty of the k-1th keyframe k U _k-1 is projected to the current keyframe, the specific method is as follows:

D _pri =D _k-1 -t _z (6)

Among them, t _z is the translation of the camera along the viewing axis, and σ ² represents the standard deviation of the initialization noise. The initialized depth map will be continuously repaired according to the subsequent picture frames. The repair process first retrieves the matching pixel points of each high-gradient pixel point p in the current frame along the bipolar line in the key frame k, where the bipolar line The search interval on is determined by the depth uncertainty of pixel p; once a matching pixel is found, the depth value of p can be calculated by triangulation. The present invention uses a function F to represent the whole process of pixel point matching and triangulation. Based on F, the observation result D _obs of the depth value obtained by the present invention can be expressed as follows:

Among them, I _k and I _cur represent the key frame k and the current picture frame respectively, Represents the camera motion between the key frame k and the current picture frame, and K represents the internal parameter matrix of the camera. Observations of depth (D _obs ) with uncertainties (U _obs ) are generated by noise that exists between I _k and I _cur pixel matching and camera motion in the estimation process. The repaired depth map and its corresponding uncertainty are actually a fusion of initial depth information and observed depth information, which conforms to the following distribution:

(2) Online adaptive training CNN, and using the CNN to predict key frames to obtain a dense map (densedepth map):

Online Adaptive CNN: First, the present invention uses a state-of-the-art weakly supervised method as a basis to estimate the depth of a single image. The network architecture of this weakly supervised method is mainly composed of two parts. The first part is the full convolutional layer (ConvLayers) based on ResNet-50; the second part is the pooling layer and the full connection layer located behind ResNet-50. (FCLayers) are replaced by a sequence of upsampling regions consisting of deconvolutional layers and short-connection layers. Training the entire CNN requires pairs of rectified stereo images for which both the baseline B _pre-train and the camera focal length f _pre-train are fixed. The output of the network is a disparity map. According to the disparity map, the reconstruction map of the source image can be generated. The photometric error between the source image and the reconstructed image plus the smoothing term constitutes the loss function of the entire network. In the experiment, a series of key frame {I ₁ ,…I _j } image translation {T _1,2 ,…,T _i-1,i ,…,T _j-1,j } and Rotation changes {R _1,2 ,…,R _i-1,i ,…,R _j-1,j }, the present invention takes this information as the true value, and learns to minimize the reprojection error between any two key frames The depth map of the 2D image.

Pre-training CNN: The entire online adaptive CNN part is based on the pre-trained CNN network model of the present invention. When pre-training the network model, the present invention uses 6510 pairs of pictures of the CMU dataset (the Wean Hall dataset) according to the traditional CNN training method In addition to the 35588 pairs of pictures recorded by myself in this laboratory, a total of 42098 pairs of pictures are used as the training set to train the network. The baseline of the training set pictures is 0.12 meters. The laboratory scene pictures were taken by ZED stereo camera, and random color, scale and mirror changes were used to enhance the data of the training set. All the images in the training set are processed and input to the network for iteration. The number of iterations is 40,000, and the learning rate is 0.0001. Finally, the desired pre-training model is obtained. The entire system will also learn and update online based on this model.

Under a single video scene sequence, each time CNN is trained, the model of the network is saved and updated, and a new model is used to generate a depth map. There are four main online adaptive CNN strategies:

1) Online learning image screening: the depth prediction network needs images taken by pairs of stereo cameras as training images, and these stereo images have a fixed baseline B _pre-train . In order to train and update the CNN network model in real time, the present invention collects paired monocular pictures according to the rules of binocular cameras to simulate stereoscopic pictures while the monocular camera is moving. The present invention adopts high standard requirements to collect reliable training pictures to reduce the over-fitting phenomenon of the CNN network model generated by the noise to the wrong samples. The present invention designs four main screening conditions: first, camera motion constraints. The displacement in the horizontal direction between two frames of pictures satisfies |t _x |>0.9*T, where T represents the baseline distance between two frames of pictures Second, the parallax constraint. For each pair of pictures, the optical flow method is used to calculate the average vertical disparity Dis _avg between the pictures. Only when Dis _avg is smaller than the threshold δ (taken as 5 in the experiment) will the pair of pictures be saved as candidate training pictures. The effect is shown in Figure 3. (a) and (b) are two pairs of pictures respectively. When the pixel relationship in each pair of pictures satisfies the relationship shown in (a), such image pairs will be screened as Candidate images for training are discarded when their relationship is shown in (b); third, diversity constraints. The screening of each pair of training pictures is uniquely corresponding to the key frame picture, that is to say, the same key frame can only generate a pair of training pictures at most; Fourth, the capacity of the training pool is constrained. Whenever the number of training picture pairs reaches the threshold V (4 in the experiment), the pictures in the training pool are sent to the network, the network is trained online, the network model obtained from the training is saved, and the training pool is emptied to continue training screening of data;

2) Camera parameter adjustment: The monocular camera focal length f _adapted for online acquisition of training data and the baseline B _adapted for binocular training pictures are likely to be used to train the original CNN network model. The focal length f _pre-train and baseline B of training pictures _Pre-train is very different. The relationship between camera parameters and scene depth values has been implicitly integrated into the network structure, so if images with different focal lengths are used for network testing, the absolute scale of the 3D reconstruction results obtained may be inaccurate. Therefore, the whole network needs to be adjusted to adapt to the changes of different camera parameters, but doing so will make the update speed of each online learning slower. In order to solve this problem, a new idea of adjusting the output depth map is proposed. The basic idea is shown in Figure 4. The basic idea is to multiply the depth value of each pixel in the depth map by a scale coefficient To ensure the accuracy of the depth map;

3) Block-by-block SGD method: Stochastic gradient descent (SGD) is the most comprehensive optimization algorithm for deep learning that is mainstream today. The main idea is that for the training data set, it is first divided into n batches, and each batch contains m samples. Each time the parameters of the network are updated, only one batch of data is used, not the entire training set.

The advantages are: when there is a lot of training data, using batch can reduce the pressure on the machine and converge faster; when the training set has a lot of redundancy (similar samples appear multiple times), the batch method converges faster.

The disadvantage is: it is easy to converge to the local optimum, not the global optimum.

Our proposed block-wise gradient descent (block-wise SGD) is an innovative improvement on top of stochastic gradient descent (SGD).

The present invention uses ResNet-50 to extract different levels of feature information in pictures, and these feature information will be encoded into the disparity map through a series of down-sampling operations. In order to reduce the risk of CNN overfitting due to the limitations of training pictures, the present invention proposes a new method of "block-wise SGD" (block-wise SGD), which divides the convolutional layer in ResNet-50 into 5 blocks, as shown in Figure 5, where each block is specifically represented as conv1, conv2_x, conv3_x, conv4_x, conv5_x. Conv1 consists of a single 7X7 fully convolutional layer; conv2_x consists of a 3X3 convolutional layer and 3 bottleneck building blocks (each bottleneck building block is 1X1 64, 3X3 64, 1X1 256) a total of 10 layers; conv3_x consists of 4 bottleneck building blocks (each bottleneck building block is 1X1 128, 3X3 128, 1X1 512) a total of 12 layers; conv4_x consists of 6 bottleneck building blocks (each bottleneck building block is 1X1 256, 3X3 256, 1X1 1024) 18-layer composition: conv5_x consists of 3 bottleneck building blocks (each bottleneck building block is 1X1 512, 3X3 512, 1X1 2048) with a total of 9 layers, and the five parts add up to form the 50-layer structure of ResNet-50. During each online learning and updating process, and each iteration k, only a part of the parameters W _i (i=1, 2, 3, 4, 5) is updated, and the remaining 4 parts of the network layer parameters are kept unchanged. In the next iteration, the parameters of the i-th block (i=(k+1)%5) are updated, and the parameters of other layers remain unchanged, thereby reducing the complexity of updating the network each time. The entire iteration of online learning and updating continues until the stopping condition is met (such as the limit on the number of iterations, or the training loss function reaches a preset threshold);

4) Selective update: Online learning and updating of the CNN network model are carried out whenever suitable training data is generated, which is likely to cause unnecessary computational overhead. As long as the current CNN network model can provide sufficiently accurate depth prediction results for the current scene, the current CNN network model will be used until the network model has to be adjusted. Based on this idea, the present invention designs a "system selective update" working mode, by calculating the training loss function of each batch of pictures input into the CNN network model, once the loss functions of all pictures in a batch of pictures are greater than The preset threshold L _high will start the process of online learning and updating. The process of online learning and updating will continue until the loss function of the training image drops below L _low , or the number of iterations reaches the preset threshold. This strategy not only greatly reduces the amount of calculation, but also meets the requirements for the accuracy of network depth prediction results.

(3) Depth scale regression: The camera pose with accurate scale information is of great significance for selecting appropriate training pictures, which directly affects the output of the network. Since the monocular SLAM system cannot obtain an absolute scale, the present invention proposes a method of "accurate scale regression based on adaptive CNN". We draw the relationship diagram between D _sd (p)-D _gt (p), as shown in Fig. 6, where the black line in (b) is the camera pose of GroundTruth (true value) in the scene, and the blue line is The camera pose obtained by monocular SLAM, and the red line is the result of using the RANSAC algorithm to regress to obtain the scale and apply it to the camera pose; it is found that D _sd (p) (the depth of the high gradient point p obtained by monocular SLAM) and D _gt The ratio of (p) (true depth value of pixel point p) represents the absolute scale information of point p. Based on this, the present invention proposes the idea of using the depth relationship of all high gradient points to regress the absolute scale information, but the real depth information is unknown in practical application, so the present invention uses the prediction results of CNN to perform scale regression. Considering that there are some adverse effects of outliers in the predicted depth of CNN, we experimented with the RANSAC algorithm and the least squares algorithm for scale regression respectively. The experimental results are shown in the green and red lines in Figure 6(a), which proves that RANSAC The algorithm can achieve a more accurate fitting effect, so the embodiment of the present invention adopts the RANSAC method. After we use this method to calculate the absolute scale of the depth information, the scale information of the attitude can be obtained according to the mapping relationship, which will in turn improve the tracking accuracy of the monocular SLAM system, as shown in Figure 6(b). Two scenes in the TUM dataset were tested. The blue part is the attitude of monocular SLAM tracking, the black part is the real depth information, and the red part is the result of adding scale information to monocular SLAM tracking, showing this method. It can better fit the tracking scale.

(4) Data fusion: For each key frame, we can get two depth maps, one is the optimized result D _sd of monocular SLAM, and the other is the prediction result D _cnn of CNN. The present invention designs a method of "combining NCC score voting and Gaussian fusion" to achieve the best combination effect. The process consists of two parts, the first being the NCC scoring vote. NCC (Normalized Cross Correlation) is the abbreviation of Normalized Cross Correlation, which is used to calculate the correlation between two image areas A and B. The calculation formula is For each pixel point p in the key frame i, according to the depth map D _cnn (p) predicted by CNN and the pose transformation Project the pixel point into the nearest key frame i-1, and the result of the projection is expressed as p'_cnn; similarly, another projection is made on the pixel point p in the key frame i, and it is mapped to Denote p' _sd in keyframe i-1, but the projection is based on the result D _sp (p) of the semi-dense map and the absolute scale factor. Select a small area near the projection points p′ _cnn and p′ _sd in the key frame i-1 respectively, and calculate the normalized _{cross-correlation coefficients NCC cnn and} The normalized cross-correlation coefficient NCC _sd between the regions R(p) and R _sd (p′). If NCC _cnn is smaller than NCC _sd , it indicates that the depth prediction result of the semi-dense depth map is better than that of CNN, and D _sp (p) is selected as the final depth prediction value of pixel p, otherwise, R _cnn (p′) is selected. If some points only have CNN prediction results, we use R _cnn (p′) as the final depth of pixel p. The second part is Gaussian fusion. For the further processing of the depth map obtained in the previous step, joint optimization is performed according to the contextual relationship between key frames and the uncertainty map of the key frame depth map. This is the so-called Gaussian fusion. The final depth map is obtained through joint optimization. In the experiment, we tested the scene sequences of multiple datasets and achieved good results.

Due to the use of CNN, our monocular dense SLAM system requires GPU acceleration to achieve good real-time performance. On the TUM dataset and the ICL-NUIM dataset, our algorithm is tested, compared with the state-of-the-art monocular SLAM system LSD-SLAM based on the current direct method, the absolute trajectory error of our attitude tracking accuracy From 0.622m to 0.231m. The completeness rate of the key frame depth map (the proportion of points with an error within 10% in the depth map to the whole map) has increased from 0.61% to 26.47%; From 21.05% to 26.47%. In addition, the operating speed of the entire system can also achieve real-time effects.

Further, as shown in Figure 7, the present invention also provides a real-time dense monocular SLAM system based on online learning depth prediction network, including direct method monocular SLAM module 1, online adaptive CNN prediction module 2, absolute scale regression Module 3 and depth map fusion module 4, in which:

The direct method monocular SLAM module 1 is used to select key frames from the image sequence collected by the monocular vision sensor through rotation and translation motion, and obtain the camera pose of the key frame by minimizing the photometric error optimization of high gradient points, and adopt Triangulation predicts the depth of high gradient points to obtain a semi-dense map of the current frame;

The online adaptive CNN prediction module 2 is used to select an online training picture pair according to the key frame, and use block-by-block stochastic gradient descent method to train and update the CNN network model online according to the online training picture pair, and use the trained CNN The network model performs depth prediction on the current frame picture to obtain a dense map;

The absolute scale regression module 3 is configured to perform depth scale regression according to the semi-dense map of the current frame and the predicted dense map to obtain an absolute scale factor of the depth information of the current frame;

The depth map fusion module 4 is used to project the predicted dense map to the previous key frame through pose transformation according to the camera pose, and project the semi-dense map to the previous key frame according to the absolute scale factor In this method, the NCC score voting method is used to select the predicted depth values of each pixel of the current frame according to the two projection results to obtain a predicted depth map, and Gaussian fusion is performed on the predicted depth map to obtain a final depth map.

It is easy for those skilled in the art to understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (10)

1. A real-time dense monocular SLAM method based on online learning depth prediction network, characterized in that, comprising the steps: (1) Select the key frame from the image sequence collected by the monocular vision sensor through the rotation and translation motion, optimize the camera pose of the key frame by minimizing the photometric error of the high gradient point, and use the triangulation method to predict the depth of the high gradient point Get the semi-dense map of the current frame; (2) Select an online training picture pair according to the key frame, use the block-by-block stochastic gradient descent method to train and update the CNN network model online according to the online training picture pair, and use the trained CNN network model to perform depth prediction on the current frame picture get a dense map; (3) performing depth scale regression according to the semi-dense map of the current frame and the predicted dense map to obtain the absolute scale factor of the depth information of the current frame; (4) Project the predicted dense map to the previous key frame through pose transformation according to the camera pose, and project the semi-dense map to the previous key frame according to the absolute scale factor, using the NCC score voting method Selecting the predicted depth values of each pixel in the current frame according to the two projection results to obtain a predicted depth map, and performing Gaussian fusion on the predicted depth map to obtain a final depth map. 2. The real-time dense monocular SLAM method based on online learning depth prediction network as claimed in claim 1, wherein the online training picture is selected according to the key frame, specifically: adopting the following constraints before and after the key frame In the frame picture, the selected picture frame and the key frame form a picture pair: First, the camera motion constraint: the displacement in the horizontal direction between the two frames of pictures satisfies |t _x |>0.9*T, where T represents the baseline distance between the two frames of pictures; Second, disparity constraints: For each pair of pictures, the optical flow method is used to calculate the average disparity Dis _avg in the vertical direction between the pictures, and only when Dis _avg is smaller than the preset threshold δ, the pair of pictures will be saved as candidate training pictures; Third, diversity constraints: the same key frame can only generate a pair of training pictures; Fourth, training pool capacity constraints: Whenever the number of training picture pairs reaches the set threshold V, the pictures in the training pool are sent to the network, the network is trained online, the network model obtained from the training is saved, and the training time is cleared. The pool continues to filter the training data. 3. the real-time dense monocular SLAM method based on online learning depth prediction network as claimed in claim 1 or 2, is characterized in that, adopt block-by-block stochastic gradient descent method to carry out online training update CNN network model according to described online training picture, Specifically: The convolutional layer in ResNet-50 is divided into 5 blocks, each of which is specifically represented as conv1, conv2_x, conv3_x, conv4_x, conv5_x; conv1 consists of a single 7X7 full convolutional layer; conv2_x consists of a 3X3 The convolution layer and 3 bottleneck building blocks are composed of 10 layers; conv3_x is composed of 4 bottleneck building blocks with 12 layers; conv4_x is composed of 6 bottleneck building blocks with 18 layers; conv5_x is composed of 3 bottleneck building blocks with 9 layers , the five parts together constitute the 50-layer structure of ResNet-50; In each online learning and updating process, each iteration k, only update a part of the parameters W _i (i=1,2,3,4,5), keep the remaining 4 parts of the network layer parameters unchanged, In the next iteration, update the i-th block parameter, where i=(k+1)%5; other layer parameters remain unchanged, and the entire iteration of online learning and updating continues until the preset stop condition is met. 4. the real-time dense monocular SLAM method based on online learning depth prediction network as claimed in claim 1 or 2, is characterized in that, described online training update CNN network model is selective update, specifically: Calculate the training loss function of each batch of pictures input into the CNN network model. Once the loss function of all pictures in a batch of pictures is greater than the preset threshold L _high , the process of online learning and updating will be started. Online learning and updating The process will continue until the loss function of the training image drops below the threshold L _low , or the number of iterations reaches the preset threshold. 5. The real-time dense monocular SLAM method based on online learning depth prediction network according to claim 1 or 2, wherein the depth scale regression method is: RANSAC algorithm or least squares algorithm. 6. the real-time dense monocular SLAM method based on online learning depth prediction network as claimed in claim 1 or 2, is characterized in that, described prediction dense map is projected in last key frame by pose transformation, and Project the semi-dense map into the previous key frame according to the absolute scale factor, and use the NCC score voting method to select the predicted depth values of each pixel in the current frame according to the two projection results to obtain a predicted depth map, specifically: : For each pixel point p in the key frame i, according to the dense map D _cnn (p) predicted by CNN and the pose transformation Project the pixel point into the key frame i-1 closest to it, and the result of the projection is expressed as _p'cnn ; Make another projection of the pixel point p in the key frame i, and map it to the key frame i-1 as p′ _sd , the projection is based on the result D _sp (p) of the semi-dense map and the absolute scale factor; Select a small area near the projection points p′ _cnn and p′ _sd in the key frame i-1 respectively, and calculate the normalized _{cross-correlation coefficients NCC cnn and} The normalized cross-correlation coefficient NCC _sd between the region R(p) and R _sd (p′), if NCC _cnn is smaller than NCC _sd , it indicates that the depth prediction result of the semi-dense depth map is better than the result of CNN, choose D _sp ( p) as the final depth prediction value of pixel p, otherwise select R _cnn (p′), if some points only have CNN prediction results, use R _cnn (p′) as the final depth of pixel p. 7. The real-time dense monocular SLAM method based on the online learning depth prediction network as claimed in claim 1 or 2, wherein the Gaussian fusion is carried out to the predicted depth map to obtain the final depth map, specifically: The depth map obtained by the NCC score voting method is further processed, according to the context relationship between key frames, and combined with the uncertainty map of the key frame depth map for joint optimization, and the final depth map is obtained through joint optimization. 8. the real-time dense monocular SLAM method based on online learning depth prediction network as claimed in claim 1 or 2, it is characterized in that, utilize CNN network model after training to carry out depth prediction to current frame picture and obtain dense map and also include: Multiply the depth value of each pixel in the depth map by a scale factor Among them, f _adapted is the focal length of the monocular camera used to obtain the training data online, B _adapted is the baseline of the binocular training picture, f _pre-train and B _pre-train are the focal length and the training picture used to train the original CNN network model, respectively. baseline. 9. the real-time dense monocular SLAM method based on online learning depth prediction network as claimed in claim 1 or 2, is characterized in that, described key frame is: Define the entire image sequence or the first picture obtained by the camera in real time as a key frame. In addition to the first frame, a part of the following picture frames will also be defined as key frames. The principle of defining key frames is to monitor the current frame and the previous one. Whether the translation and rotation between keyframes reached a preset threshold. 10. A real-time dense monocular SLAM system based on online learning depth prediction network, characterized in that it includes a direct method monocular SLAM module, an online adaptive CNN prediction module, an absolute scale regression module and a depth map fusion module, wherein: The direct method monocular SLAM module is used to select key frames from the image sequence collected by the monocular vision sensor through rotation and translation motion, and optimize the camera pose of the key frame by minimizing the photometric error of high gradient points, and adopts triangulation The measurement method predicts the depth of high gradient points to obtain a semi-dense map of the current frame; The online adaptive CNN prediction module is used to select an online training picture pair according to the key frame, and use block-by-block stochastic gradient descent method to train and update the CNN network model online according to the online training picture pair, and use the trained CNN network The model performs depth prediction on the current frame picture to obtain a dense map; The absolute scale regression module is used to perform depth scale regression according to the semi-dense map of the current frame and the predicted dense map to obtain the absolute scale factor of the depth information of the current frame; The depth map fusion module is used to project the predicted dense map to the previous key frame through pose transformation according to the camera pose, and project the semi-dense map to the previous key frame according to the absolute scale factor , using the NCC score voting method to select the predicted depth values of each pixel of the current frame according to the two projection results to obtain a predicted depth map, and performing Gaussian fusion on the predicted depth map to obtain a final depth map.