CN110717917B - CNN-based semantic segmentation depth prediction method and device - Google Patents

Abstract

The present application discloses a depth prediction method and device based on CNN semantic segmentation, and relates to the field of semantic segmentation. The method includes: acquiring a current frame from a data set, calculating an ORB feature and a map point cloud of the current frame; according to the ORB feature and map point cloud of the current frame, determining that the current frame is selected as a key when a specified condition is satisfied frame; the CNN convolutional neural network is used to associate the key frame with the depth map to make depth prediction, and the CNN is used to label the depth map with semantic labels to achieve semantic association. The device includes: a calculation module, a selection module and a prediction module. This application no longer inputs each image into the model, which solves the memory problem, and adjusts the depth regression through CNN, which optimizes the problem of inaccurate absolute scale during 3D reconstruction.

Description

Depth prediction method and device for semantic segmentation based on CNN

technical field

The present application relates to the field of semantic segmentation, and in particular, to a method and device for depth prediction based on CNN semantic segmentation.

Background technique

Semantic SLAM (Simultaneous Localization And Mapping) is applied in image semantic segmentation, semantic map construction and other directions, expanding the research content of traditional SLAM problems. At present, there have been some researches that integrate semantic information into SLAM. For example, using the geometric consistency between the images obtained in the SLAM system to promote image semantic segmentation, or using the results of semantic segmentation/mapping to promote SLAM localization/loop closure, etc.

Existing semantic SLAM models such as MaskFusion are essentially RGBD-SLAM+ semantic segmentation mask-rcnn, that is, the fusion of geometric SLAM model and deep learning SLAM model, which mainly solves the problem of understanding the environment from object to plane. While accurately segmenting moving targets, it can Identify, detect, track and reconstruct objects. For each frame of new data, the algorithm includes the following processes: 1) Tracking obtains independent model objects through segmentation, and the 6DoF (degree of freedom) of each object is determined by minimizing an activation function. Among them, only those dynamic models are tracked. 2) Compare two methods of determining whether an object is in motion: based on motion inconsistency and Treating objects which are being touched by a person asdynamic. 3) Segmentation is performed using Mask RCNN and a segmentation algorithm based on depth and surface normal discontinuities. 4) Fusion is performed by combining the geometric structure of the object with the label.

However, the semantic SLAM model based on deep learning does not have the concept of key frames, and is calculated one by one. The geometry-based monocular SLAM cannot solve the depth problem and is not robust enough in terms of geometric consistency.

SUMMARY OF THE INVENTION

The purpose of the present application is to overcome the above-mentioned problems or at least partially solve or alleviate the above-mentioned problems.

According to an aspect of the present application, there is provided a depth prediction method based on CNN semantic segmentation, including:

Obtain the current frame from the data set, and calculate the ORB feature and map point cloud of the current frame;

According to the ORB feature of the current frame and the map point cloud, determine that the current frame is selected as a key frame when the specified condition is satisfied;

A CNN (Convolutional Neural Networks, convolutional neural network) is used to associate the key frame with the depth map to make depth prediction, and the CNN is used to label the depth map with semantic labels to achieve semantic association.

Optionally, according to the ORB feature of the current frame and the map point cloud, it is determined that the current frame is selected as a key frame when a specified condition is satisfied, including:

When the local map construction is in an idle state or the current frame has exceeded the specified number of M frames from the previous key frame, the current frame is selected as the key frame; or,

If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified number N, then the current frame is selected as the key frame; or,

If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified ratio of the map cloud points, the current frame is selected as a key frame.

Optionally, a CNN convolutional neural network is used to associate the key frame with the depth map to make depth prediction, including:

CNN is used to estimate the depth of the key frame to generate a depth map, and the pixel direction confidence of each depth prediction is measured to construct a corresponding uncertainty map;

The generated depth map and uncertainty map of the key frame are fused with the depth map and uncertainty map of the nearest key frame, and the reconstructed 3D scene is obtained after updating the point cloud.

Optionally, using the CNN to mark semantic labels for the depth map to achieve semantic association, including:

A CNN that extends the residual network to a fully convolutional network is used to predict semantic labels, and label them to achieve semantic associations on the depth map.

Optionally, the computation of the CNN employs backpropagation and SGD stochastic gradient descent to minimize the softmax layer and the cross-entropy loss function.

According to another aspect of the present application, a depth prediction device based on CNN semantic segmentation is provided, including:

a computing module, configured to obtain the current frame from the data set, and calculate the ORB feature and map point cloud of the current frame;

A selection module, which is configured to select the current frame as a key frame when it is determined that the specified condition is satisfied according to the ORB feature of the current frame and the map point cloud;

A prediction module, which is configured to use a CNN convolutional neural network to associate the key frame with the depth map to perform depth prediction, and use the CNN to label the depth map with semantic labels to achieve semantic association.

Optionally, the selection module is specifically configured to:

When the local map construction is in an idle state or the current frame has exceeded the specified number of M frames from the previous key frame, the current frame is selected as the key frame; or,

If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified number N, then the current frame is selected as the key frame; or,

If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified ratio of the map cloud points, the current frame is selected as a key frame.

Optionally, the prediction module is specifically configured to:

CNN is used to estimate the depth of the key frame to generate a depth map, and the pixel direction confidence of each depth prediction is measured to construct a corresponding uncertainty map;

The generated depth map and uncertainty map of the key frame are fused with the depth map and uncertainty map of the nearest key frame, and the reconstructed 3D scene is obtained after updating the point cloud.

Optionally, the prediction module is specifically configured to:

A CNN that extends the residual network to a fully convolutional network is used to predict semantic labels, and label them to achieve semantic associations on the depth map.

Optionally, the computation of the CNN employs backpropagation and SGD stochastic gradient descent to minimize the softmax layer and the cross-entropy loss function.

According to yet another aspect of the present application, there is provided a computing device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor executes the computer program implement the method described above.

According to yet another aspect of the present application, there is provided a computer-readable storage medium, preferably a non-volatile readable storage medium, in which a computer program is stored, the computer program, when executed by a processor, realizes the above-mentioned Methods.

According to yet another aspect of the present application, there is provided a computer program product comprising computer readable code which, when executed by a computer device, causes the computer device to perform the above method.

The technical solution provided by the present application is to obtain the current frame from the data set, calculate the ORB feature and map point cloud of the current frame, and determine the current frame when the specified condition is satisfied. The frame is selected as the key frame, and the CNN convolutional neural network is used to associate the key frame with the depth map to make depth prediction, and the CNN is used to label the depth map with semantic labels to achieve semantic association. This is based on ORB features. The method of selecting key frames eliminates the need to input each image into the model, which solves the memory problem. Moreover, by adjusting the depth regression through CNN, the problem of inaccurate absolute scale during 3D reconstruction is optimized.

The above and other objects, advantages and features of the present application will be more apparent to those skilled in the art from the following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings.

Description of drawings

Hereinafter, some specific embodiments of the present application will be described in detail by way of example and not limitation with reference to the accompanying drawings. The same reference numbers in the figures designate the same or similar parts or parts. It will be understood by those skilled in the art that the drawings are not necessarily to scale. In the attached picture:

1 is a flowchart of a method for depth prediction based on CNN semantic segmentation according to an embodiment of the present application;

2 is a flowchart of a method for depth prediction based on CNN semantic segmentation according to another embodiment of the present application;

3 is a structural diagram of a depth prediction device based on CNN semantic segmentation according to another embodiment of the present application;

4 is a structural diagram of a computing device according to another embodiment of the present application;

FIG. 5 is a structural diagram of a computer-readable storage medium according to another embodiment of the present application.

Detailed ways

The experimental data set selected in this application is the KITTI data set (co-founded by Karlsruhe Institute of Technology in Germany and Toyota American Institute of Technology), which is currently the largest international computer vision algorithm evaluation data set in autonomous driving scenarios. The acquisition platform of the KITTI dataset includes: 2 grayscale cameras, 2 color cameras, a Velodyne 3D lidar, 4 optical lenses, and a GPS navigation system. The entire dataset consists of 389 pairs of stereo images and optical flow maps, 39.2 kilometers of visual ranging sequences, and more than 200,000 images of 3D annotated objects. Each image includes up to 15 vehicles and 30 pedestrians, and also contains varying degrees of occlude.

FIG. 1 is a flowchart of a depth prediction method based on CNN semantic segmentation according to an embodiment of the present application. Referring to Figure 1, the method includes:

101: Obtain the current frame from the dataset, and calculate the ORB feature and map point cloud of the current frame;

102: According to the ORB feature of the current frame and the map point cloud, determine that the current frame is selected as a key frame when the specified condition is satisfied;

103: Use the CNN convolutional neural network to associate key frames with the depth map to make depth prediction, and use CNN to label the depth map with semantic labels to achieve semantic association.

In this embodiment, optionally, according to the ORB feature of the current frame and the map point cloud, it is determined that the current frame is selected as a key frame when the specified condition is satisfied, including:

When the local map construction is in an idle state or the current frame is more than the specified number of M frames from the previous key frame, the current frame is selected as the key frame; or,

If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified number N, the current frame is selected as the key frame; or,

If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified ratio of the map cloud points, the current frame is selected as the key frame.

In this embodiment, optionally, a CNN convolutional neural network is used to associate the key frame with the depth map to make depth prediction, including:

CNN is used to estimate the depth of key frames to generate a depth map, and the pixel-wise confidence of each depth prediction is measured to construct a corresponding uncertainty map;

The depth map and uncertainty map of the generated key frame are fused with the depth map and uncertainty map of the nearest key frame, and the reconstructed 3D scene is obtained after updating the point cloud.

In this embodiment, optionally, a CNN is used to mark semantic labels for the depth map to achieve semantic association, including:

A CNN that extends the residual network to a fully convolutional network is used to predict semantic labels, and annotate them on the depth map to achieve semantic association.

In this embodiment, optionally, backpropagation and SGD stochastic gradient descent are used in the calculation of CNN to minimize the softmax layer and the cross-entropy loss function.

The above method provided in this embodiment obtains the current frame from the data set, calculates the ORB feature and map point cloud of the current frame, and determines that the ORB feature and map point cloud of the current frame are satisfied when the specified condition is satisfied. The current frame is selected as the key frame, and the CNN convolutional neural network is used to associate the key frame with the depth map to make depth prediction, and the CNN is used to label the depth map with semantic labels to achieve semantic association. This ORB-based approach The method of feature selection key frame, no longer input each picture into the model, solve the memory problem, and adjust the depth regression through CNN, optimize the problem of inaccurate absolute scale during 3D reconstruction.

FIG. 2 is a flowchart of a depth prediction method based on CNN semantic segmentation according to another embodiment of the present application. Referring to Figure 2, the method includes:

201: Obtain the current frame from the dataset, and calculate the ORB feature and map point cloud of the current frame;

In this embodiment, the ORB is a form of feature composition, and the ORB feature is composed of "key points" and "descriptors".

202: When the local map construction is in an idle state or the current frame has exceeded the specified number of M frames from the previous key frame, the current frame is selected as the key frame;

In this embodiment, the fact that the local map construction is in an idle state means that a certain part of the local map has missing values. Among them, the specified number of frames M can be set as required, such as set to 20 frames and so on.

203: If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified number N, select the current frame as a key frame;

Among them, the specified number N can be set as required, such as set to 50 and so on.

204: If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified ratio of the map cloud points, select the current frame as a key frame;

The specific value of the specified ratio can also be set as required, such as set to 90% and so on.

205: CNN is used to estimate the depth of the key frame to generate a depth map, and the pixel direction confidence of each depth prediction is measured to construct a corresponding uncertainty map;

In this embodiment, when the feature matching is unsuccessful (that is, the tracking fails), the global relocation needs to be started. Preferably, after the distance from the last global relocation, if the feature matching is unsuccessful and exceeds 20 frames of images, restart the relocation again. position.

206: Integrate the depth map and uncertainty map of the generated key frame with the depth map and uncertainty map of the nearest key frame, and update the point cloud to obtain a reconstructed 3D scene;

207: Using a CNN that extends the residual network to a fully convolutional network, predicts semantic labels, and annotates them on the depth map to achieve semantic association.

In this embodiment, optionally, back-propagation and SGD stochastic gradient descent are used in the calculation of the above-mentioned CNN to minimize the softmax layer and the cross-entropy loss function.

In this embodiment, the key frame is selected to reduce the number of frames to be optimized, and the key frame can represent the nearby frames, which can reduce the amount of calculation. Therefore, after the key frame is selected, the CNN-based SLAM model mainly targets the key frame. frame for data processing.

The above method provided in this embodiment obtains the current frame from the data set, calculates the ORB feature and map point cloud of the current frame, and determines that the ORB feature and map point cloud of the current frame are satisfied when the specified condition is satisfied. The current frame is selected as the key frame, and the CNN convolutional neural network is used to associate the key frame with the depth map to make depth prediction, and the CNN is used to label the depth map with semantic labels to achieve semantic association. This ORB-based approach The method of feature selection key frame, no longer input each picture into the model, solve the memory problem, and adjust the depth regression through CNN, optimize the problem of inaccurate absolute scale during 3D reconstruction.

FIG. 3 is a structural diagram of a depth prediction apparatus based on CNN semantic segmentation according to another embodiment of the present application. Referring to Figure 3, the device includes:

A calculation module 301, which is configured to obtain the current frame from the data set, and calculate the ORB feature and the map point cloud of the current frame;

Selection module 302, which is configured to select the current frame as a key frame when it is determined that the specified condition is satisfied according to the ORB feature of the current frame and the map point cloud;

The prediction module 303 is configured to use a CNN convolutional neural network to associate the key frames with the depth map to make depth prediction, and use CNN to label the depth map with semantic labels to achieve semantic association.

In this embodiment, optionally, the selection module is specifically configured as:

When the local map construction is in an idle state or the current frame is more than the specified number of M frames from the previous key frame, the current frame is selected as the key frame; or,

If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified number N, the current frame is selected as the key frame; or,

If the ORB feature points in the current frame that are successfully matched with the map point cloud are less than the specified ratio of the map cloud points, the current frame is selected as the key frame.

In this embodiment, optionally, the prediction module is specifically configured to:

CNN is used to estimate the depth of key frames to generate a depth map, and the pixel-wise confidence of each depth prediction is measured to construct a corresponding uncertainty map;

The depth map and uncertainty map of the generated key frame are fused with the depth map and uncertainty map of the nearest key frame, and the reconstructed 3D scene is obtained after updating the point cloud.

In this embodiment, optionally, the prediction module is specifically configured to:

A CNN that extends the residual network to a fully convolutional network is used to predict semantic labels, and annotate them on the depth map to achieve semantic association.

In this embodiment, optionally, backpropagation and SGD stochastic gradient descent are used in the calculation of the above-mentioned CNN to minimize the softmax layer and the cross-entropy loss function.

The foregoing apparatus provided in this embodiment can execute the method provided by any of the foregoing method embodiments. For details, refer to the description in the method embodiment, which is not repeated here.

The above-mentioned device provided in this embodiment obtains the current frame from the data set, calculates the ORB feature and map point cloud of the current frame, and determines that the ORB feature and map point cloud of the current frame are satisfied when the specified condition is satisfied. The current frame is selected as the key frame, and the CNN convolutional neural network is used to associate the key frame with the depth map to make depth prediction, and the CNN is used to label the depth map with semantic labels to achieve semantic association. This ORB-based approach The method of feature selection key frame, no longer input each picture into the model, solve the memory problem, and adjust the depth regression through CNN, optimize the problem of inaccurate absolute scale during 3D reconstruction.

The above and other objects, advantages and features of the present application will be more apparent to those skilled in the art from the following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings.

The embodiment of the present application also provides a computing device, referring to FIG. 4 , the computing device includes a memory 1120, a processor 1110, and a computer program stored in the memory 1120 and executable by the processor 1110, the computer program Space 1130 stored in the memory 1120 for program code which, when executed by the processor 1110, implements for performing any one of the method steps 1131 according to the invention.

Embodiments of the present application also provide a computer-readable storage medium. 5, the computer-readable storage medium comprises a storage unit for program codes provided with a program 1131' for performing the method steps according to the invention, the program being executed by a processor.

Embodiments of the present application also provide a computer program product including instructions. The computer program product, when run on a computer, causes the computer to perform the method steps according to the invention.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer loads and executes the computer program instructions, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, or the like that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, Solid State Disk (SSD)), among others.

Professionals should be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. Interchangeability, the above description has generally described the components and steps of each example in terms of function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those of ordinary skill in the art can understand that all or part of the steps in the method of implementing the above embodiments can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is non-transitory ( English: non-transitory) media, such as random access memory, read only memory, flash memory, hard disk, solid state disk, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), optical disc (English: optical disc) and any combination thereof.

The above are only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited to this. Substitutions should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (2)

1. A CNN semantic based depth prediction method for segmentation comprises the following steps:

acquiring a current frame from a data set, and calculating ORB characteristics and map point clouds of the current frame; the ORB features are composed of key points and descriptors;

according to the ORB characteristics of the current frame and the map point cloud, when the specified conditions are met, the current frame is selected as a key frame;

estimating the depth of the key frame by adopting a CNN convolutional neural network to generate a depth map, measuring the confidence of each depth predicted pixel direction, and constructing a corresponding uncertainty map; fusing the generated depth map and uncertainty map of the key frame with the depth map and uncertainty map of the nearest key frame, and updating the point cloud to obtain a reconstructed 3D scene;

adopting a CNN (convolutional neural network) which expands a residual network to a full convolutional network, predicting a semantic label, and marking the semantic label on the depth map to realize semantic association; wherein the CNN calculation employs back propagation and SGD random gradient descent to minimize the softmax layer and cross entropy loss function;

according to the ORB characteristics of the current frame and the map point cloud, when determining that specified conditions are met, selecting the current frame as a key frame, wherein the method comprises the following steps:

when the local map construction is in an idle state or the distance between the current frame and the previous key frame exceeds the specified frame number M, selecting the current frame as the key frame; or the like, or, alternatively,

if the number of ORB characteristic points successfully matched with the map point cloud in the current frame is less than the specified number N, selecting the current frame as a key frame; or the like, or, alternatively,

and if the ORB characteristic points successfully matched with the map point cloud in the current frame are less than the specified proportion of the map point cloud, selecting the current frame as a key frame.

2. A CNN semantic based depth prediction device for segmentation comprises:

a calculation module configured to obtain a current frame from a dataset, calculate ORB features and a map point cloud of the current frame; wherein, the ORB characteristics are composed of key points and descriptors;

the selecting module is configured to select the current frame as a key frame when determining that a specified condition is met according to the ORB characteristics of the current frame and the map point cloud;

a prediction module configured to estimate depths of the key frames using a CNN convolutional neural network to generate a depth map, and measure a pixel direction confidence of each depth prediction to construct a corresponding uncertainty map; fusing the generated depth map and uncertainty map of the key frame with the depth map and uncertainty map of the nearest key frame, and updating the point cloud to obtain a reconstructed 3D scene;

adopting a CNN (compressed natural network) for expanding a residual error network to a full convolution network, predicting semantic labels, and labeling the semantic labels on the depth map to realize semantic association; wherein the CNN calculation employs back propagation and SGD random gradient descent to minimize the softmax layer and cross entropy loss function;

the selection module is specifically configured to:

when the local map construction is in an idle state or the distance between the current frame and the previous key frame exceeds the specified frame number M, selecting the current frame as the key frame; or the like, or, alternatively,

if the number of ORB characteristic points successfully matched with the map point cloud in the current frame is less than the specified number N, selecting the current frame as a key frame; or the like, or, alternatively,

and if the ORB characteristic points successfully matched with the map point cloud in the current frame are less than the specified proportion of the map point cloud, selecting the current frame as a key frame.

Images