CN116342800B - Semantic three-dimensional reconstruction method and system for multi-mode pose optimization - Google Patents

Abstract

The application belongs to the technical field of semantic three-dimensional reconstruction, and particularly relates to a semantic three-dimensional reconstruction method and a semantic three-dimensional reconstruction system for multi-mode pose optimization, wherein the method comprises the following steps: collecting RGB images and depth images of a target area; performing semantic segmentation on the RGB image to obtain a final mask image of the RGB image; acquiring the global pose of the RGB image by utilizing the RGB image, the depth image and the final mask map; and acquiring a global semantic three-dimensional point cloud according to the global pose of the depth image and the RGB image. The technical scheme provided by the application has low computational force requirements and wide applicability, and improves the real-time performance and efficiency of semantic three-dimensional reconstruction.

Description

A semantic 3D reconstruction method and system for multi-modal pose optimization

Technical field

The invention belongs to the technical field of semantic three-dimensional reconstruction, and specifically relates to a semantic three-dimensional reconstruction method and system for multi-modal pose optimization.

Background technique

Semantic 3D reconstruction is to semantically annotate the 3D reconstruction model and segment structures with different categories of attributes from the overall model to facilitate the use of downstream tasks. One of the important factors that determines the quality of semantic 3D reconstruction is the calculation of camera pose, which is related to the quality of the reconstruction model and the distribution of semantic information. The positioning in the current algorithm uses purely geometric features, which requires the design of a complex screening mechanism to determine matching points. Semantic 3D reconstruction can be divided into three parts: semantic segmentation, 3D reconstruction, and semantic mapping. 3D reconstruction includes pose calculation and mapping.

Each part of the existing algorithm is relatively independent and performs data communication in a loosely coupled manner. For example, semantic 3D reconstruction mostly uses the GPU-based dense ICP algorithm to calculate the camera pose. At the same time, semantic segmentation also requires GPU resources, which has to be improved in terms of operating efficiency. Without compromise, the entire framework is too heavy and the scale of global optimization is large. Therefore, the semantic segmentation and reconstruction methods used in current semantic 3D reconstruction algorithms are not good in real-time performance, require high computing power, and impose a large burden on the overall operation of the algorithm, requiring high-performance computers to run smoothly.

Contents of the invention

In order to overcome the problems existing in related technologies at least to a certain extent, this application provides a semantic three-dimensional reconstruction method and system for multi-modal pose optimization.

According to a first aspect of the embodiments of the present application, a semantic three-dimensional reconstruction method for multi-modal pose optimization is provided. The method includes:

Collect RGB images and depth images of the target area;

Perform semantic segmentation on the RGB image and obtain the final mask image of the RGB image;

Using the RGB image, the depth image and the final mask image, obtain the global pose of the RGB image;

A global semantic three-dimensional point cloud is obtained according to the global pose of the depth image and the RGB image.

Preferably, the semantic segmentation of the RGB image and obtaining the final mask image of the RGB image include:

Use the STDC network based on deep learning to perform semantic segmentation on the RGB image and obtain the semantic category probability of each pixel in the RGB image;

Let the category corresponding to the category with the highest probability among the semantic category probabilities of each pixel be encoded as the mask of each pixel, and obtain the initial mask map of the RGB image;

The superpixel segmentation method gSLICr is used to perform edge optimization on the initial mask image of the RGB image to obtain the final mask image of the RGB image.

Preferably, using the RGB image, the depth image and the final mask image to obtain the global pose of the RGB image includes:

Extract ORB feature points of the RGB image;

Obtain the depth value of the ORB feature point in the depth image, and use the depth value of the ORB feature point in the depth image to perform projection coordinate conversion on the two-dimensional coordinates of the ORB feature point to obtain the The coordinates of the ORB feature points in the three-dimensional space;

Based on the coordinates of the ORB feature points in the three-dimensional space and the final mask map, the sparse SLAM method is used to obtain the global pose of the RGB image.

Preferably, based on the coordinates of the ORB feature points in the three-dimensional space and the final mask image, the sparse SLAM method is used to obtain the global pose of the RGB image, including:

Extract ORB feature points with the same BRIEF descriptor in two adjacent RGB images;

Using the final mask image, the outliers in the ORB feature points with the same BRIEF descriptor in the two adjacent frames of RGB images are eliminated, and the outliers in the two adjacent frames of the RGB image with the same BRIEF descriptor are eliminated. ORB feature points are matching points;

Based on the coordinates in the three-dimensional space of the matching point in the previous frame of the RGB image in the two adjacent frames of RGB images and the two-dimensional coordinates of the matching point in the latter frame of the RGB image in the two adjacent frames of RGB images, the PnP algorithm is used to obtain The pose between the first frame of two adjacent frames of RGB images;

Based on the first inter-frame pose, the beam adjustment method is used to obtain the global pose of the RGB image.

Preferably, using the final mask map to eliminate outliers in ORB feature points with the same BRIEF descriptor in two adjacent frames of RGB images includes:

According to the final mask map, determine the mask corresponding to each ORB feature point in each group of ORB feature points with the same BRIEF descriptor in two adjacent frames of RGB images;

When two ORB feature points in a group of ORB feature points with the same BRIEF descriptor have different masks, then the ORB feature points in the group with the same BRIEF descriptor are outliers, and the outlier points are eliminated.

Preferably, based on the first inter-frame pose, using the beam adjustment method to obtain the global pose of the RGB image includes:

Group all RGB images according to the preset number of groups;

The beam adjustment method is used to optimize the first inter-frame pose to obtain the second inter-frame pose, and the second inter-frame pose is used to obtain the distance between two adjacent second inter-frame poses. Relative posture;

Let the second inter-frame pose corresponding to the first group of two adjacent frames in each set of RGB images be the key pose, and use the beam adjustment method to optimize all key poses to obtain the optimized key pose ;

Based on the relative pose, the optimized key pose is used to update the second inter-frame pose, and all updated second inter-frame poses and all optimized key poses constitute a global pose.

Preferably, obtaining a global semantic three-dimensional point cloud based on the global pose of the depth image and the RGB image includes:

Generate point clouds of depth images;

Based on the voxblox framework, according to the global pose of the RGB image and the point cloud of the depth image, a weighted average algorithm is used to obtain a global three-dimensional model represented by TSDF;

The marching cube algorithm is used to obtain the three-dimensional point cloud in the global three-dimensional model, and all three-dimensional point clouds in the global three-dimensional model constitute a global semantic three-dimensional point cloud.

Preferably, the method further includes:

When the weighted average algorithm is used to obtain the global three-dimensional model represented by the TSDF, the Bayesian method is used to update the semantic category probability of the TSDF in the global three-dimensional model.

According to a second aspect of the embodiments of the present application, a multi-modal pose optimized semantic three-dimensional reconstruction system is provided. The system includes:

Acquisition module, used to acquire RGB images and depth images of the target area;

A semantic segmentation module, used to perform semantic segmentation on RGB images and obtain the final mask image of the RGB images;

A pose module used to obtain the global pose of the RGB image using the RGB image, the depth image and the final mask image;

A semantic three-dimensional reconstruction module, configured to obtain a global semantic three-dimensional point cloud based on the global pose of the depth image and the RGB image.

According to a third aspect of the embodiments of the present application, a computer device is provided, including: one or more processors;

The processor is used to store one or more programs;

When the one or more programs are executed by the one or more processors, the above-mentioned multi-modal pose optimized semantic three-dimensional reconstruction method is implemented.

According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the above-mentioned semantic three-dimensional reconstruction method of multi-modal pose optimization is implemented.

One or more of the above technical solutions of the present invention have at least one or more of the following beneficial effects:

The invention provides a multi-modal posture optimized semantic three-dimensional reconstruction method and system, which includes: collecting RGB images and depth images of the target area; performing semantic segmentation on the RGB image to obtain the final mask image of the RGB image; using RGB image, depth image and final mask map to obtain the global pose of the RGB image; obtain the global semantic three-dimensional point cloud based on the depth image and the global pose of the RGB image. The technical solution provided by the present invention not only requires low computing power and has wide applicability, but also improves the real-time performance and efficiency of semantic three-dimensional reconstruction.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of a semantic three-dimensional reconstruction method for multi-modal pose optimization according to an exemplary embodiment;

Figure 2 is a schematic diagram of a global pose acquisition process according to an exemplary embodiment;

Figure 3 is a schematic diagram illustrating the use of a Bayesian method to update the semantic category probability of TSDF in a global three-dimensional model according to an exemplary embodiment;

Figure 4 is a main structural block diagram of a multi-modal pose optimized semantic three-dimensional reconstruction system according to an exemplary embodiment.

Detailed ways

The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

As disclosed in the background art, semantic 3D reconstruction is to semantically annotate the 3D reconstruction model, and segment structures with different categories of attributes from the overall model to facilitate the use of downstream tasks. One of the important factors that determines the quality of semantic 3D reconstruction is the calculation of camera pose, which is related to the quality of the reconstruction model and the distribution of semantic information. The current positioning in the algorithm uses purely geometric features, which requires the design of a complex screening mechanism to determine matching points, and does not use the advanced feature of semantics. Semantic 3D reconstruction can be divided into three parts: semantic segmentation, 3D reconstruction, and semantic mapping. 3D reconstruction includes pose calculation and mapping.

Each part of the existing algorithm is relatively independent and performs data communication in a loosely coupled manner. For example, semantic 3D reconstruction mostly uses the GPU-based dense ICP algorithm to calculate the camera pose. At the same time, semantic segmentation also requires GPU resources, which has to be improved in terms of operating efficiency. Without compromise, the entire framework is too heavy and the scale of global optimization is large. Therefore, the semantic segmentation and reconstruction methods used in current semantic 3D reconstruction algorithms are not good in real-time performance, require high computing power, and impose a large burden on the overall operation of the algorithm, requiring high-performance computers to run smoothly.

In order to improve the above problems, improve the real-time performance and efficiency of semantic 3D reconstruction, and reduce the computing power requirements of semantic 3D reconstruction.

The above scheme is elaborated below.

Embodiment 1

The present invention provides a semantic three-dimensional reconstruction method for multi-modal pose optimization, as shown in Figure 1. This method can be, but is not limited to, used in terminals, and includes the following steps:

Step 101: Collect RGB images and depth images of the target area;

Step 102: Perform semantic segmentation on the RGB image and obtain the final mask image of the RGB image;

Step 103: Use the RGB image, depth image and final mask image to obtain the global pose of the RGB image;

Step 104: Obtain the global semantic three-dimensional point cloud based on the global pose of the depth image and RGB image.

In some embodiments, the RGB image can be acquired through, but is not limited to, an RGB camera, and the depth image can be acquired through an RGB-D depth camera; or the RGB image and the depth image can be acquired through a binocular camera; or the RGB image can be acquired through an RGB camera, and then deep learning can be used Calculate the depth image of an RGB image.

Further, step 102 includes:

Step 1021: Use the STDC network based on deep learning to perform semantic segmentation on the RGB image, and obtain the semantic category probability of each pixel in the RGB image;

Step 1022: Let the category corresponding to the category with the highest probability among the semantic category probabilities of each pixel be encoded as the mask of each pixel, and obtain the initial mask map of the RGB image;

For example, assume that there are three semantic categories of a certain pixel in an RGB image: A, B, and C. The category codes corresponding to semantic categories A, B, and C are 1, 2, and 3 respectively. The probability of semantic category A is 0.7. The probability of B is 0.2, and the probability of semantic category C is 0.1, then the mask of this pixel is the category code 1 of semantic category A;

Step 1023: Use the superpixel segmentation method gSLICr to perform edge optimization on the initial mask image of the RGB image to obtain the final mask image of the RGB image.

It is understandable that after semantic segmentation of RGB images through the STDC network based on deep learning, the initial mask image obtained may have problems such as incomplete images or jagged and rough image edges. By using the superpixel segmentation method gSLICr performs edge optimization on the initial mask map of the RGB image to make the mask map outline clearer. Therefore, semantic segmentation is performed by combining the lightweight deep learning-based STDC network with the fast superpixel segmentation method gSLICr. While ensuring speed, the segmentation results are more accurate, the contours are clearer, and the segmentation quality is improved.

It should be noted that the STDC network is a Short-Term Dense Concatenate (STDC) module embedded in U-Net, which can obtain a larger receptive field with very few parameters. In order to improve the accuracy of detail segmentation, STDC designed larger channels in the shallow network and designed the Detail Guidance module to learn detail characteristics, while reducing the channels of the deep network to reduce computational redundancy. The STDC network is a fast and lightweight semantic segmentation algorithm that can reach a segmentation speed of 250 frames per second on some data sets. It has low computational overhead and can accurately classify most pixels.

Superpixel segmentation is to cluster pixels with similar texture, brightness, color and other characteristics on the image to form irregular pixel blocks. The pixels in superpixels all have similar properties, which is equivalent to expressing a large number of pixels in the original image with a small number of image blocks, which greatly reduces the scale of subsequent image processing and has obvious dividing lines at the edges of objects. gSLICr is the implementation of the SLIC algorithm on the GPU and can achieve a segmentation speed of 250 frames per second.

The present invention integrates two segmentation methods with each other, uses superpixels to optimize the edges of the neural network segmentation results, and obtains a structured segmentation effect. The segmentation methods based on STDC network and gSLICr superpixel are very fast. Experiments have proved that the speed of pose calculation is dozens of frames per second, and the speed of three-dimensional reconstruction is more than ten frames per second. Therefore, applying the semantic segmentation results to pose calculation and semantic mapping will not affect the overall speed.

It should be noted that the "STDC network based on deep learning" and "superpixel segmentation method gSLICr" methods involved in the embodiments of the present invention are well known to those skilled in the art. Therefore, their specific implementation methods will not be described in detail. .

Further, step 103 includes:

Step 1031: Extract ORB feature points of the RGB image;

In some embodiments, the OpenCV library may be used, but is not limited to, to extract ORB feature points in RGB images;

It can be understood that ORB feature points are essentially pixel points. ORB feature points are significant pixel points selected from the pixel points of the RGB image, such as contour points, bright spots in darker areas, or dark spots in lighter areas. Click and wait;

Step 1032: Obtain the depth value of the ORB feature point in the depth image, and use the depth value of the ORB feature point in the depth image to perform projection coordinate conversion on the two-dimensional coordinates of the ORB feature point to obtain the ORB feature point in the three-dimensional space. coordinate;

In some optional embodiments, the two-dimensional coordinates of the ORB feature points are transformed into projected coordinates to obtain the coordinates of the ORB feature points in the three-dimensional space as follows:

In the above formula, P = (X, Y, Z) is the coordinate of the ORB feature point in the three-dimensional space, (u, v) is the two-dimensional coordinate of the ORB feature point, Z is the depth value corresponding to the ORB feature point, and K is Camera internal parameters; the semantic category corresponding to the two-dimensional coordinates of the ORB feature point is the same as the semantic category corresponding to the three-dimensional coordinates of the ORB feature point, because they all correspond to the same pixel;

Step 1033: Based on the coordinates of the ORB feature points in the three-dimensional space and the final mask map, use the sparse SLAM method to obtain the global pose of the RGB image.

Further, step 1033 includes:

Step 1033a: Extract ORB feature points with the same BRIEF descriptor in two adjacent frames of RGB images;

Step 1033b: Use the final mask image to eliminate outliers in the ORB feature points with the same BRIEF descriptors in the two adjacent frames of RGB images, and make the BRIEF descriptors in the two adjacent frames of RGB images after eliminating outliers the same. The ORB feature points are matching points;

Specifically, in step 1033b, the final mask image is used to eliminate outliers in ORB feature points with the same BRIEF descriptor in two adjacent frames of RGB images, including:

According to the final mask map, determine the mask corresponding to each ORB feature point in each group of ORB feature points with the same BRIEF descriptor in the two adjacent frames of RGB images;

When the masks corresponding to two ORB feature points in a group of ORB feature points with the same BRIEF descriptor are different, then the ORB feature points in the group with the same BRIEF descriptor are outliers, and the outlier points are eliminated;

For example, assume that the ORB feature points in the RGB image in the previous frame include D1, D2, and D3, and the ORB feature points in the RGB image in the next frame include D4, D5, and D6. The BRIEF descriptor of the ORB feature point D1 is the same as the ORB feature points D4 and D5. The BRIEF descriptors are the same, then there are two groups of ORB feature points with the same BRIEF descriptor, namely D1 and D4, D1 and D5; however, the mask of the ORB feature point D1 is 1, and the mask of the ORB feature point D4 is 2, and the mask of ORB feature point D5 is 1, then ORB feature points D1 and D4 with the same BRIEF descriptor are outliers, and ORB feature point D4 is eliminated; after eliminating outliers, each group of RGB images is adjacent to ORB feature points D1 and D5 with the same BRIEF descriptors in the two RGB images are matching points;

Step 1033c: Based on the coordinates in the three-dimensional space of the matching point in the previous RGB image in the two adjacent RGB images and the two-dimensional coordinates of the matching point in the subsequent RGB image in the two adjacent RGB images, use The PnP algorithm obtains the pose between the first frame of two adjacent frames of RGB images;

Step 1033d: Based on the first inter-frame pose, use the beam adjustment method to obtain the global pose of the RGB image.

For example, assume there are 12 frames of RGB images {a1, a2, a3, a4, a5, a6, a7, a8, a9, a10, a11, a12}, a1-a12 are RGB images;

Extract ORB feature points with the same BRIEF descriptor in two adjacent frames of RGB images, that is, extract "a1 and a2, a2 and a3, a3 and a4, a4 and a5, a5 and a6, a6 and a7, a7 and a8, a8 and a9, a9 and a10, a10 and a11, a11 and a12" are 11 groups of ORB feature points with the same BRIEF descriptor in two adjacent frames of RGB images;

Using the final mask image, eliminate the outliers in the ORB feature points with the same BRIEF descriptor in the above 11 groups of adjacent two frames of RGB images, and make the two adjacent frames of RGB images in each group of RGB images after eliminating outliers ORB feature points with the same BRIEF descriptor are matching points;

Based on the coordinates in the three-dimensional space of the matching point in the previous frame of the RGB image among the above 11 groups of adjacent two frames of RGB images and the two-dimensional matching point of the following frame of the RGB image among the above 11 groups of adjacent two frames of RGB images. Coordinates, use the PnP algorithm to obtain the first inter-frame pose of the above 11 groups of two adjacent frames of RGB images;

Based on the pose between the first frames, the beam adjustment method is used to obtain the global pose of the RGB image.

It can be understood that by eliminating outliers, the impact of outliers on the pose calculation accuracy can be reduced, thereby improving the accuracy and reliability of the pose.

Further, step 1033d includes:

Group all RGB images according to the preset number of groups;

Use the beam adjustment method to optimize the first inter-frame pose to obtain the second inter-frame pose, and use the second inter-frame pose to obtain the relative pose between two adjacent second inter-frame poses;

Let the second inter-frame pose corresponding to the first group of two adjacent frames in each set of RGB images be the key pose, and use the beam adjustment method to optimize all key poses to obtain the optimized key pose ;

Based on the relative pose, the optimized key pose is used to update the second inter-frame pose, and all updated second inter-frame poses and all optimized key poses constitute the global pose.

It should be noted that the present invention does not limit the "preset number of groups" and can be set by those skilled in the art based on experimental data or expert experience.

In some embodiments, all RGB images may be grouped according to, but are not limited to, a preset number of frames in each group; for example, the preset number of frames in each group may be, but is not limited to, 10 frames.

As shown in Figure 2, the RGB images are grouped according to the preset number of groups, and the PnP algorithm is used to calculate the pose T ₁ between the first frames between two adjacent frames of RGB images. However, the first inter-frame pose only considers the matching relationship between two adjacent frames of RGB images, and there is a large error, so it is used as the initial value for subsequent optimization.

Then perform local optimization and global optimization on the pose between the first frames. Specifically, the bundle adjustment method (BundleAdjustment) is used to optimize the pose T ₁ between the first frames to obtain the pose T ₂ between the second frames; the first RGB image frame in each group of RGB images is the key frame (Key Frame). ), the second inter-frame pose between the key frame and its adjacent RGB image is the key pose T ₃ . The key pose is optimized using the bundle adjustment method to obtain the optimized result of each set of RGB images. The key pose T ₄ .

In the global optimization stage, the present invention does not need to project the feature points of each frame in the group to the first frame, but directly uses the relative pose and the optimized key pose T ₄ of each group of RGB images to obtain the optimized The other poses T ₅ of the key pose are the updated second inter-frame pose. The updated second inter-frame pose T ₅ and the optimized key pose T ₄ constitute the global pose. This is because the relative pose T between two adjacent inter-frame poses remains unchanged, and the key pose is optimized and changed globally, so the poses between frames in the group also change accordingly.

For example, assume there are 4 frames of RGB images {a1, a2, a3, a4}, and a1-a4 are RGB images;

Use the PnP algorithm to obtain the first inter-frame poses of two adjacent frames of RGB images, and obtain three first inter-frame poses, namely the first inter-frame poses t1 and t1 of a1 and a2, a2 and a3, and a3 and a4. t2 and t3;

The beam adjustment method is used to optimize the poses t1, t2, and t3 between the first frames, and the poses t4, t5, and t6 between the second frames are obtained, that is, local optimization is performed; and the adjacent poses are obtained based on the poses between the second frames. The relative poses between the two second frames, that is, the relative poses △t 1 and △t ₂ between t4 and _t5 , t5 and t6; specifically, △t ₁ =t ₅ /t ₄ , △t ₂ =t ₆ /t ₅ ;

The default number of groups is 1, then there is a group of RGB images A1 = {a1, a2, a3, a4}, so that the second inter-frame pose corresponding to the first group of two adjacent frame images in each group of RGB images is The key pose, that is, the pose t4 between the second frames is the key pose. All key poses are optimized using the beam adjustment method to obtain the optimized key pose t7;

Since the relative pose between two adjacent frames remains unchanged, and the key pose changes globally, the pose of each second interframe in the group should also change accordingly to obtain all images. The result of frame global optimization, that is, based on the relative pose between two adjacent second inter-frame poses, the optimized key pose is used to update the second inter-frame pose (update the second inter-frame pose t5 and After t6, the updated second inter-frame poses t8 and t9 are obtained respectively. All the updated second inter-frame poses and all optimized key poses constitute the global pose (that is, the optimized key pose Poses t7, updated second inter-frame poses t8 and t9) to achieve global optimization.

It can be understood that by using the beam adjustment method and adopting a three-layer pose optimization strategy of inter-frame, local and global, the accuracy and real-time performance of pose calculation are improved, thereby further realizing the semantic three-dimensional multi-modal pose optimization. Reconstruction method.

It should be noted that the method of "updating the second inter-frame pose based on the relative pose and using the optimized key pose" involved in the embodiment of the present invention is well known to those skilled in the art. Therefore, its specific implementation The method will not be described too much.

Further, step 104 includes:

Step 1041: Generate a point cloud of the depth image;

In some embodiments, the point cloud of the depth image may be generated by, but is not limited to, camera back-projection;

Step 1042: Based on the voxblox framework, based on the global pose of the RGB image and the point cloud of the depth image, use the weighted average algorithm to obtain the global three-dimensional model represented by TSDF;

Step 1043: Use the marching cube algorithm to obtain the three-dimensional point cloud in the global three-dimensional model. All three-dimensional point clouds in the global three-dimensional model constitute the global semantic three-dimensional point cloud.

It should be noted that Voxblox uses the TSDF representation method based on voxel hashing and provides three fusion methods for points at the same spatial location, namely simple, merged and fast. Among them, the simple method performs fusion calculation on the TSDF value in each voxel, which has high accuracy and a large amount of calculation; the fast method treats multiple voxels as a group and fuses all TSDF values in the group, which reduces the calculation time. Scale speeds up the calculation, but the accuracy is reduced; merged is a compromise between the two methods. This invention considers that the semantic information of all points needs to be fused later, so a simple fusion method is used for reconstruction. Although each frame of image participates in the reconstruction, which will make the data richer and more fully integrated, the similarity of adjacent image pixels is high, which will cause more redundant calculations and increase the running cost. Considering the accuracy and speed of the algorithm, the present invention only 3D reconstruction using key poses.

This invention aims at the problem that semantic three-dimensional reconstruction involves many modules and requires high computing power. It uses a multi-layer pose calculation method based on sparse SLAM to solve the camera pose, and uses the semantic segmentation results to filter out external points. Use a lightweight real-time semantic segmentation network to implement pixel classification, use a voxblox-based real-time 3D reconstruction framework for scene 3D modeling, and finally achieve fast and accurate lightweight semantic 3D reconstruction.

Further, the method also includes:

When the weighted average algorithm is used to obtain the global three-dimensional model represented by the TSDF, the Bayesian method is used to update the semantic category probability of the TSDF in the global three-dimensional model.

It should be noted that by calculating the two-dimensional pixels of the same object on multiple frames of images, the same TSDF can be theoretically obtained. However, due to the measurement error of the sensor (camera), the actually calculated TSDFs do not overlap, and a weighted average algorithm needs to be used for fusion. The TSDF calculated from different images (before fusion) corresponds to a semantic category probability, and the Bayesian method is used to obtain the final semantic category after multi-frame TSDF fusion.

For example, as shown in Figure 3, for the same TSDF, there will be pixels corresponding to different poses, and the semantic segmentation results of the pixels at each pose are not exactly the same, as shown in the figure P(u=c _i ), i={1, 2, 3, 4, 5}. Therefore, the results of multiple segmentations need to be fused and updated to obtain the most possible semantic category P (u=c _Fusion ). This invention uses the same Bayesian update method as SemanticFusion to fuse semantic information, as shown in the formula:

Among them, P represents the probability that a pixel belongs to a certain category (probability), c _i is a certain category (class), I is a certain frame of image, and K is the total number of frames of the image. P(u=c _i |I _k ) represents the probability that pixel u belongs to category c _i obtained through semantic segmentation in the k-th frame image; P(c _i |I ₁ ,..., _k-1 ) represents the first comprehensive 1 Based on the semantic segmentation results of k-1 frames, the probability that pixel u belongs to c _i ; P(c _i |I ₁ ,..., _k ) represents the fusion probability of pixel u belonging to c _i in the first 1 to k frames; z is the normalization constant. Through continuous updating, the semantic categories of the same TSDF at all times can be integrated to obtain its final category attributes.

It should be noted that the Bayesian method is used to update and fuse semantic categories from multiple angles, ultimately achieving globally consistent semantic three-dimensional reconstruction and improving the reliability of semantic three-dimensional reconstruction.

The invention provides a semantic three-dimensional reconstruction method for multi-modal pose optimization. By collecting RGB images and depth images of the target area, the RGB image is semantically segmented to obtain the final mask image of the RGB image. Using the RGB image, depth image and the final mask map, obtain the global pose of the RGB image, and obtain the global semantic three-dimensional point cloud based on the depth image and the global pose of the RGB image, which not only achieves low computing power requirements, wide applicability, but also improves semantics Real-time and efficient 3D reconstruction.

Embodiment 2

A multi-modal pose optimized semantic 3D reconstruction system, as shown in Figure 4. The system includes:

Acquisition module, used to acquire RGB images and depth images of the target area;

The semantic segmentation module is used to perform semantic segmentation on RGB images and obtain the final mask image of the RGB image;

The pose module is used to obtain the global pose of the RGB image using the RGB image, depth image and final mask map;

The semantic 3D reconstruction module is used to obtain the global semantic 3D point cloud based on the global pose of the depth image and RGB image.

Further, the semantic segmentation module includes:

The first acquisition module is used to use the STDC network based on deep learning to perform semantic segmentation on the RGB image and obtain the semantic category probability of each pixel in the RGB image;

The second acquisition module is used to encode the category corresponding to the category with the highest probability among the semantic category probabilities of each pixel into the mask of each pixel, and obtain the initial mask map of the RGB image;

The third acquisition module is used to perform edge optimization on the initial mask image of the RGB image using the superpixel segmentation method gSLICr to obtain the final mask image of the RGB image.

Further, the pose module includes:

The extraction module can be used to extract ORB feature points of RGB images;

The fourth acquisition module is used to obtain the depth value of the ORB feature point in the depth image, and use the depth value of the ORB feature point in the depth image to perform projection coordinate conversion on the two-dimensional coordinates of the ORB feature point to obtain the location of the ORB feature point. Coordinates in three-dimensional space;

The fifth acquisition module is used to obtain the global pose of the RGB image using the sparse SLAM method based on the coordinates of the ORB feature points in the three-dimensional space and the final mask map.

Further, the fifth acquisition module is specifically used for:

Extract ORB feature points with the same BRIEF descriptor in two adjacent RGB images;

Using the final mask image, eliminate the outliers in the ORB feature points with the same BRIEF descriptor in the two adjacent frames of RGB images, and make the ORB features with the same BRIEF descriptor in the two adjacent frames of RGB images after eliminating the outliers Points are matching points;

Based on the coordinates in the three-dimensional space of the matching point in the previous frame of the RGB image in the two adjacent frames of RGB images and the two-dimensional coordinates of the matching point in the latter frame of the RGB image in the two adjacent frames of RGB images, the PnP algorithm is used to obtain The pose between the first frame of two adjacent frames of RGB images;

Based on the pose between the first frames, the beam adjustment method is used to obtain the global pose of the RGB image.

Furthermore, the final mask image is used to eliminate outliers in ORB feature points with the same BRIEF descriptor in two adjacent frames of RGB images, including:

According to the final mask map, determine the mask corresponding to each ORB feature point in each group of ORB feature points with the same BRIEF descriptor in the two adjacent frames of RGB images;

When the masks corresponding to two ORB feature points in a certain group of ORB feature points with the same BRIEF descriptor are different, the ORB feature points in the group with the same BRIEF descriptor are regarded as outliers, and the outlier points are eliminated.

Further, based on the pose between the first frames, the beam adjustment method is used to obtain the global pose of the RGB image, including:

Group all RGB images according to the preset number of groups;

Use the beam adjustment method to optimize the first inter-frame pose to obtain the second inter-frame pose, and use the second inter-frame pose to obtain the relative pose between two adjacent second inter-frame poses;

Let the second inter-frame pose corresponding to the first group of two adjacent frames in each set of RGB images be the key pose, and use the beam adjustment method to optimize all key poses to obtain the optimized key pose ;

Based on the relative pose, the optimized key pose is used to update the second inter-frame pose, and all updated second inter-frame poses and all optimized key poses constitute the global pose.

Further, the semantic three-dimensional reconstruction module is specifically used for:

Generate point clouds of depth images;

Based on the voxblox framework, based on the global pose of the RGB image and the point cloud of the depth image, the weighted average algorithm is used to obtain the global three-dimensional model represented by TSDF;

The marching cube algorithm is used to obtain the three-dimensional point cloud in the global three-dimensional model. All three-dimensional point clouds in the global three-dimensional model constitute the global semantic three-dimensional point cloud.

Furthermore, the system also includes:

The update module is used to update the semantic category probability of the TSDF in the global three-dimensional model using the Bayesian method when the weighted average algorithm is used to obtain the global three-dimensional model represented by the TSDF.

The invention provides a multi-modal pose optimized semantic three-dimensional reconstruction system. The RGB image and depth image of the target area are collected through the acquisition module. The semantic segmentation module performs semantic segmentation on the RGB image to obtain the final mask image of the RGB image. The pose module uses the RGB image, depth image and final mask map to obtain the global pose of the RGB image. The semantic 3D reconstruction module obtains the global semantic 3D point cloud based on the depth image and the global pose of the RGB image, which not only achieves the goal of computing power It has lower requirements, wide applicability, and improves the real-time performance and efficiency of semantic 3D reconstruction.

It can be understood that the above-mentioned system embodiments correspond to the above-mentioned method embodiments, and the corresponding specific contents can be referred to each other, and will not be described again here.

It can be understood that the same or similar parts in the above-mentioned embodiments can be referred to each other, and the content that is not described in detail in some embodiments can be referred to the same or similar content in other embodiments.

Embodiment 3

Based on the same inventive concept, the present invention also provides a computer device. The computer device includes a processor and a memory. The memory is used to store computer programs. The computer programs include program instructions. The processor is used to execute program instructions stored in a computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), or off-the-shelf programmable gate Array (Field-Programmable GateArray, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, are suitable for implementing one or more instructions, and are specifically suitable for Loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding functions to implement the steps of a multi-modal pose optimized semantic three-dimensional reconstruction method in the above embodiment.

Embodiment 4

Based on the same inventive concept, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device and is used to store programs and data. It can be understood that the computer-readable storage medium here may include a built-in storage medium in the computer device, and of course may also include an extended storage medium supported by the computer device. The computer-readable storage medium provides storage space, and the storage space stores the operating system of the terminal. Furthermore, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the steps of a multi-modal pose optimized semantic three-dimensional reconstruction method in the above embodiment.

Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that the present invention can still be modified. Modifications or equivalent substitutions may be made to the specific embodiments, and any modifications or equivalent substitutions that do not depart from the spirit and scope of the invention shall be covered by the scope of the claims of the invention.

Claims (10)

1. A semantic three-dimensional reconstruction method for multi-modal pose optimization, the method comprising:

collecting RGB images and depth images of a target area;

performing semantic segmentation on an RGB image to obtain a final mask image of the RGB image;

acquiring the global pose of the RGB image by using the RGB image, the depth image and the final mask map;

acquiring a global semantic three-dimensional point cloud according to the global pose of the depth image and the RGB image;

the obtaining the global pose of the RGB image by using the RGB image, the depth image and the final mask map includes:

extracting ORB characteristic points of the RGB image;

obtaining a depth value of the ORB characteristic point in the depth image, and performing projection coordinate conversion on the two-dimensional coordinates of the ORB characteristic point by utilizing the depth value of the ORB characteristic point in the depth image to obtain the coordinates of the ORB characteristic point in a three-dimensional space;

and acquiring the global pose of the RGB image by using a sparse SLAM method based on the coordinates of the ORB feature points in the three-dimensional space and the final mask map.

2. The method of claim 1, wherein the semantically segmenting the RGB image to obtain a final mask map of the RGB image comprises:

Performing semantic segmentation on the RGB image by using an STDC network based on deep learning to obtain semantic class probability of each pixel point in the RGB image;

coding the category corresponding to the category with the highest probability in the semantic category probability of each pixel point as a mask of each pixel point to obtain an initial mask map of the RGB image;

and performing edge optimization on the initial mask map of the RGB image by using a super-pixel segmentation method gSLICr to obtain a final mask map of the RGB image.

3. The method of claim 1, wherein the acquiring the global pose of the RGB image using the sparse SLAM method based on the coordinates of the ORB feature points in three-dimensional space and the final mask map comprises:

extracting ORB characteristic points with the same BRIEF descriptors in two adjacent RGB images;

removing outer points in ORB characteristic points with the same BRIEF descriptors in two adjacent frames of RGB images by using the final mask map, and enabling the ORB characteristic points with the same BRIEF descriptors in the two adjacent frames of RGB images after removing the outer points to be matching points;

acquiring a first frame pose of two adjacent RGB images by using a PnP algorithm based on coordinates of a matching point in a previous RGB image in the two adjacent RGB images in a three-dimensional space and two-dimensional coordinates of a matching point in a next RGB image in the two adjacent RGB images;

And acquiring the global pose of the RGB image by using a beam adjustment method based on the first frame pose.

4. A method according to claim 3, wherein the removing outliers in ORB feature points with identical BRIEF descriptors in two adjacent frames of RGB images using the final mask map comprises:

determining masks corresponding to ORB feature points in ORB feature points with the same BRIEF descriptors in each group of BRIEF descriptors in two adjacent frames of RGB images according to the final mask map;

when the masks corresponding to two ORB feature points in the ORB feature points with the same BRIEF descriptors in a certain group are different, the ORB feature points with the same BRIEF descriptors in the group are outliers, and the outliers are eliminated.

5. A method according to claim 3, wherein the obtaining the global pose of the RGB image using the beam adjustment method based on the first frame pose comprises:

grouping all RGB images according to a preset group number;

optimizing the first frame pose by using a beam adjustment method to obtain a second frame pose, and obtaining the relative pose between two adjacent second frame poses by using the second frame pose;

the second frame pose corresponding to the first group of two adjacent frames of images in each group of RGB images is made to be the key pose, and all the key poses are optimized by utilizing a beam adjustment method to obtain the optimized key pose;

Based on the relative pose, updating the second frame pose by using the optimized key pose, wherein all updated second frame poses and all optimized key poses form a global pose.

6. The method of claim 1, wherein the obtaining a global semantic three-dimensional point cloud from the global pose of the depth image and the RGB image comprises:

generating a point cloud of the depth image;

based on a voxblox frame, acquiring a global three-dimensional model represented by TSDF by using a weighted average algorithm according to the global pose of the RGB image and the point cloud of the depth image;

and acquiring three-dimensional point clouds in the global three-dimensional model by using a marking cube algorithm, wherein all three-dimensional point clouds in the global three-dimensional model form global semantic three-dimensional point clouds.

7. The method of claim 6, wherein the method further comprises:

when the global three-dimensional model represented by the TSDF is obtained by using a weighted average algorithm, the semantic class probability of the TSDF in the global three-dimensional model is updated by using a Bayesian method.

8. A multi-modal pose optimized semantic three-dimensional reconstruction system, the system comprising:

The acquisition module is used for acquiring RGB images and depth images of the target area;

the semantic segmentation module is used for carrying out semantic segmentation on the RGB image to obtain a final mask image of the RGB image;

the pose module is used for acquiring the global pose of the RGB image by utilizing the RGB image, the depth image and the final mask image;

the semantic three-dimensional reconstruction module is used for acquiring a global semantic three-dimensional point cloud according to the depth image and the global pose of the RGB image;

the pose module comprises:

the extraction module can be used for extracting ORB characteristic points of the RGB image;

the fourth acquisition module is used for acquiring the depth value of the ORB characteristic point in the depth image, and performing projection coordinate conversion on the two-dimensional coordinates of the ORB characteristic point by utilizing the depth value of the ORB characteristic point in the depth image to obtain the coordinates of the ORB characteristic point in the three-dimensional space;

and a fifth acquisition module, configured to acquire a global pose of the RGB image by using a sparse SLAM method based on coordinates of the ORB feature points in the three-dimensional space and the final mask map.

9. A computer device, comprising: one or more processors;

the processor is used for storing one or more programs;

The multi-modal pose optimized semantic three-dimensional reconstruction method according to any one of claims 1 to 7 when the one or more programs are executed by the one or more processors.

10. A computer readable storage medium, characterized in that a computer program is stored thereon, which, when executed, implements the multi-modal pose optimized semantic three-dimensional reconstruction method according to any one of claims 1 to 7.