CN117576303A - Three-dimensional image generation method, device, equipment and storage medium - Google Patents

Abstract

The application discloses a three-dimensional image generation method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring original RGB image data; inputting the original RGB image data into a preset image processing model, and carrying out fusion processing on pixel coordinate system data and camera coordinate data of the original RGB image data based on the preset image processing model to obtain a low-resolution depth image; wherein the camera coordinate data includes depth image information of the original RGB image data and two-dimensional camera coordinate data, and the depth image information is not used as an original input but for a coarse-grained synthetic image; and performing three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image. The present application improves target detection and spatial localization performance for target instances in input RGB image data.

Description

Three-dimensional image generation method, device, equipment and storage medium

Technical field

The present application relates to the field of computer vision technology, and in particular to a three-dimensional image generation method, device, equipment and storage medium.

Background technique

In recent years, with the rapid development of artificial intelligence, fields such as robotic operations and virtual reality have received great attention. The detected target instances are perceived, understood and interpreted through the three-dimensional understanding technology corresponding to the sensor equipment, and then the target is analyzed. Three-dimensional scene reconstruction based on examples is an important technical link in these fields, and is also the basis for the subsequent decision-making and execution stages.

Existing 3D understanding technology is mainly based on a single-stage scheme that directly infers the overall 3D information of multiple targets. This scheme regards the target instance as the center point of its 2D image, and each center point contains the complete 3D of the target instance. Information, although this method has advantages in real-time performance, occlusion scene performance and three-dimensional shape accuracy, it only completes the migration from bounding box mode to bounding box mode in the training method, without the mandatory area caused by instance segmentation Constraints, the target detection task architecture used is also difficult to extract category-level features on small data sets, so it cannot handle the intra-class differences of target instances well, resulting in poor performance in target detection and spatial localization.

Contents of the invention

The main purpose of this application is to provide a three-dimensional image generation method, device, equipment and storage medium, aiming to solve the problem in related technologies that the single-stage solution of directly inferring the overall three-dimensional information of multi-targets cannot detect target instances well. Handling intra-class differences in target instances leads to technical problems with poor performance in target detection and spatial localization.

To achieve the above objectives, embodiments of the present application provide a three-dimensional image generation method, which method includes:

Get original RGB image data;

The original RGB image data is input into a preset image processing model. Based on the preset image processing model, the pixel coordinate system data of the original RGB image data and the camera coordinate system data are fused to obtain a low-resolution depth. Image; wherein the camera coordinate system data includes the depth image information of the original RGB image data and the two-dimensional camera coordinate data, and the depth image information is not used as original input, but is used for coarse-grained synthetic images;

Perform three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image.

In a possible implementation of the present application, the step of fusing the pixel coordinate system data of the original RGB image data and the camera coordinate system data to obtain a low-resolution depth image includes:

Perform vector calculation on the original RGB image data to obtain first normal vectors of multiple pixels in the original RGB image data;

Perform clustering processing on the coordinate system point sets of the pixel coordinate system data and the camera coordinate system data to obtain cluster feature data;

Feature fusion is performed on the first normal vector and the cluster feature data to obtain a low-resolution depth image.

In a possible implementation of the present application, the step of clustering the coordinate system point sets of the pixel coordinate system data and the camera coordinate system data to obtain cluster feature data includes:

Perform feature extraction on the coordinate system point set of the pixel coordinate system data and the camera coordinate system data to obtain pixel coordinate feature data and camera coordinate feature data;

According to the similarity between the pixel coordinate feature data and the camera coordinate feature data, feature aggregation is performed on multiple point set feature data to obtain cluster feature data.

In a possible implementation of the present application, the step of performing vector calculation on the original RGB image data to obtain the first normal vectors of multiple pixels in the original RGB image data includes:

Perform data enhancement processing on the original RGB image data to obtain first enhanced data;

Perform two-dimensional gradient calculation on the first enhanced data to obtain a gradient calculation value;

Select the maximum value among the gradient calculation values, and determine the maximum value among the gradient calculation values as the first normal vector of each pixel in the first enhancement data.

In a possible implementation of the present application, the step of performing three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image includes:

Generate a target Gaussian heat value map according to multiple target examples in the original RGB image data;

Based on the aggregated features in the low-resolution depth image and the target Gaussian heat value map, three-dimensional point cloud reconstruction is performed to obtain a target three-dimensional image.

In a possible implementation of the present application, after the step of generating a target Gaussian heat value map based on multiple target examples in the original RGB image data, the step includes:

According to the target Gaussian heat value map, calculate the center coordinates of multiple target instances in the target Gaussian heat value map, and determine the overall three-dimensional information parameter map based on the multiple target instance center coordinates;

Complete three-dimensional information of all target instances in the overall three-dimensional information parameter map is extracted.

In a possible implementation of the present application, after performing three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image, the step includes:

Calculate the center distance estimation result according to the center point data of the target instance of the target three-dimensional image on the two-dimensional coordinate system;

Consistency constraint processing is performed on the regression result of the center distance estimation result and the overall three-dimensional information parameter map, so that the input target instance converges.

This application also provides a three-dimensional image generating device, which includes:

Acquisition module, used to obtain original RGB image data;

The first processing module is used to input the original RGB image data into a preset image processing model, and fuse the pixel coordinate system data of the original RGB image data and the camera coordinate system data based on the preset image processing model. Process to obtain a low-resolution depth image; wherein the camera coordinate system data includes the depth image information of the original RGB image data and the two-dimensional camera coordinate data, and the depth image information is not used as original input, but is used for Coarse-grained synthetic images;

A conversion module is used to perform three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image.

This application also provides a three-dimensional image generation device. The three-dimensional image generation device is a physical node device. The three-dimensional image generation device includes: a memory, a processor, and a device that is stored on the memory and can run on the processor. The program of the three-dimensional image generating method can realize the steps of the above-mentioned three-dimensional image generating method when the program of the three-dimensional image generating method is executed by a processor.

In order to achieve the above object, a storage medium is also provided. A three-dimensional image generation program is stored on the storage medium. When the three-dimensional image generation program is executed by a processor, the steps of any one of the three-dimensional image generation methods described above are implemented.

This application provides a three-dimensional image generation method, device, equipment and storage medium. Compared with related technologies, a single-stage scheme that directly infers the overall three-dimensional information of multiple targets to detect target instances cannot handle the intra-class differences of target instances well, resulting in poor performance in target detection and spatial positioning. In this application, the original RGB image data is obtained; the original RGB image data is input to a preset image processing model, and based on the preset image processing model, the pixel coordinate system data and camera coordinates of the original RGB image data are compared System data are fused to obtain a low-resolution depth image; wherein the camera coordinate system data includes the depth image information of the original RGB image data and the two-dimensional camera coordinate data, and the depth image information is not used as the original input, Instead, it is used for coarse-grained synthesis of images; the low-resolution depth image is subjected to three-dimensional conversion to obtain a target three-dimensional image. In this application, by acquiring the original RGB image data, the original RGB image data is divided into camera coordinate system data and pixel coordinate system data for double-end input, and the depth image information of the original RGB image data is used for coarse-grained synthetic images, Instead of being used as the original input, this avoids the network performing relevant operations on each feature network on the original input image, destroying the hidden three-dimensional structure in the image, improving the performance of target detection and spatial positioning, and then accurately detecting the detected image data. Perform three-dimensional reconstruction.

Description of the drawings

Figure 1 is a schematic flow chart of the first embodiment of the three-dimensional image generation method of the present application;

Figure 2 is a schematic diagram of the overall processing flow of the preset image processing model involved in the three-dimensional image generation method of the present application;

Figure 3 is a schematic diagram of the equipment structure of the hardware operating environment involved in the embodiment of the present application;

Figure 4 is a schematic diagram of a Gaussian heat value diagram of a target instance in the image data involved in the three-dimensional image generation method of this application;

Figure 5 is a schematic diagram of the normal vector of the pixel coordinate system information involved in the three-dimensional image generation method of the present application;

Figure 6 is a two-dimensional coordinate diagram of the pixel coordinate system information involved in the three-dimensional image generation method of the present application;

Figure 7 is a schematic diagram of the three-dimensional point cloud clustering autoencoder involved in the three-dimensional image generation method of this application;

Figure 8 is a schematic diagram of the single-stage processing flow of the three-dimensional point cloud clustering autoencoder involved in the three-dimensional image generation method of this application;

Figure 9 is a schematic diagram of the image data feature extraction process involved in the three-dimensional image generation method of the present application;

Figure 10 is a schematic diagram of a low-resolution synthetic depth image involved in the three-dimensional image generation method of this application;

Figure 11 is a schematic diagram of the cyclic geometric consistency constraint calculation process involved in the three-dimensional image generation method of this application;

Figure 12 is a schematic diagram of a reconstructed three-dimensional image of a target instance involved in the three-dimensional image generation method of this application;

Figure 13 is a schematic diagram of the three-dimensional reconstruction effect of the test set involved in the three-dimensional image generation method of this application.

Detailed ways

It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

An embodiment of the present application provides a three-dimensional image generation method. In the first embodiment of the three-dimensional image generation method of the present application, referring to Figure 1, the method includes:

Step S10, obtain original RGB image data;

Step S20: Input the original RGB image data into a preset image processing model. Based on the preset image processing model, fuse the pixel coordinate system data and the camera coordinate system data of the original RGB image data to obtain a low Resolution depth image; wherein the camera coordinate system data includes the depth image information of the original RGB image data and the two-dimensional camera coordinate data, and the depth image information is not used as the original input, but is used for coarse-grained synthetic images ;

Step S30: perform three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image.

This embodiment aims to: divide the original RGB image data into camera coordinate system data and pixel coordinate system data for double-ended input, and use the depth image information of the original RGB image data for coarse-grained synthetic images instead of as original input. This prevents the network from destroying the hidden three-dimensional structure in the image after performing relevant operations on each feature network on the original input image, and improves the performance in target detection and spatial positioning.

In this embodiment, the research and development background targeted is:

This method belongs to the single-view overall 3D understanding technology, which mainly involves target pose estimation and 3D scene reconstruction. The existing overall 3D understanding technology can be divided into two categories: instance-level overall 3D understanding technology and category-level overall 3D understanding technology;

1. Instance-level overall three-dimensional understanding technology: In terms of target pose estimation, instance-level pose estimation methods based on traditional operators or deep learning technology are used, relying on a priori matching information or accurate three-dimensional models calculated from target instances; in For 3D scene reconstruction, the 3D model of the target instance is directly used. The main drawback of this type of method is poor versatility and real-time performance. Take the robot operation of grabbing specified categories of objects as an example. When the category contains a large number of objects with different appearances and shapes, the instance-level pose estimation method usually requires a lot of time. Spending a lot of time performing template matching operations, and then considering transplanting them to new environments containing unseen target objects, the performance of such methods will be seriously degraded or even unusable because there is no prior information about the instances.

2. Category-level overall 3D understanding technology: Different from instance-level overall 3D understanding technology, category-level overall 3D understanding technology only uses the category-level pose scale and shape prior information learned during training, and generalizes to the future in the inference stage. The target category instances that have been seen before are therefore more challenging. Depending on the complexity of the model, the category-level overall 3D understanding technology can be divided into a multi-stage scheme that first segments the region of interest from the image and a direct inference of the multi-target overall 3D A one-stage scheme for information.

In the multi-stage solution using pre-segmentation, some work directly uses the color-depth information of the target area to infer the six-degree-of-freedom pose through a complex architecture including a point cloud processing network and a pose estimation network to generate a target canonical point cloud. As a by-product, the main defects of this type of method are as follows: 1. Its three-dimensional reconstruction capability is regarded as supplementary information inference for pose estimation, which is derived from the average shape prior of all known instances in the category through deformation prediction, and the reconstruction performance is limited. ; 2. The computational cost is high and it relies heavily on image segmentation quality, and its performance is poor in complex multi-target occlusion scenes and small sample learning tasks.

In a single-stage scheme that directly infers the overall 3D information of multiple targets, the target object is detected in a bounding box-less manner and its six-degree-of-freedom pose, 3D shape, and true size are simultaneously inferred. This method treats target instances in image data as The center point of its two-dimensional image, and each center point contains complete three-dimensional information of the target instance. Although it has advantages in real-time performance, occlusion scene performance, and three-dimensional shape accuracy, it only completes the training method by the boundary. The migration from box mode to boundless box mode ignores the differences between different fields, that is, in the overall three-dimensional understanding task where the data set is small, the reasoning is difficult, and it is strongly related to the real space geometric information, it is still used for very large-scale The inference method and model architecture of the data set and single task, and this design that relies entirely on the feature extraction network cannot handle the differences within the target object class well, and lacks the mining of real spatial geometric information. In target detection and space Positioning performance is poor.

In this scheme, an improvement is made on the basis of a single-stage scheme that directly infers the overall three-dimensional information of multiple targets. By using the coordinate system to separate the input, the three-dimensional scene understanding is divided into target instance abstraction and scale position understanding in the hidden layer feature space transformation. Two stages, to explicitly mine the real spatial geometric information; to hierarchically aggregate features of different preferences through similarity measurement, while establishing feature mapping of pixel coordinate system information and camera coordinate system information, making the network pay more attention to the overall inter-cluster Differences rather than shape detail matching; construct confidence geometric consistency constraints to eliminate errors between generated results and true values.

Specific steps are as follows:

Step S10, obtain original RGB image data;

As an example, the three-dimensional image generation method can be applied to a three-dimensional image generation device, which belongs to a three-dimensional image generation system, and the three-dimensional image generation system belongs to a three-dimensional image generation device.

As an example, the original RGB image data is a single RGB-D image obtained. In the single-view overall three-dimensional understanding technology, the image data to be input is obtained, where the image data can be photos, RGB images, etc., according to the image Target objects/target instances in the data are used for 3D scene reconstruction.

As an example, the original RGB image data can be a single RGB-D image input (I∈R ^w×h×3 , D∈R ^w×h ) with a pixel width of w and a pixel height of h. The goal is to detect three-dimensional All objects of interest in the scene and infer their six-degree-of-freedom pose P∈SE(3), one-dimensional real scale factor s, and three-dimensional shape point cloud C∈R ^K×N×3 , where K represents the object of interest in the three-dimensional scene The number ^of The pose P, the normalized 3D shape point cloud C and the one-dimensional real scale factor s completely define the target instance of interest relative to the camera coordinate system in the 3D scene, which is the complete 3D information required for most vision tasks.

Step S20: Input the original RGB image data into a preset image processing model. Based on the preset image processing model, fuse the pixel coordinate system data and the camera coordinate system data of the original RGB image data to obtain a low Resolution depth image; wherein the camera coordinate system data includes the depth image information of the original RGB image data and the two-dimensional camera coordinate data, and the depth image information is not used as the original input, but is used for coarse-grained synthetic images ;

As an example, the preset image processing model is an image processing model based on deep reinforcement learning. The preset image processing model includes the overall three-dimensional understanding algorithm CoCFusion, and the image is processed through the overall three-dimensional understanding algorithm.

As an example, RGB image data contains pixel coordinate system information and camera coordinate system information, where the pixel coordinate system data is the pixel two-dimensional coordinate point in the input image data, and the pixel coordinate system data includes multiple pixel points, The camera coordinate system data includes multiple two-dimensional coordinate points and three-dimensional coordinate points, which are mainly used to represent the spatial position relationship of each target instance in the RGB image data.

As an example, the low-resolution depth image is the image information obtained by fusing the pixel coordinate system data of the original RGB image data with the camera coordinate system data. The conversion process from the original RGB image to the low-resolution depth image is shown in Figure 10 shows that the low-resolution depth image is the coarse-grained synthetic depth image in (c) in the figure. The prediction process of the low-resolution depth information is also a learning process of the mapping relationship between the two-dimensional pixel coordinate system information and the three-dimensional camera coordinate system information. It can improve the network's understanding of shape information distribution.

As an example, the depth image information is the scale position relationship information of each target instance in the camera coordinate system. Compared with the method of fully utilizing the neural network fitting ability to understand the three-dimensional scene from the original depth information, the input end processing method of coordinate system separation The normal vector DNR of the depth image and the two-dimensional coordinates (x _c , y _c ) in the camera coordinate system are used as the camera coordinate system information, and the depth information z _c is synthesized at the end of the shallow network in a coarse-grained manner, so that the three-dimensional scene understanding can be achieved implicitly. The layer feature space transformation is divided into two stages: target instance abstraction and scale position understanding. The main advantages of this method are as follows: (1) It has a certain similarity with the multi-stage scheme based on region of interest segmentation and can more explicitly mine the real space. Geometric information.

Step S30: perform three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image.

As an example, the process of three-dimensional conversion can be to infer its six-degree-of-freedom pose, three-dimensional shape and true size while detecting the target object, thereby extracting the complete three-dimensional information of all target instances in the input image data. In order to learn Three-dimensional point cloud encoding of target instances. This solution uses a training method similar to the category-level pose method SPD to design an autoencoder (AutoEncoder, AE) to train all three-dimensional shapes from the ShapeNetCore-CAD data set. The autoencoder The implement has representation invariance and can be used for all three-dimensional shape representation work, as shown in Figure 7.

As an example, after extracting complete three-dimensional information, the complete three-dimensional image reconstruction process is completed through PointMLP. PointMLP is a simple and effective three-dimensional point cloud analysis network, using the residual point MLP module (ResidualPoint MLP block, ResP Block) step by step Extracting local features is simpler and deeper than the PointNet or PointNet++ 3D point cloud encoder used in existing category-level pose estimation methods, with better performance. However, PointMLP is not good at mining local geometric details. For complex components constructed Based on the structured 3D model, the reconstructed local shape is relatively blurry. Therefore, this application adds two spaced context clustering modules in the preprocessing steps of all stages, which serve to cluster together three-dimensional points belonging to the same component and enhance the local structural details of the three-dimensional model.

As an example, the three-dimensional point cloud encoder PointMLP-CoCs is shown in Figure 8, where φ _pre (·) is the feature preprocessing sequence with the interval context clustering module added, φ _pos (·) is the feature post-processing sequence, N and M are the number of modules in the sequence. In a single stage, the point feature abstraction process of PointMLP-CoCs is shown in formula (3-1). For point i, the k-Nearest Neighbors (kNN) method is used to perform feature extraction from a local area containing K points. Extraction, φ _pre (·) is designed to learn shared weights from local areas, the aggregation function A performs feature aggregation through the maximum pooling operation, and φ _pos (·) is used to extract deep aggregate features. The specific formula is as follows :

g _i =Φ _pos (A(Φ _pre (f _{i, j} ), | j = 1,...,K)). (3-1)

As an example, considering the real-time nature of 3D reconstruction, CoCFusion low-dimensionally maps the normalized 3D point cloud P∈R ^2048×3 to the 3D shape hidden layer vector z∈R ¹²⁸ . Therefore, only three-layer MLPs with sizes of 128→512, 128→1024, and 1024→2048×3 are used on the decoder side for 3D point cloud reconstruction. For the selection of the loss function, CoCFusion uses the Chamfer Distance (CD) representing the point cloud reconstruction error to optimize the three-dimensional point cloud clustering autoencoder. The calculation process is shown in Formula (3-2).

Among them, i can be 1, 2, 3, representing each point, x and y represent coordinates, and p represents the three-dimensional point cloud.

As an example, after the training process is completed, the three-dimensional reconstruction effect of the three-dimensional point cloud clustering autoencoder PointMLP-CoCs designed by the present invention on the ShapeNetCore-CAD test data set is shown in Figure 12 and Figure 13, where the first row The image shows the original model display of the brimmed hat instance, the headphone instance, the seat instance, and the guitar instance. The second line is the corresponding three-dimensional reconstruction result.

Wherein, the step of performing three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image includes:

Step S301: Generate a target Gaussian heat value map based on multiple target examples in the original RGB image data;

As an example, the target instance is an object in the image data. By taking the training method of treating the target as a pixel point, the target instance is modeled as a single point (the center point of the target bounding box in the two-dimensional image), and then through Key point estimation is used to find the center point and return to the complete three-dimensional information of the target instance. The complete three-dimensional information includes the six-degree-of-freedom pose P, the normalized three-dimensional shape point cloud C and the one-dimensional true scale factor s. The target is regarded as the center The point training method is simple, fast, end-to-end differentiable, does not require any non-maximum suppression (NMS) post-processing operations, and can estimate a series of additional object attributes in a single inference process, which is real-time Important research in the field of target detection and related tasks, but it should be noted that because the overall three-dimensional understanding task is too complex, CoCFusion (the algorithm used in this application) only performs instance center point estimation, and the complete three-dimensional information is processed by other tasks in the network The inference result is extracted at the center point, which acts as an anchor point and does not attach any additional information.

As an example, the target Gaussian heat value map is a heat value map generated from the center point corresponding to the target instance of the ground truth data. It is the first step in representing the center point of the instance. The generated target Gaussian heat value map is shown in Figure 4.

Step S302: Perform three-dimensional point cloud reconstruction based on the aggregated features in the low-resolution depth image and the target Gaussian heat value map to obtain the target three-dimensional image.

As an example, the heat value image of each target instance in the image data can be seen through the target Gaussian heat value map. In the process of generating the target three-dimensional image, it is also necessary to extract the aggregate features of the low-resolution depth image.

As an example, feature extraction is performed on the above input point set through a hierarchical feature fusion network composed of the context clustering module CoC Block and the spatial-channel attention module GAM, and the features of different preferences are hierarchically aggregated in the form of similarity measurement and At the end of the shallow network, the depth image is coarse-grainedly synthesized, and finally the target instance heat value map, semantic segmentation image, high-resolution depth image, and overall target-centered three-dimensional information parameters are predicted through different task processing networks. Compared with existing The ResNet-FPN feature extraction architecture is a single-stage solution for overall three-dimensional understanding. The feature extraction method based on the context clustering module makes the network pay more attention to the overall differences between clusters rather than the detailed matching of appearance and shape, thus reflecting the characteristics of each target instance. Features, and then, based on the Gaussian heat value map and the characteristics of each target instance, the reconstruction of the three-dimensional scene is completed.

Wherein, after the step of generating a target Gaussian heat value map based on multiple target examples in the original RGB image data, the step includes:

Step A1, according to the target Gaussian heat value map, calculate the center coordinates of multiple target instances in the target Gaussian heat value map, and determine the overall three-dimensional information parameter map based on the multiple target instance center coordinates;

Step A2: Extract complete three-dimensional information of all target instances in the overall three-dimensional information parameter map.

As an example, after the target Gaussian heat value map is generated, there is a corresponding instance center point in the target Gaussian heat value map, and the complete three-dimensional information is extracted at the center point position in the inference results of other task processing networks.

Wherein, after the step of performing three-dimensional conversion on the low-resolution depth image to obtain the target three-dimensional image, the step includes:

Step B1: Calculate the center distance estimation result according to the center point data of the target instance of the target three-dimensional image on the two-dimensional coordinate system;

As an example, the center point data is the center point of each target instance on the two-dimensional coordinate system.

As an example, after generating a three-dimensional image, it is necessary to construct a cyclic geometric consistency constraint through a loss function, penalize the inconsistency between the real mask area, predicted mask area, and projected mask area, and construct a cyclic geometric consistency constraint, The orthogonal axis loss of FS-Net, the center distance estimation of PoseCNN and the confidence of direct regression loss are combined with the cycle geometric consistency constraint to form a confidence geometric consistency constraint.

As an example, the inference of confidence parameters is achieved by adding additional predictions using sigmoid post-processing in the corresponding task processing network, which is very easy to do. Furthermore, in addition to the optimization process of auxiliary 3D information, other loss functions They are all mask loss functions based on the ground truth target instance heat value map Y∈[0,1] ^w×h×1 . They will only be executed when the pixel heat value h(i,j) is greater than δ to prevent Fuzziness occurs in areas where no objects exist. δ is the heat mask threshold, which is taken as 0.35 in this application.

As an example, the auxiliary 3D information loss function consists of a coarse-grained synthetic depth image loss function, a predicted high-resolution depth image loss function, and a predicted class mask image loss function.

Step B2: Perform consistency constraint processing on the regression result of the center distance estimation result and the overall three-dimensional information parameter map, so that the input target instance converges.

As an example, the center distance estimation result is when the camera internal parameter matrix is known, by estimating the three-dimensional translation vector of the center point of the target instance on the two-dimensional image and the depth of the center point, through the formula (3-3 ) to obtain the center distance estimation result t _cd ′, combine the center distance estimation result t _cd ′ with the direct regression result t _3d in the overall three-dimensional information parameter map with confidence, and consider the loss of the one-dimensional true scale factor s, using the L ₁ distance Construct a loss function with cosine similarity to make the input target instance converge. The loss function in three-dimensional rotation is shown in formula (3-4).

As an example, the introduction of the predicted category mask image SSI∈R ^{w×h×(cls+1)} as auxiliary three-dimensional information can enable the network to better detect target instances of interest in the two-dimensional image plane, but in the three-dimensional space The real geometric information of The learning ability of the mapping process between the two-dimensional pixel plane and the real three-dimensional space.

As an example, this goal is achieved through the following design: (1) Suppose the known camera intrinsic parameter matrix is K, then the predicted three-dimensional rotation matrix R∈SO(3), and the predicted three-dimensional translation vector t∈R ³ define 3D→2D Mapping relationship π=K[R|t] ^-1 . For the three-dimensional shape p′=π(s*p) of the real target instance represented by the one-dimensional real scale factor s and the normalized three-dimensional point cloud P∈R ^2048×3 , the center point coordinates of the target instance (c _x , c _y ) can be and the ground truth category labels to generate the projected mask supervision map Proj′∈R ^{w×h×(cls+1)} . Obviously, reducing the difference between the projected mask supervision map Proj′ and the category mask image SSI helps the model understand the two-dimensional The conversion process of pixel → three-dimensional space mapping → two-dimensional pixel reprojection. (2) If you only focus on the consistency between the real mask area and the predicted mask area, and the consistency between the projected mask area and the predicted mask area, the "cyclic mask misalignment" problem may occur: the network spends a lot of cost to learn distortions The geometric mapping relationship causes the model to deviate from the correct understanding of the target instance, and the deepening of this deviation will in turn cause the network to abandon the learned geometric mapping understanding, repeating the cycle and making it difficult to converge. In order to solve this problem, a projection mask is added The consistency between the code area and the real mask area constructs the cyclic geometric consistency constraint, as shown in Figure 11, and the cyclic geometric consistency constraint loss function is shown in formula (3-5).

As an example, the overall processing flow chart of the preset image processing model is shown in Figure 2, which mainly includes five parts: the center point representation of the complete three-dimensional information of the instance, the input end processing method of coordinate system separation, pose size and three-dimensional Shape point cloud coding, hierarchical feature fusion network based on context clustering, joint optimization of confidence geometric consistency constraints and auxiliary three-dimensional information; the implementation method is as follows: by given a single RGB-pixel with a pixel width of w and a pixel height of h- D image input (I∈R ^w×h×3 , D∈R ^w×h ). In the inference stage, the heat threshold is first calculated from the target Gaussian heat value map Y∈[0,1] ^w×h×1 The local peak coordinates are used as the target instance center point (c _x , c _y ), and secondly, from the overall three-dimensional information parameter map The complete and complete three-dimensional information O _3d (c _x ,c _y ) of the target instance is sampled, and the three-dimensional point cloud decoder MLPs-Decoder is used for inference to obtain the reconstructed point cloud P. Finally, from O _3d (c _x ,c Extract the coordinate axes a _x , a _z , the confidence level c _x and c _z , c _cd , and the one-dimensional true scale factor s from y ), extract D(c _x , c _{y )} from the HRD, and use a _x , a _z , c _x , c _z calculates the predicted rotation matrix R, and calculates the predicted translation vector t∈R ³ from t _cd ′, D (c _x , c _y ), c _cd , completing the inference of the target instance in the real scene from the two-dimensional image plane The overall three-dimensional understanding task of reconstructing the point cloud P ^recon = [R|t]*s*P, that is, multi-target category-level pose estimation and three-dimensional shape reconstruction based on a single RGB-D image.

This application provides a three-dimensional image generation method, device, equipment and storage medium. Compared with related technologies, a single-stage scheme that directly infers the overall three-dimensional information of multiple targets to detect target instances cannot handle the intra-class differences of target instances well, resulting in poor performance in target detection and spatial positioning. In this application, the original RGB image data is obtained; the original RGB image data is input to a preset image processing model, and based on the preset image processing model, the pixel coordinate system data and camera coordinates of the original RGB image data are compared System data are fused to obtain a low-resolution depth image; wherein the camera coordinate system data includes the depth image information of the original RGB image data and the two-dimensional camera coordinate data, and the depth image information is not used as the original input, Instead, it is used for coarse-grained synthesis of images; the low-resolution depth image is subjected to three-dimensional conversion to obtain a target three-dimensional image. In this application, by acquiring the original RGB image data, the original RGB image data is divided into camera coordinate system data and pixel coordinate system data for double-end input, and the depth image information of the original RGB image data is used for coarse-grained synthetic images, Instead of being used as the original input, this avoids the network performing relevant operations on each feature network on the original input image, destroying the hidden three-dimensional structure in the image, improving the performance of target detection and spatial positioning, and then accurately detecting the detected image data. Perform three-dimensional reconstruction.

Further, based on the first embodiment of the present application, another embodiment of the present application is provided. In this embodiment, the pixel coordinate system data and the camera coordinate system data of the original RGB image data are fused, The steps to obtain low-resolution depth images include:

Step C1: Perform vector calculation on the original RGB image data to obtain the first normal vectors of multiple pixels in the original RGB image data;

As an example, the first normal vector is the normal vector of the depth image in the original RGB image data,

As an example, the three-dimensional visual projection process refers to establishing the corresponding relationship between the target object from the three-dimensional coordinates of the real world to the two-dimensional pixel points of the imaging plane through the camera imaging geometric model. When the world coordinate system is defined as a coordinate system centered on the target instance , the process of projecting the target instance in the real three-dimensional scene to the two-dimensional image plane is shown in formula (3-3). This process describes a point (x _w , y _w , z _w ) in the coordinate system of the target instance (first It is rotated by the rotation matrix R∈SE(3), then translated to the position (x _c , y _c , z _c ) in the camera coordinate system by the three-dimensional translation vector t∈R ³ , and finally projected to on the pixel at image plane (u,v).

Step C2: Perform clustering processing on the coordinate system point sets of the pixel coordinate system data and the camera coordinate system data to obtain cluster feature data;

As an example, clustering feature data is obtained by aggregating features of different preferences in the input data through similarity measurement.

Step C3: Perform feature fusion on the first normal vector and the cluster feature data to obtain a low-resolution depth image.

As an example, when generating a low-resolution depth image, two sets of point sets in the pixel coordinate system space and the camera coordinate system space are extracted from a single RGB-D image through the input-side processing method of coordinate system separation: The pixel coordinate system point set P _l ∈R ^5×n consists of the normal vector CNR∈R ^w×h×3 of the RGB image and the two-dimensional coordinates UV∈R ^w×h×2 of the pixel plane; it is composed of the normal vector DNR∈ of the depth image The camera coordinate system point set P _r ∈ R ^5×n composed of ^{R w×h×3} and the camera plane two-dimensional coordinates XY∈R ^w×h×2 is then aggregated according to the different features of each point set to classify the data. , so that the extracted features can better reflect the overall differences between each cluster, rather than the matching in appearance and shape, so as to accurately restore the three-dimensional scene.

As an example, in the fusion process, the fusion operation using the spatial-channel attention module can establish the feature mapping of pixel coordinate system information and camera coordinate system information while avoiding the hierarchical clustering method of fixed center point sampling. Handling early corruption of implicit real space geometric information.

Wherein, the step of clustering the coordinate system point sets of the pixel coordinate system data and the camera coordinate system data to obtain cluster feature data includes:

Step D1: Perform feature extraction on the coordinate system point set of the pixel coordinate system data and the camera coordinate system data to obtain pixel coordinate feature data and camera coordinate feature data;

Step D2: Perform feature aggregation on multiple point set feature data based on the similarity between the pixel coordinate feature data and the camera coordinate feature data to obtain cluster feature data.

As an example, before classifying target instances in multiple image data, it is necessary to perform feature extraction on the data to obtain corresponding feature data.

As an example, the feature extraction process is shown in Figure 9. The most significant feature of the output hierarchical features {F ₁ , F ₂ , F ₃ , F ₄ } is that the feature map generation process is bottom-up, and as As the depth of the network increases, the resolution of the generated feature map is lower and the dimension of the feature channel is larger. In order to complete the learning process of semantic information from concrete to abstract, from dispersion to enrichment, a feature pyramid network is constructed to transform the output hierarchical features { F ₁ , F ₂ , F ₃ , F ₄ } are semantic hierarchical features {P ₁ , P ₂ , P ₃ , P ₄ }. After obtaining semantic hierarchical features {P ₁ , P ₂ , P ₃ , P ₄ } Finally, CoCFusion uses different task processing networks Depth Head, Seg Head, and 3D-info maps Head to use these hierarchical features for the pixel-level output expected by different tasks.

As an example, the process of aggregating features through similarity measurement is to classify each feature data according to the degree of similarity, thereby obtaining a set of multiple clusters, in which the data in each cluster has the highest similarity, and the cluster The degree of similarity between them is small.

In this embodiment, the similarity measure method is used to classify feature data, so that the neural network pays more attention to the overall difference between clusters rather than the detailed matching of appearance and shape, and uses the fusion operation of the spatial-channel attention module. While establishing feature mapping of pixel coordinate system information and camera coordinate system information, it can avoid the destruction of implicit real spatial geometric information in the early stage of hierarchical processing by the clustering method of fixed center point sampling.

Further, based on the first embodiment and the second embodiment of the present application, another embodiment of the present application is provided. In this embodiment, vector calculation is performed on the original RGB image data to obtain the original RGB image data. The steps for determining the first normal vectors of multiple pixels in image data include:

Step E1, perform data enhancement processing on the original RGB image data to obtain first enhanced data;

As an example, in a single-stage scheme without pre-segmentation, the detection task of target instances of interest cannot be completed well only by relying on imprecise and missing depth information. Accurate color features must be introduced in order to detect target instances. To find a balance between detection and intra-class differences, the input-side processing method of coordinate system separation uses enhanced image normal vectors to replace the original RGB values.

As an example, the data enhancement process is: data enhancement is performed on the original RGB image through color jitter technology (ColorJitter), and its brightness, contrast, saturation and hue are randomly changed.

Step E2: Perform two-dimensional gradient calculation on the first enhanced data to obtain a gradient calculation value;

Step E3: Select the maximum value among the gradient calculation values, and determine the maximum value among the gradient calculation values as the first normal vector of each pixel in the first enhancement data.

As an example, the value obtained by performing two-dimensional gradient calculation on the enhanced RGB image sub-channel is the gradient calculation value. At the same time, the maximum value among the gradient calculation values is taken as the final normal vector of each pixel point/first Normal vector, where the enhanced RGB image input in the pixel coordinate system space and the coordinate data display are shown in Figures 5 and 6.

In this embodiment, by performing data enhancement on the original RGB image data and using the enhanced image normal vector to replace the original RGB value, the image processing model can more easily handle differences within the target object class in order to classify multiple target instances.

As an example, this application conducts a comprehensive evaluation of the overall three-dimensional understanding ability of the CoCFusion algorithm on the NOCS public data set, analyzes the performance of the algorithm in the fields of category-level pose estimation and three-dimensional scene reconstruction, and involves comparison with existing baseline methods. There are four aspects: effectiveness of innovation points, efficiency of reasoning and qualitative results. The NOCS dataset is the most widely used evaluation benchmark in the field of class-level pose estimation. Regarding the pre-training dataset CAMERA, it is generated from 1085 rendering instances from the ShapeNetCore-CAD dataset under random views of real environment backgrounds. However, it should be noted that the NOCS data set only has six target categories of interest: bottles, bowls, cameras, cans, mugs, and laptops. The CAMERA data set contains a total of 300K synthetic rendering images. The present invention divides 275K data into a training set, and the remaining 25K data containing 184 different target instances are used for verification. Regarding the real-world data set REAL, it consists of multiple data containing different target instances. composed of real scenes.

Baseline method: This method compares six baseline methods with CoCFusion to demonstrate the effectiveness of the designed algorithm. The first five are multi-stage category-level overall three-dimensional understanding technology, and the sixth is a single-stage category-level overall three-dimensional understanding technology. (1)NOCS: The projection of NOCS coordinates on the two-dimensional image is inferred through the extended Mask R-CNN model, and then the Umeyama algorithm and RANSAC algorithm are used to solve the depth-projection similarity transformation for pose inference. (2)Synthesis: Combining the gradient-based fitting process with a parametric neural image synthesis model to implicitly represent the appearance, shape, and pose of the entire object category. (3) Metric Scale: The NOCS method is extended by separately predicting the shape of the metric scale and the projection of the NOCS space. (4) SPD: Use regular target point clouds and predict the prior deformation of the three-dimensional shape of this category based on the observed RGB-D information. (5) CASS: Use variational autoencoders to learn implicit shape expressions, and use convolutional networks and 3D point cloud reconstruction networks to directly regress implicit shapes and poses from segmented area and target area point clouds. (6) CenterSnap: Treat the target instance as the center point of its two-dimensional image, and propose the first single-stage solution to detect the target instance and simultaneously infer its six-degree-of-freedom pose, three-dimensional shape, and true size. The comparison results are as follows: Shown:

Table 4-2 Quantitative comparison of three-dimensional target detection and six-degree-of-freedom pose estimation on the NOCS data set

In terms of 3D target detection, CoCFusion has made some progress compared to the single-stage solution CenterSnap, but there are still gaps in more multi-stage solutions, and its performance on the REAL275 data set is inferior to the CAMERA25 data set. This is understandable because it is based on the target area. The segmentation method constrains the spatial location of instances, and the smaller the data set, the more obvious this effect is. In terms of six-degree-of-freedom pose estimation, CoCFusion showed excellent performance on the REAL275 data set: the mAP value at 5°5cm was 25.3%, the mAP value at 5°10cm was 28.4%, and the mAP value at 10°5cm The value is 55.2%, and the mAP value at 10°10cm is 62.2%. Compared with CenterSnap, it has achieved absolute improvements of 3.5%, 4.4%, 8.7% and 8.0%, and achieved better performance when the rotation error tolerance is large. .

Referring to Figure 3, Figure 3 is a schematic diagram of the equipment structure of the hardware operating environment involved in the embodiment of the present application.

As shown in Figure 3, the three-dimensional image generation device may include: a processor 1001, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize connection communication between the processor 1001 and the memory 1005.

Optionally, the three-dimensional image generation device may also include a user interface, a network interface, a camera, an RF (Radio Frequency, radio frequency) circuit, a sensor, a WiFi module, and so on. The user interface may include a display screen (Display) and an input sub-module such as a keyboard (Keyboard). The optional user interface may also include standard wired interfaces and wireless interfaces. Network interfaces may include standard wired interfaces and wireless interfaces (such as WI-FI interfaces).

Those skilled in the art can understand that the structure of the three-dimensional image generating device shown in Figure 3 does not constitute a limitation on the three-dimensional image generating device. It may include more or fewer components than shown in the figure, or some components may be combined or different. component layout.

As shown in Figure 3, memory 1005, which is a storage medium, may include an operating system, a network communication module, and a three-dimensional image generation program. The operating system is a program that manages and controls the hardware and software resources of the three-dimensional image generation device, and supports the operation of the three-dimensional image generation program and other software and/or programs. The network communication module is used to implement communication between components within the memory 1005, as well as communication with other hardware and software in the three-dimensional image generation system.

In the three-dimensional image generation device shown in FIG. 3, the processor 1001 is used to execute the three-dimensional image generation program stored in the memory 1005 to implement the steps of any of the three-dimensional image generation methods described above.

The specific implementation of the three-dimensional image generation device of the present application is basically the same as the above-mentioned embodiments of the three-dimensional image generation method, and will not be described again here.

This application also provides a three-dimensional image generating device, which includes:

Acquisition module, used to obtain original RGB image data;

The first processing module is used to input the original RGB image data into a preset image processing model, and fuse the pixel coordinate system data of the original RGB image data and the camera coordinate system data based on the preset image processing model. Process to obtain a low-resolution depth image; wherein the camera coordinate system data includes the depth image information of the original RGB image data and the two-dimensional camera coordinate data, and the depth image information is not used as original input, but is used for Coarse-grained synthetic images;

A conversion module is used to perform three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image.

In a possible implementation of this application, the first processing module includes:

A calculation unit configured to perform vector calculation on the original RGB image data to obtain the first normal vectors of multiple pixels in the original RGB image data;

A processing unit configured to perform clustering processing on the coordinate system point sets of the pixel coordinate system data and the camera coordinate system data to obtain cluster feature data;

A fusion unit configured to perform feature fusion on the first normal vector and the cluster feature data to obtain a low-resolution depth image.

In a possible implementation of the present application, the processing unit includes:

An extraction subunit, used to perform feature extraction on the coordinate system point set of the pixel coordinate system data and the camera coordinate system data, to obtain pixel coordinate feature data and camera coordinate feature data;

The aggregation subunit is used to perform feature aggregation on multiple point set feature data based on the similarity between the pixel coordinate feature data and the camera coordinate feature data to obtain cluster feature data.

In a possible implementation of this application, the computing unit includes:

A processing subunit, used to perform data enhancement processing on the original RGB image data to obtain first enhanced data;

A calculation subunit, used to perform two-dimensional gradient calculation on the first enhanced data to obtain a gradient calculation value;

A selection subunit is used to select the maximum value among the gradient calculation values, and determine the maximum value among the gradient calculation values to be the first normal vector of each pixel in the first enhancement data.

In a possible implementation of this application, the conversion module includes:

A generation unit configured to generate a target Gaussian heat value map based on multiple target examples in the original RGB image data;

The reconstruction unit is configured to perform three-dimensional point cloud reconstruction based on the aggregated features in the low-resolution depth image and the target Gaussian heat value map to obtain the target three-dimensional image.

In a possible implementation of this application, the conversion module further includes:

A determination unit configured to calculate the center coordinates of multiple target instances in the target Gaussian heat value map based on the target Gaussian heat value map, and determine the overall three-dimensional information parameter map based on the multiple target instance center coordinates;

An extraction unit is used to extract complete three-dimensional information of all target instances in the overall three-dimensional information parameter map.

In a possible implementation of the present application, the device further includes:

A calculation module configured to calculate the center distance estimation result based on the center point data of the target instance of the target three-dimensional image on the two-dimensional coordinate system;

The second processing module is used to perform consistency constraint processing on the regression result of the center distance estimation result and the overall three-dimensional information parameter map, so that the input target instance converges.

It should be noted that, as used herein, the terms "include", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or system that includes a list of elements not only includes those elements, but It also includes other elements not expressly listed or that are inherent to the process, method, article or system. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

The above serial numbers of the embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product that is essentially or contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM) as mentioned above. , magnetic disk, optical disk), including several instructions to cause a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of this application.

The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of the present application may be directly or indirectly used in other related technical fields. , are all equally included in the patent protection scope of this application.

Claims (10)

1. A method of generating a three-dimensional image, the method comprising the steps of:

Acquiring original RGB image data;

inputting the original RGB image data into a preset image processing model, and carrying out fusion processing on pixel coordinate system data and camera coordinate data of the original RGB image data based on the preset image processing model to obtain a low-resolution depth image; wherein the camera coordinate data includes depth image information of the original RGB image data and two-dimensional camera coordinate data, and the depth image information is not used as an original input but for a coarse-grained synthetic image;

and performing three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image.

2. The three-dimensional image generation method according to claim 1, wherein the step of fusing pixel coordinate system data of the original RGB image data with camera coordinate data to obtain a low-resolution depth image comprises:

vector calculation is carried out on the original RGB image data to obtain first normal vectors of a plurality of pixel points in the original RGB image data;

clustering the pixel coordinate system data and the coordinate system point set of the camera coordinate system data to obtain clustering characteristic data;

And carrying out feature fusion on the first normal vector and the clustering feature data to obtain a low-resolution depth image.

3. The three-dimensional image generating method according to claim 2, wherein the step of clustering the pixel coordinate system data and the coordinate system point set of the camera coordinate system data to obtain cluster feature data includes:

extracting features of the pixel coordinate system data and the coordinate system point set of the camera coordinate system data to obtain pixel coordinate feature data and camera coordinate feature data;

and carrying out feature aggregation on the feature data of the plurality of point sets according to the similarity degree of the feature data of the pixel coordinates and the feature data of the camera coordinates to obtain cluster feature data.

4. The method of generating a three-dimensional image according to claim 2, wherein the step of performing vector calculation on the original RGB image data to obtain a first normal vector of a plurality of pixels in the original RGB image data comprises:

performing data enhancement processing on the original RGB image data to obtain first enhancement data;

performing two-dimensional gradient calculation on the first enhancement data to obtain a gradient calculation value;

And selecting the maximum value in the gradient calculated values, and determining the maximum value in the gradient calculated values as a first normal vector of each pixel point in the first enhancement data.

5. The three-dimensional image generation method according to claim 1, wherein the step of three-dimensionally converting the low-resolution depth image to obtain the target three-dimensional image comprises:

generating a target stell value map according to a plurality of target examples in the original RGB image data;

and carrying out three-dimensional point cloud reconstruction according to the aggregation characteristics in the low-resolution depth image and the target Gaussian heat value image to obtain a target three-dimensional image.

6. The three-dimensional image generating method according to claim 5, wherein after the step of generating the target gaussian heat value map from the plurality of target examples in the original RGB image data, the method comprises:

calculating to obtain a plurality of target instance center coordinates in the target Gaussian heat value diagram according to the target Gaussian heat value diagram, and determining an overall three-dimensional information parameter diagram based on the plurality of target instance center coordinates;

and extracting the complete three-dimensional information of all target examples in the overall three-dimensional information parameter map.

7. The three-dimensional image generation method according to claim 6, wherein after the step of three-dimensionally converting the low-resolution depth image to obtain the target three-dimensional image, the method comprises:

calculating to obtain a center distance estimation result according to the center point data of the target instance of the target three-dimensional image on a two-dimensional coordinate system;

and carrying out consistency constraint processing on the center distance estimation result and the regression result of the integral three-dimensional information parameter graph so as to enable the input target instance to be converged.

8. A three-dimensional image generation apparatus, characterized in that the three-dimensional image generation apparatus includes:

the acquisition module is used for acquiring the original RGB image data;

the first processing module is used for inputting the original RGB image data into a preset image processing model, and carrying out fusion processing on pixel coordinate system data and camera coordinate coefficient data of the original RGB image data based on the preset image processing model to obtain a low-resolution depth image; wherein the camera coordinate data includes depth image information of the original RGB image data and two-dimensional camera coordinate data, and the depth image information is not used as an original input but for a coarse-grained synthetic image;

And the conversion module is used for carrying out three-dimensional conversion on the low-resolution depth image to obtain a target three-dimensional image.

9. A three-dimensional image generating apparatus, characterized in that the apparatus comprises: a memory, a processor, and a three-dimensional image generation program stored on the memory and executable on the processor, the three-dimensional image generation program configured to implement the steps of the three-dimensional image generation method of any one of claims 1 to 7.

10. A computer storage medium, characterized in that the computer storage medium has stored thereon a three-dimensional image generation program which, when executed by a processor, implements the steps of the three-dimensional image generation method according to any one of claims 1 to 7.