CN116245940A - Category-level six-degree-of-freedom object pose estimation method based on structure difference perception - Google Patents

Abstract

The invention relates to a category-level six-degree-of-freedom object pose estimation method based on structural difference perception, which comprises the following steps: inputting the depth map into a target detection segmentation network for recognition, obtaining an observation point cloud of an object instance according to a recognition result, and selecting a category prior corresponding to the target object based on the observation point cloud of the object instance; extracting the observation point cloud and the category priori features to obtain instance geometric features and category geometric features; inputting the instance geometric features and the category geometric features into an information interaction enhancement module to obtain enhanced instance geometric features and category geometric features; then, the semantic and geometric information are fused through the semantic dynamic fusion module, so that instance fusion characteristics and category fusion characteristics are obtained; obtaining an instance NOCS model based on the category fusion features; and matching the example NOCS model with the observation point cloud through a matching network, and calculating according to the similarity to obtain the 6D pose and the size of the target object. The method and the device can improve the accuracy of 6D pose estimation.

Description

Class-level six-degree-of-freedom object pose estimation method based on structural difference perception

technical field

The invention relates to the technical field of computer vision, in particular to a category-level six-degree-of-freedom object pose estimation method based on structural difference perception.

Background technique

Estimating the six degrees of freedom (6DegreeofFreedom, 6D) pose of a real object from a picture is a very critical task, that is, estimating the position and orientation of the object in the camera coordinate system, which consists of a three-dimensional rotation matrix and a three-dimensional translation vector. . Object 6D pose estimation tasks are widely used in many real-world scenarios, such as 3D scene understanding, robot grasping, virtual reality and augmented reality. The 6D pose estimation task can be divided into two categories according to the level of the estimated object: 1. Instance-level 6D pose estimation for a specific object; 2. Category-level 6D pose estimation for the same type of object. The instance-level 6D pose estimation task needs to know its position in the world coordinate system in advance when calculating the pose of the object. Generally, the center of the world coordinate system falls on the center of the object, that is, its CAD model. For new objects without a defined CAD model in the real scene, the instance-level 6D pose estimation algorithm cannot estimate the pose of the object, which seriously limits the application of the instance-level 6D pose estimation algorithm in real scenes. Therefore, to break the limitation of instance-level 6D pose estimation methods, the category-level 6D pose estimation task is proposed, which is able to estimate the 6D poses of different object instances in the same category even if some object instances do not have CAD models.

Wang et al. first proposed the concept of category-level object 6D pose estimation tasks. In order to solve the problem of lack of CAD models when estimating the 6D pose of objects, they introduced a normalized object coordinate space (NOCS)—a category under A shared canonical representation of all possible object instances, by first reconstructing the object instance in NOCS, and then computing the pose transformation relation of the object instance from the NOCS space to the camera coordinate system, that is, the 6D pose of the object. Since different object instances under the same category may have large structural differences, it is a very difficult task to reconstruct their NOCS models, which is the difficulty of category-level object 6D pose estimation tasks. In response to this problem, SPD proposes to learn a category prior for each category, and then deform the category prior according to different object instances to reconstruct the NOCS model of the object instance, which further increases the accuracy of pose estimation, but the category prior Fuzzy information makes the reconstructed NOCS model imprecise.

Contents of the invention

The technical problem to be solved by the present invention is to provide a category-level six-degree-of-freedom object pose estimation method based on structural difference perception, which can improve the accuracy of 6D pose estimation.

The technical solution adopted by the present invention to solve the technical problem is to provide a category-level six-degree-of-freedom object pose estimation method based on structural difference perception, including the following steps:

Input the depth map to the target detection segmentation network to obtain the image block of the target object and the segmentation mask of the target object;

Obtaining the observation point cloud of the object instance according to the segmentation mask of the target object and the depth map, and selecting the category prior corresponding to the target object based on the observation point cloud of the object instance;

Extract the features of the observed point cloud and category priors, and obtain the instance geometric features and category geometric features;

Input the instance geometric features and category geometric features into the information interaction enhancement module, use the information interaction enhancement module to implicitly model the geometric differences between the instance geometry features and category geometry features, and analyze the instance geometry features and categories The geometric features are supplemented, and the enhanced instance geometric features and category geometric features are obtained;

Input the geometric difference between the instance geometric feature and the class geometric feature, the enhanced instance geometric feature and the class geometric feature to the semantic dynamic fusion module, and carry out the fusion of semantic and geometric information through the semantic dynamic fusion module to obtain instance fusion Feature and category fusion features;

Sending the category fusion feature into the deformation network to obtain a deformation field, and using the deformation field to deform the category prior to obtain an instance NOCS model;

Match the example NOCS model and the observed point cloud through the matching network, and calculate the 6D pose and size of the target object according to the similarity.

The target detection segmentation network uses a Mask-RCNN network.

Convolutional neural network and PointNet++ network are used to realize the extraction of observed point cloud and category prior features.

The information interaction enhancement module includes: a fully connected layer, which is used to map the instance geometric features and category geometric features to the same feature subspace; a matrix multiplication unit, which is used to map the instance geometry features to the same feature subspace. The matrix multiplication operation is performed on the feature and the category geometric feature to obtain the structural relationship matrix between the instance geometric feature and the category geometric feature; a normalization unit is used to normalize the structural relationship matrix to a weight coefficient; a weighted summation unit, It is used to weight and sum the geometric projection features in the structural relationship matrix by using the weight coefficient to obtain the relationship between the instance geometric features and the category geometric features; the multi-layer perceptron is used to compare the geometric differences with the example Geometric features and class geometric features are fused to obtain enhanced instance geometric features and class geometric features.

The semantic dynamic fusion module adopts a pixel-level fusion strategy for the enhanced instance geometric features to realize the corresponding point fusion module to explore the internal mapping between data sources and obtain instance fusion features. For the enhanced category geometric features and instance geometric features of different individuals , using the geometric difference between the instance geometry feature and the class geometry feature to dynamically adjust the enhanced instance geometry feature, and fusing the adjusted enhanced instance geometry feature with the enhanced category geometry feature to obtain a category fusion feature.

Beneficial effect

Due to the adoption of the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art: the present invention uses the structural difference between the object instance and the category prior to enhance the learning of shape information within the class, further through The semantic dynamic fusion module dynamically adjusts the semantic information according to the geometric relationship between the object instance and the class prior, and then fuses with the enhanced class prior to dynamically supplement the lack of geometric information to improve the robustness to noise.

Description of drawings

1 is a flowchart of a method for estimating the pose of a category-level six-degree-of-freedom object based on structural difference perception according to an embodiment of the present invention;

2 is a schematic diagram of an information interaction enhancement module in an embodiment of the present invention;

Fig. 3 is a schematic diagram of a semantic dynamic fusion module in an embodiment of the present invention;

Figure 4 is a schematic diagram of observed point clouds of different object instances;

Fig. 5 is a comparison of the results of the embodiment of the present invention and the SPD method.

Detailed ways

Below in conjunction with specific embodiment, further illustrate the present invention. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

The embodiment of the present invention relates to a category-level six-degree-of-freedom object pose estimation method based on structural difference perception. The dynamic fusion module dynamically adjusts the semantic information according to the geometric relationship between the object instance and the category prior, and then fuses with the enhanced category prior to dynamically supplement the lack of geometric information. As shown in Figure 1, the following steps are included:

Step 1. Input the depth map to the target detection and segmentation network to obtain the image block of the target object and the segmentation mask of the target object. In this step, an existing target detection and segmentation network can be used to obtain the image block of the target object and its segmentation mask, for example, a Mask-RCNN network can be used.

Step 2: Obtain the observed point cloud of the object instance according to the segmentation mask of the target object and the depth map, and select the category prior corresponding to the target object based on the observed point cloud of the object instance.

Step 3, extract the features of the observation point cloud and category priors, and obtain the instance geometric features and category geometric features. In this step, the convolutional neural network and PointNet++ can be used to extract image semantic features and point cloud geometric features to obtain instance geometric features and category geometric features.

Step 4, input the instance geometric features and class geometric features into the information interaction enhancement module, use the information interaction enhancement module to implicitly model the geometric differences between the instance geometric features and the category geometric features, and analyze the instance geometry Feature and class geometric features are supplemented, and enhanced instance geometric features and class geometric features are obtained.

The information interaction enhancement module in this step aims to learn the structural relationship between instance point clouds and category priors to help construct their structural difference information at the feature level. It supplements the original geometric features with features of structural differences, so that the enhanced geometric features include the unique individuality of instance structures and the general commonality of class priors. On the one hand, the enhanced category geometry features can reconstruct a more accurate instance NOCS model due to the complementarity of instance structure properties. On the other hand, instance geometry features increase the commonality of category shapes, so that the reconstructed correspondence matrix better associates the observed point cloud with the NOCS model. In addition, since the class prior and the geometric differences between different instances under the same class are different, adopting the information interaction augmentation module is able to adapt to instances of various shapes that have not been seen before, greatly increasing the generalization of this embodiment .

The structure of the information interaction enhancement module is shown in Figure 2, including: a fully connected layer, which is used to map the instance geometric features and category geometric features to the same feature subspace; a matrix multiplication unit, which is used to map to the same The instance geometric features and category geometric features of the feature subspace are matrix multiplied to obtain the structural relationship matrix between the instance geometric features and the category geometric features; the normalization unit is used to normalize the structural relationship matrix to a weight coefficient ; The weighted sum unit is used to adopt the weight coefficient to carry out weighted summation to the geometric projection features in the structure relationship matrix to obtain the relationship between the instance geometric features and the category geometric features; the multi-layer perceptron is used to combine the geometric features The difference is fused with the instance geometric features and category geometric features respectively to obtain enhanced instance geometric features and category geometric features.

In this way, for instance geometric features and category geometric features, they can be mapped to the same feature subspace using a fully connected network layer, and then their structural relationship matrices can be obtained through matrix multiplication operations. Then, the structural relationship matrix is normalized to weight coefficients, and the geometric projection features are weighted and summed to obtain structural difference features. Finally, the multi-layer perceptron is used to fuse the original geometric features and structural difference features to obtain enhanced geometric features.

Step 5, inputting the geometric difference between the instance geometric feature and the class geometric feature, the enhanced instance geometric feature and the class geometric feature to the semantic dynamic fusion module, and performing the fusion of semantic and geometric information through the semantic dynamic fusion module, Get instance fusion features and category fusion features.

As shown in Figure 4, the object instance point cloud obtained through the target detection and segmentation model may contain certain noise points. When the influence of these noise points is transmitted to the category prior, it will theoretically have a negative impact on the reconstruction accuracy of the NOCS model, resulting in a deviation in the correspondence between the object instance point cloud and its NOCS model. In order to solve this problem, this embodiment designs a semantic dynamic fusion module, which improves the robustness of the network to noise points by fully fusing geometric and semantic information.

Figure 3 shows the semantic dynamic fusion module. The semantic dynamic fusion module adopts the pixel-level fusion strategy to realize the corresponding point fusion module for the enhanced instance geometric features to explore the internal mapping between data sources and obtain instance fusion features. For the enhancement of different individuals category geometric features and instance geometric features, using the geometric difference between the instance geometric features and category geometric features to dynamically adjust the enhanced instance geometric features, and fusing the adjusted enhanced instance geometric features with the enhanced category geometric features, Get category fusion features. That is to say, this implementation mode uses the method in DenseFusion for reference, and uses a pixel-level fusion strategy to realize the corresponding point fusion module to explore the internal mapping between data sources. For category geometric features and instance semantic features from different individuals, since there is no pixel-level correspondence between them, the pixel-level fusion strategy cannot be used directly, so this embodiment adopts two different fusion strategies. The first is the general idea of feature fusion, splicing the two together and then fusing them through the MLP function, which is called direct fusion. Although the direct fusion strategy can improve performance by absorbing semantic information, it is still insufficiently considered for cross-individual problems. For this reason, this embodiment also designs a semantic fusion strategy, dynamically adjusts the semantic features of the instance according to the structural relationship matrix of the instance and the category, and then fuses it with the geometric features of the category.

Step 6: Send the category fusion features into the deformation network to obtain a deformation field, and use the deformation field to deform the category prior to obtain an instance NOCS model.

Step 7: Match the instance NOCS model with the observed point cloud through the matching network, and calculate the 6D pose and size of the target object according to the similarity.

As shown in Figure 5, one of the two sets of frame lines outside the target object is the true value, and the other is the predicted result. Compared with the SPD method, it can be seen that the pose estimated by the method of this embodiment is more accurate, especially for categories such as cameras (objects pointed by arrows in the figure) with relatively large shape changes, the estimation results of the method of this embodiment are It is much better than the estimation result of the SPD method, which also proves that the method of this embodiment can well deal with the problem of intra-class shape variation.

It is not difficult to find that the present invention utilizes the structural difference between the object instance and the category prior to enhance the learning of shape information within the class, and further dynamically adjusts the semantic information according to the geometric relationship between the object instance and the category prior through the semantic dynamic fusion module, and then and The enhanced class prior fusion dynamically complements the lack of geometric information to improve robustness to noise.

Claims (5)

1. The category-level six-degree-of-freedom object pose estimation method based on structural difference perception is characterized by comprising the following steps of:

inputting the depth map into a target detection segmentation network to obtain an image block of a target object and a segmentation mask of the target object; obtaining an observation point cloud of an object instance according to the segmentation mask of the target object and the depth map, and selecting a category prior corresponding to the target object based on the observation point cloud of the object instance;

extracting the observation point cloud and the category priori features to obtain instance geometric features and category geometric features;

inputting the instance geometric features and the category geometric features into an information interaction enhancement module, implicitly modeling geometric differences between the instance geometric features and the category geometric features through the information interaction enhancement module, and supplementing the instance geometric features and the category geometric features to obtain enhanced instance geometric features and category geometric features;

inputting the geometric difference between the instance geometric features and the category geometric features, the enhanced instance geometric features and the category geometric features into a semantic dynamic fusion module, and fusing semantic and geometric information through the semantic dynamic fusion module to obtain instance fusion features and category fusion features;

the category fusion features are sent to a deformation network to obtain a deformation field, and the category prior is deformed by using the deformation field to obtain an instance NOCS model;

and matching the example NOCS model with the observation point cloud through a matching network, and calculating according to the similarity to obtain the 6D pose and the size of the target object.

2. The method for estimating the pose of the class-level six-degree-of-freedom object based on the structural difference perception according to claim 1, wherein the object detection segmentation network adopts a Mask-RCNN network.

3. The method for estimating the pose of the category-level six-degree-of-freedom object based on the structural difference perception according to claim 1, wherein the feature of the observation point cloud and the category prior is extracted by adopting a convolutional neural network and a PointNet++ network.

4. The method for estimating the pose of the category-level six-degree-of-freedom object based on the structural difference perception according to claim 1, wherein the information interaction enhancement module comprises: a full connection layer for mapping the instance geometric features and the category geometric features to the same feature subspace, respectively; the matrix multiplication unit is used for carrying out matrix multiplication operation on the instance geometric features and the category geometric features mapped to the same feature subspace to obtain a structural relation matrix between the instance geometric features and the category geometric features; the normalization unit is used for normalizing the structural relation matrix into a weight coefficient; the weighted summation unit is used for carrying out weighted summation on the geometric projection features in the structural relation matrix by adopting the weight coefficient to obtain the example geometric features and the category geometric features; and the multi-layer perceptron is used for respectively fusing the geometric difference with the instance geometric feature and the category geometric feature to obtain the enhanced instance geometric feature and the category geometric feature.

5. The method for estimating the pose of the class-level six-degree-of-freedom object based on the structural difference perception according to claim 1, wherein the semantic dynamic fusion module adopts a pixel-level fusion strategy to realize a corresponding point fusion module for the enhanced instance geometric feature to explore internal mapping between data sources, so as to obtain an instance fusion feature, adopts geometric differences between the instance geometric feature and the class geometric feature to dynamically adjust the enhanced instance geometric feature for the enhanced class geometric feature and the instance geometric feature of different individuals, and fuses the adjusted enhanced instance geometric feature and the enhanced class geometric feature to obtain the class fusion feature.

Images