CN115880690B - Method for quickly labeling objects in point cloud under assistance of three-dimensional reconstruction - Google Patents

Abstract

The application relates to the field of artificial intelligence, in particular to a method for quickly marking objects in point clouds under the assistance of three-dimensional reconstruction. And then, according to the optimized RGB-D sample acquisition positions, automatically converting the object labels under the world coordinate system into the RGB-D sample coordinate systems. And obtaining the pose and the label of each object under each RGB-D sample. For each real object in the environment, the method can automatically obtain the object labels of the objects in different RGB-D samples only by manually marking once, and the workload of manual marking can be greatly reduced.

Description

A method for fast labeling of objects in point clouds with the aid of 3D reconstruction

technical field

The invention relates to the field of artificial intelligence, in particular to a method for quickly marking objects in a point cloud with the aid of three-dimensional reconstruction.

Background technique

Supervised learning is still one of the mainstream methods of deep learning algorithms, which is very data demanding and requires a large number of labeled samples for training and evaluation. For example, the popular target detection algorithm yolov4, yoloX uses about 40GB data set MS, COCO for training to detect objects from RGB images. 3D object detection such as VoteNet, Group-Free uses SUN, RGB-D datasets for training to obtain the ability to detect 3D objects from point clouds. Similarly, the huge data set Nuscene is also used to support the research of intelligent algorithms for autonomous driving.

In addition to public datasets, the application of algorithms in certain industries also requires the preparation of task-related datasets. Crowdsourcing is a popular method at present. Crowdsourcing organizes many people to perform manual labeling work at the same time. Finally, all marked samples constitute the final data set, but this work is time-consuming and expensive.

To solve this problem, it is very important to improve the efficiency of labeling. One class of methods exploits unsupervised or weakly supervised learning to accelerate object annotation. Features are extracted from RGB samples through an unsupervised method, a small number of samples are manually annotated based on these features, and then the labels are propagated to other unlabeled RGB samples. In addition, in order to reduce the difficulty of annotating objects in the point cloud, an existing method is to indirectly realize the annotation of the point cloud by annotating the objects in the aligned RGB image. The process is to only mark the 2D box of the object on the RGB image, and then run an automatic tagger, which can automatically estimate a high-quality 3D box from the sub-point cloud derived from the 2D box, so as to realize the use of automatic driving. Automatic labeling of objects in point clouds.

Although the above-mentioned existing methods have their own advantages, the collection of RGB-D samples in general indoor scenes is dense, and the fields of view of adjacent collection positions overlap significantly, and the same object in the environment will appear repeatedly in the field of view of multiple RGB-D samples. . In this case, traditional object labeling needs to label the same object multiple times in multiple RGB-D samples, which greatly increases the cost of unnecessary manual labeling.

Contents of the invention

In view of this, in order to solve the above technical problems, the present invention provides a method for quickly labeling objects in point clouds with the assistance of 3D reconstruction.

The present invention adopts following technical scheme:

A method for quickly labeling objects in a point cloud with the aid of three-dimensional reconstruction, comprising the following steps:

Obtain the RGB-D sample sequence to be labeled;

Construct the SE(3) pose graph according to the sequence of RGB-D sample collection positions, and use the matching relationship between RGB-D samples to construct the spatial constraints between the sampling positions, and then optimize the SE(3) pose graph to obtain the RGB-D The optimized collection position sequence of D samples; based on the optimized collection position sequence, the RGB-D sample sequence is fused into a global 3D map;

Use manual labeling to mark objects in the global 3D map;

According to the labeling results, the pose of each object in the world coordinate system is converted into the pose of the sensor coordinates corresponding to the collection position, and the pose of each object at each collection position, that is, the RGB-D sample, is obtained, and then compared with the object Classes and sizes together form object labels under RGB-D samples.

Further, after the acquisition of the RGB-D sample sequence to be labeled, the method for fast labeling objects in the point cloud further includes: realizing 3D reconstruction of the environment by constructing and optimizing the SE(3) pose graph.

Further, the SE(3) pose graph is constructed according to the RGB-D sample collection position sequence, including:

The SE(3) pose graph is as follows:

in, and ε are the abbreviations of Vertex and Edge, respectively; v is the variable to be optimized in the SE(3) pose graph, and is the collection position sequence of all RGB-D samples; ε is the constraint set of the optimization problem, and each constraint/ > denote two vertices s _* and /> Spatial constraints between points; set RGB-D sample sequence Q={f _i |i=1...n}, where f _i is an RGB-D sample, including an RGB color image I _i and a depth image, etc. valence 3D point cloud D _i , f _i =(I _i ,D _i ); the collection position sequence is P={s _i |i=0,...,n}, s _i is a 6-DOF position and attitude in a 3D space point, represented by the SE(3) variable.

Further, the space constraint between the sampling positions is constructed by using the matching relationship between the RGB-D samples, and then the SE(3) pose graph is optimized to obtain the optimized acquisition position sequence of the RGB-D samples, including:

Spatial constraints between acquisition locations are determined by point cloud registration:

where <p,q> represents D _* and pair of dots between, s′ _* and /> s _* and /> respectively initial value, /> As the initial value of the variable e;

Predict the collection position s _* and Offset between:

According to the above space constraints and predicted offsets, the optimization objective of the SE(3) pose graph is obtained as follows:

Among them, X={s ₁ ,s ₂ ,…,s _n } is a vector of parameters to be optimized, each as an error term The information matrix of ; use g2o, solve the above optimization problem and obtain the optimized acquisition point,

Further, the RGB-D sample sequence is fused into a global 3D map based on the optimized collection position sequence, including:

Transform all, RGB-D, samples into world coordinates:

All RGB-D samples are represented in the same world coordinate system and together constitute the 3D map M of the environment:

Further, the manual labeling method is used to mark objects in the global 3D map, including:

Load the above three-dimensional map M into the software for object labeling, and perform artificial object labeling. For any object o _j , the shape of the object is represented by a 3D bounding box (3D-BBox). The format of the object label is as follows:

Among them, c _j is the category of the object, S _j = [w _j , l _j , h _j ] is the size of the 3D-BBox, is the pose of the object in the world coordinate system.

Further, according to the labeling results, the pose of each object in the world coordinate system is converted into the pose of the sensor coordinates corresponding to the collection position, that is, the pose of each object in each sample is obtained, and then the RGB-D The object label corresponding to the sample, including:

Using the following matrix, the pose of the object o _j in the world coordinate system is expressed as the following matrix form:

in, is the position of the object in the world coordinate system, /> is the Euler angle Equivalent three-dimensional space rotation matrix;

Then, based on the optimized acquisition position of each RGB-D sample f _i Transform the pose of the object o _j in the global coordinate system to the pose in the sensor coordinates at the time of collection:

The above pose rotation matrix Contains a small rotation around the x and y axes, so through two transformations between the rotation matrix and Euler angles, the rotation matrix is simplified as follows:

Here, rot2Euler means to obtain a Euler angle equivalent to a rotation matrix, and Euler2rot means to obtain a rotation matrix equivalent to an Euler angle.

Finally, RGB-D is obtained, and the pose matrix of the object o _j in the sample f _i is as follows:

Finally, the parsed object tags are automatically obtained through the above method as follows:

in, is the pose of object o _j in vector form in RGB-D samples.

Further, the method for quickly labeling objects in point clouds with the aid of three-dimensional reconstruction also includes a step of label verification, specifically as follows:

For an RGB-D frame f _i , use Open3D point cloud segmentation technology to obtain subpoint cloud within

Computing sub-point clouds The volume S _j of the smallest convex hull of ;

The validity of the label is judged according to the volume V _j of the 3D-BBox representing the object o _j :

Among them, V _j = w _j l _j h _j , which means the volume of the object shape 3D-BBox; r is the ratio of the volume of the smallest convex hull of the sub-point cloud to the volume of the 3D-BBox, Indicates that under the current RGB-D sample, there is an object o _j and its label /> The reliability of , only if /> when label /> is valid and accepted as the label for an object o _j in an RGB-D sample f _i .

The beneficial effects of the present invention are as follows: first construct SE(3) pose graph according to the RGB-D sample sequence, and use the matching relationship between RGB-D samples to construct the spatial constraints between sampling positions, and then optimize the SE(3) position Pose map, obtain the optimized collection position sequence of RGB-D samples, based on the obtained collection position sequence, fuse the RGB-D sample sequence into a global 3D map, and then use manual labeling to mark objects in the global 3D map, Finally, according to the labeling results, the pose of each object in the world coordinate system is converted into the pose of the sensor coordinates corresponding to the acquisition position, and the pose of the object in each RGB-D sample is obtained, which becomes the same as that of the RGB-D sample. Corresponding object labels, in the present invention, each object that appears in different RGB-D samples multiple times only needs to be manually marked once in the fused 3D map, and the object in each RGB-D sample sequence can be automatically obtained. The labels in the system greatly reduce the cost of manual labeling.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings used in the embodiments:

FIG. 1 is a schematic diagram of the overall flow of a method for quickly labeling objects in a point cloud with the aid of 3D reconstruction provided by the present application.

Detailed ways

In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and/or components, but does not exclude one or more other Presence or addition of features, wholes, steps, operations, elements, components and/or collections thereof.

It should also be understood that the term "and/or" used in the description of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

As used in this specification and the appended claims, the term "if" may be construed, depending on the context, as "when" or "once" or "in response to determining" or "in response to detecting ". Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be construed, depending on the context, to mean "once determined" or "in response to the determination" or "once detected [the described condition or event] ]” or “in response to detection of [described condition or event]”.

In addition, in the description of the specification and the appended claims of the present application, the terms "first", "second", "third" and so on are only used to distinguish descriptions, and should not be understood as indicating or implying relative importance.

Reference to "one embodiment" or "some embodiments" or the like in the specification of the present application means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically stated otherwise. The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless specifically stated otherwise.

In order to illustrate the technical solution described in the present application, the following description will be given through specific implementation methods.

This embodiment provides a method for quickly labeling objects in point clouds with the assistance of three-dimensional reconstruction, which can create and optimize SE(3) pose graphs (Special Euclidean, group, for, n=3, SE(3)), which fuses sequences of RGB-D samples into a global 3D map. Then perform artificial object labeling in the 3D global map, and then automatically convert these labels into object labels in each RGB-D sample through label analysis, and finally achieve fast object labeling. Referring to FIG. 1 , the implementation steps of a method for quickly labeling objects in a point cloud with the aid of 3D reconstruction provided by this embodiment will be described below.

Step S1: Obtain the RGB-D sample sequence to be labeled:

As a specific application scenario, a custom-developed robot platform is used to collect data. The Kinect,v2 sensor is fixed above the robot at a height of 1.5 meters, and is used to collect RGB-D sample sequences to be marked. The odometry is obtained through the data of the robot kinematics model and the motor driver, and the odometry can provide the initial acquisition position sequence during the construction of the SE(3) pose graph. In order to reduce the side effects of robot shaking on 3D reconstruction, 6Dof is used to acquire poses to represent the trajectory of the sensor.

The coordinate system of the RGB-D sensor is x-rightward, y-downward and z-forward. Because the sensor is installed obliquely, in the point cloud represented by the sensor coordinate system, the ground is not parallel to the xoy plane, which does not meet the requirements of the labeling software labelCloud used. In order to solve this problem, it is necessary to define an x-rightward, y-forward, z-upward coordinate system first, and its xoy plane is parallel to the ground, so the ground in the processed point cloud is parallel to the xoy plane.

Step S2: Construct the SE(3) pose graph according to the RGB-D sample collection position sequence, and use the matching relationship between RGB-D samples to construct the spatial constraints between the sampling positions, and then optimize the SE(3) pose graph, Obtain the optimized acquisition position sequence of RGB-D samples; based on the optimized acquisition position sequence X ^* , fuse the RGB-D sample sequence into a global 3D map:

As a specific implementation, after obtaining the RGB-D sample sequence to be marked, it is necessary to perform three-dimensional reconstruction of the environment, as follows:

Set RGB-D sample sequence Q={f _i |i=1...n}, where f _i is an RGB-D sample, including an RGB color image I _i and a 3D point cloud D _i equivalent to a depth image , f _i =(I _i ,D _i ); then, one goal of 3D reconstruction is to optimize the acquisition position sequence P={s _i |i=0,…,n}, s _i is a 6-DOF position in 3D space and Attitude point, represented by SE(3) variable.

In order to achieve the above goals, based on the RGB-D sample sequence, the following process is used to construct the SE(3) pose graph:

The SE(3) pose graph is as follows:

in, and ε are the abbreviations of vertex Vertex and edge Edge respectively, representing vertex and edge respectively; /> is the variable to be optimized in the SE(3) pose graph, and is the collection position sequence set of all RGB-D samples; ε is the constraint set of the optimization problem, each constraint denote two vertices s _* and /> Spatial constraints between points.

In this embodiment, it is assumed that the sequence of acquisition positions is continuous, and the distance between adjacent acquisition positions is very small, so the overlap between adjacent RGB-D samples accounts for a considerable proportion of the RGB-D sensor field of view. Based on the above settings, a specific implementation process of optimizing the SE(3) pose graph according to the spatial constraints between RGB-D samples to obtain all the acquisition position sequences after the RGB-D samples are optimized is given as follows:

Determining the spatial constraints between spatially acquired poses through point cloud alignment:

where <p,q> represents D _* and pair of dots between, s′ _* and /> s _* and /> respectively initial value, /> as the initial value of the variable e for the optimization process. According to * and /> The above constraints are divided into two cases. , when /> When , the above constraint is the spatial constraint between two consecutive acquisition locations; conversely, if /> It is a spatial constraint between two long-spaced acquisition locations established by detecting closed loops of sensor trajectories.

Predict the collection position s _* and Offset between:

According to the above space constraints and predicted offsets, the optimization objective of the SE(3) pose graph is obtained as follows:

Among them, X={s ₁ ,s ₂ ,…,s _n } is a vector of parameters to be optimized, each as an error term The information matrix of ; using g2o, solve the above optimization problem and obtain an optimized sequence of acquisition positions,

The following is a specific implementation process of fusing the RGB-D sample sequence into a global 3D map based on the obtained collection position sequence:

Based on the optimized acquisition position sequence, convert all RGB-D samples to the world coordinate system:

All RGB-D samples are represented in the same world coordinate system and together constitute the 3D map M of the environment:

After the above operations, a global 3D map can be obtained.

Step S3: Use manual labeling to mark objects in the global 3D map:

After obtaining the global 3D map, use manual labeling to mark objects in the global 3D map. The type of marked objects can be flexibly performed according to the needs of specific tasks. In this embodiment, labelCloud software is used to label artificial objects in 3D, map M. The labelCloud software supports labeling objects in the point cloud in the form of 3D bounding boxes (3D-BBox). For any o _j , the label format is as follows:

Among them, c _j is the category of the object, S _j = [w _j , l _j , h _j ] is the size of the 3D bounding box 3D-BBox, (3D bounding , box), is the pose of the object in the world coordinate system.

This embodiment assumes that the object stands perpendicular to the horizontal, xoy, plane, so the rotation angle around the x and y axes is 0. Among the above three quantities of labels, the category c _j and size S _j are the basic attributes of the object, which do not change with the position; on the contrary, the attitude It is related to the coordinate system representing the object.

Step S4: According to the labeling results, convert the pose of each object in the world coordinate system to the pose in the sensor coordinates corresponding to the collection position, and obtain the pose of each object at each collection position, that is, the RGB-D sample , together with the object category and size constitute the object label under the RGB-D sample:

For the labeled results, the category and size do not change with the position of the object, which is the basic attribute of the object. However, object pose is related to the coordinate system representing the object. The object pose represented in the world coordinate system and the sensor coordinate system are different. In order to realize RGB-D, labeling of sample sequences, this embodiment converts the pose of each object in the world coordinate system to the pose in the sensor coordinates where its collection position is observed. For each object o _j , its pose is expressed in the form of the following matrix:

in, is the position of the object in the world coordinate system, /> is given by the Euler angles Transformed into a rotation matrix in 3D space.

Based on the optimized acquisition position of each RGB-D sample f _i Convert the pose of the object o _j in the global coordinate system to the pose in the sensor coordinates at the time of collection;

In this embodiment, the superscript 0 on the left side of the above formula (10) indicates that this is a preliminary collection position. Because after the optimization of the problem in formula (5), the rotation matrix , there is a small rotation around the x-axis and y-axis. After the above formula (10) The conduction, pose rotation matrix /> It also includes a small rotation around the x and y axes, so through two transformations between the rotation matrix and Euler angles, the rotation matrix is simplified as follows:

Here, rot2Euler means to obtain a Euler angle equivalent to a rotation matrix, and Euler2rot means to obtain a rotation matrix equivalent to an Euler angle.

Obtain the label corresponding to the object o _j in RGB-D, sample, f _i , that is, get the pose and its label that meets the label format requirements, and become the object label corresponding to the RGB-D sample:

in, is the pose of object o _j in vector form in RGB-D samples.

In addition, a method for quickly labeling objects in a point cloud with the aid of 3D reconstruction provided in this embodiment also includes the following steps of label verification:

above label Only the spatial poses of the labels in the sample f _i are calculated, and this embodiment uses the point cloud D _i to check the validity of the labels. First, use the Open3D segmentation technique to obtain the subpoint cloud within /> it is located in the tag /> Inside the 3D-BBox. Then calculate the minimum convex hull volume S _j of the sub-point cloud, and finally judge the validity of the label according to the volume V _j of the 3D-BBox:

Among them, V _j = w _j l _j h _j , which means the volume of the object shape 3D-BBox; r is the ratio of the volume of the smallest convex hull of the sub-point cloud to the volume of the 3D-BBox, Indicates that under the current RGB-D sample, there is an object o _j and its label /> The reliability of , only if /> when label /> is valid and accepted as the label for an object o _j in an RGB-D sample f _i .

The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still implement the foregoing embodiments Modifications to the technical solutions described in the examples, or equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of each embodiment of the application, and should be included in the scope of the technical solutions of the embodiments of the application. within the scope of protection.

Claims (2)

1. The method for quickly labeling the object in the point cloud under the assistance of three-dimensional reconstruction is characterized by comprising the following steps of:

acquiring an RGB-D sample sequence to be marked;

building an SE (3) pose chart according to the RGB-D sample acquisition position sequence, and building space constraint among sampling positions by utilizing a matching relation among RGB-D samples, so as to optimize the SE (3) pose chart, and obtain an acquisition position sequence after RGB-D sample optimization; based on the optimized acquisition position sequence, fusing the RGB-D sample sequence into a global 3D map;

performing object labeling in the global 3D map by adopting a manual labeling mode;

according to the labeling result, converting the pose of each object in the world coordinate system into the pose in the sensor coordinate corresponding to the acquisition position to obtain the pose of each object in each acquisition position, namely the RGB-D sample, and further forming an object label in the RGB-D sample together with the object type and the size;

the construction of the SE (3) pose graph according to the RGB-D sample acquisition position sequence comprises the following steps:

the SE (3) pose map is specifically as follows:

wherein ,and ε are abbreviations for Vertex Vertex and Edge, respectively; />The variable to be optimized in the SE (3) pose graph is a collection position sequence set of all RGB-D samples; epsilon is a constraint set of optimization problems, each constraint +.>Representing two vertices s _* and />Space constraints between points; setting the RGB-D sample sequence Q= { f _i I= … n, where f _i Is an RGB-D sample comprising an RGB color image I _i And a 3D point cloud D equivalent to the depth image _i ，f _i ＝(I _i ，D _i ) The method comprises the steps of carrying out a first treatment on the surface of the The acquisition position sequence is P= { s _i |i＝0，...，n}，s _i Is a 3D space 6 degree of freedom position and attitude point, expressed by SE (3) variable;

the construction of spatial constraint between sampling positions by using the matching relation between RGB-D samples, and further optimizing an SE (3) pose diagram, to obtain an acquisition position sequence after RGB-D sample optimization, comprises the following steps:

determining spatial constraints between acquisition location pairs by point cloud registration:

wherein,<p，q>representation D _＊ Andpairs of points between, s' _＊ and />Respectively s _★ and />Initial value of-> As an initial value of the variable e;

the acquisition position s is predicted by the following equation _★ Andoffset between:

according to the space constraint condition and the predicted offset, the optimization target for obtaining the SE (3) pose map is as follows:

wherein x= { s ₁ ，s ₂ ，...，s _n Is a vector of parameters to be optimized, eachIs an error item->Is a matrix of information of (a); solving the above-mentioned optimization problem using g2o and obtaining an optimized acquisition position sequence +.>

The acquisition position sequence X based on optimization ^* Fusing the RGB-D sample sequence into a global 3D map, comprising:

all RGB-D samples are converted into world coordinate system:

all RGB-D samples are represented in the same world coordinate system, together constituting a 3D map M of the environment:

the method for labeling the object in the global 3D map by adopting the manual labeling mode comprises the following steps:

loading the three-dimensional map M into software for object labeling, and labeling artificial objects; for any one object o _j The object shape is represented by a 3D bounding box, and the object label format is as follows:

wherein ,c_j Is the category of the object S _j ＝[w _j ，l _j ，h _j ]Is the size of 3D-BBox, is the pose of an object in a world coordinate system;

setting the object to stand perpendicular to a horizontal xoy plane in a world coordinate system, wherein the rotation angle of the object around x and y axes is 0;

according to the labeling result, converting the pose of each object in the world coordinate system into the pose in the sensor coordinate corresponding to the acquisition position, so as to obtain the pose of each object in each sample, and further obtaining an object label corresponding to the RGB-D sample, which comprises the following steps:

the object o is represented by the following matrix _j The pose in the world coordinate system is expressed in the form of the following matrix:

wherein ,is the position of the object in world coordinate system, < >>Is Euler angleAn equivalent three-dimensional spatial rotation matrix;

then, based on each RGB-D sample f _i Optimized acquisition positionObject o _j The pose under the global coordinate system is converted into the pose in the sensor coordinate at the acquisition time:

above-mentioned pose rotation matrixIncluding slight rotations about the x-axis and the y-axis, the rotation matrix is simplified by two transformations between the rotation matrix and the euler angle as follows:

here, rot2Euler means obtaining an Euler angle equivalent to a rotation matrix, and Euler2rot means obtaining a rotation matrix equivalent to an Euler angle;

finally, an RGB-D sample f is obtained _i Intermediate object o _j The pose matrix of (a) is as follows:

finally, the object label after analysis is automatically acquired by the method as follows:

wherein ,for object o _j Pose in vector form in RGB-D samples.

2. The method for rapid labeling of objects in a point cloud with the aid of three-dimensional reconstruction of claim 1,

the method for quickly labeling the object in the point cloud under the assistance of three-dimensional reconstruction further comprises the step of label verification, and specifically comprises the following steps:

for one RGB-D frame f _i Acquiring a location tag by using Open3D point cloud segmentation technologySub-point cloud inside->

Computing sub-point cloudIs the minimum convex hull volume S of _j ；

From the representation object o _j Volume V of 3D-BBox of (B) _j Judging the validity of the label:

wherein ,V_j ＝w _j ·l _j ·h _j Representing the volume of the object shape 3D-BBox; r is the ratio of the volume of the minimum convex hull of the sub-point cloud to the volume of the 3D-BBox,represented at the current RGB-D sampleUnder, there is an object o _j And label->Is only whenWhen (1) label->Is effective and accepted as RGB-D sample f _i An object o of (a) _j Is a label of (a).