CN115937279A - Point cloud density up-sampling method, system and medium based on cross-modal data registration - Google Patents

Abstract

The invention discloses a point cloud density up-sampling method, a system and a medium based on cross-modal data registration, wherein the method comprises the steps of collecting three-dimensional point cloud data of a target object and images of a plurality of visual angles; respectively extracting image features, registering, calculating spatial correlation between the image and the image features under any visual angle, and constructing a three-dimensional point cloud framework model; extracting an original three-dimensional skeleton model from the three-dimensional point cloud data, and overlapping the original three-dimensional skeleton model with the three-dimensional point cloud skeleton model to obtain an overlapped skeleton model; and reversely acquiring pixel points in the image corresponding to the visual angle aiming at the points in the overlapped skeleton model, copying and expanding the acquired pixel points according to the specified multiple, and finally obtaining a complete sampling result on the density of the three-dimensional point cloud. The invention aims to solve the problems of large calculation amount and low accuracy of an output model in the two existing methods for point cloud densification, has the advantages of simple implementation method and low cost, and can realize the rapid densification of low-density three-dimensional point cloud based on images.

Description

Point cloud density upsampling method, system and medium based on cross-modal data registration

technical field

The invention belongs to the technical field of computer vision, and in particular relates to a point cloud density upsampling method, system and medium based on cross-modal data registration.

Background technique

A point cloud is a data set, each point in the data set represents a set of X, Y, Z geometric coordinates and an intensity value, which records the intensity of the returned signal according to the surface reflectance of the object. When these points are combined, a point cloud is formed, a collection of data points in space that represent a 3D shape or object. Point clouds can also be automatically colored for a more realistic visualization. At present, the following two methods are used for point cloud densification: (1) Semantic-based point cloud densification method: upsampling on the point cloud, that is, input a point cloud, output a denser point cloud, and it falls on the input On the underlying geometry (such as a surface) of the point cloud. The core idea of PU-Net is to learn the features of each point at multiple granularities (from local to global), then expand the point set in the feature space, and finally map the expanded point set back to three dimensions. (2) Geometry-based point cloud densification methods: Whether they are semantic-based or geometry-based point cloud densification methods, they have high complexity when modeling objects, and the geometry of the model itself is a priori. High demand, lack of accuracy and robustness when densifying the point cloud for modeling, and under actual working conditions, it is impossible to obtain a complete and accurate semantic prior for the initial point cloud obtained by scanning in a short time and in a limited environment Information, and the lack of sufficient computing resources for geometric upsampling.

Contents of the invention

In view of the above technical problems existing in the prior art, the technical problem to be solved by the present invention is to provide a point cloud density upsampling method, system and medium based on cross-modal data registration. The present invention aims to solve the problem of point cloud densification. Some of the problems encountered in the two methods are the large amount of calculation and the low accuracy of the output model. A simple and low-cost point cloud density upsampling method is provided, which can realize the low-density 3D point cloud based on the image. Rapid densification.

In order to solve the problems of the technologies described above, the technical solution adopted in the present invention is:

A point cloud density upsampling method based on cross-modal data registration, including:

S101, collecting 3D point cloud data of the target object and images from multiple perspectives;

S102, respectively extracting features of images of multiple viewing angles, registering image features of multiple viewing angles, and calculating a spatial correlation between images and image features at any viewing angle;

S103, constructing a three-dimensional point cloud skeleton model through the spatial association between image features of any two viewing angles;

S104, extracting an original 3D skeleton model from the 3D point cloud data, and overlapping the extracted original 3D skeleton model with the constructed 3D point cloud skeleton model and the original 3D skeleton model to obtain an overlapping skeleton model;

S105. For the points in the overlapping skeleton model, reversely obtain the pixel points in the image corresponding to the viewing angle, copy and expand the obtained pixel points according to a specified multiple, and finally obtain a complete 3D point cloud density upsampling result.

Optionally, the 3D point cloud data of the target object collected in step S101 is a sparse point cloud model _PL .

Optionally, when collecting images of multiple viewing angles in step S101 , the angles between adjacent viewing angles are fixed, and all viewing angles cover a range of 360 degrees of the target object, and finally images I _{1 -In} of n viewing angles are obtained.

Optionally, calculating the spatial correlation between images and image features at any viewing angle in step S102 refers to calculating the distance d _i =d(I _i , F _i ) between image I _i and image feature F _i at any viewing angle i as a spatial association.

Optionally, the distance d _i =d(I _i , F _i ) between the image I _i and the image feature F _i at any viewing angle i refers to the Euclidean distance.

Optionally, step S103 includes:

S201, taking the distance d _i = d(I _i , F _i ) between the image I _i and the image feature F _i at any viewing angle i as the association weight between the image and the feature at the viewing angle _i The image features F _i are aggregated to construct the binary group data (I ₁ , F ₁ ), (I ₂ , F ₂ ), ..., (I _n , F _n ) of each view image and feature;

S202. Reconstruct the three-dimensional skeleton model of the binary group data (I ₁ , F ₁ ), (I ₂ , F ₂ ), ..., (I _n , F _n ) of images and features of each view angle, wherein any view angle Each point in image I _i and image feature F _i under i is used as a point (x, y, z) in the three-dimensional skeleton, so as to obtain the final reconstructed overlapping skeleton model P _I .

Optionally, in step S104, overlapping the extracted original 3D skeleton model with the constructed 3D point cloud skeleton model and the original 3D skeleton model refers to combining the extracted original 3D skeleton model with the constructed 3D point cloud skeleton model and the original The three-dimensional skeleton models are superimposed in three-dimensional space to obtain the overlapping skeleton model P ₁ .

Optionally, step S105 includes:

S301, for each point (x, y, z) in the overlapping skeleton model, reversely acquire the pixel point (x, y) and the adjacent 8 pixel points (x-1, y- 1), (x-1,y), (x-1,y+1), (x,y-1), (x,y+1), (x+1,y-1,z), ( x+1,y) and (x+1,y+1);

S302, the pixel point (x, y) in the image corresponding to the viewing angle and the adjacent 8 pixel points (x-1, y-1), (-1, y), (x-1, y+1) , (x,y-1), (x,y+1), (x+1,y-1,z), (x+1,y) and (x+1,y+1) a total of nine pixels The coordinate z value of the assigned point (x, y, z) is expanded to 9 coordinate points of the point cloud, and finally the overlapping skeleton model is expanded by 9 times to obtain the dense point cloud model P ₂ as the final complete 3D point cloud Density upsampling results.

In addition, the present invention also provides a system for point cloud density upsampling based on cross-modal data registration, comprising interconnected microprocessors and memories, the microprocessor is programmed or configured to perform the cross-modal based A Point Cloud Density Upsampling Method for Data Registration.

In addition, the present invention also provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer program is used to be programmed or configured by a microprocessor to perform the cross-modal data allocation based on A standard point cloud density upsampling method.

Compared with the prior art, the present invention mainly has the following advantages: the present invention includes collecting three-dimensional point cloud data of the target object and images from multiple perspectives; extracting image features, registering and calculating the relationship between images and image features under any perspective. The spatial correlation between the three-dimensional point cloud skeleton model is constructed; the original three-dimensional skeleton model is extracted from the three-dimensional point cloud data, and the overlapping skeleton model is obtained by overlapping with the three-dimensional point cloud skeleton model; for the points in the overlapping skeleton model, the image of the corresponding perspective is reversely acquired The pixel points in , and copy and expand the acquired pixel points according to the specified multiple, and finally get the complete 3D point cloud density upsampling result. The invention can solve the problems of large amount of calculation and low accuracy of the output model encountered in the two existing methods of point cloud densification, has the advantages of simple implementation method and low cost, and can realize low-density three-dimensional point analysis based on images. Rapid densification of clouds.

Description of drawings

Fig. 1 is a schematic flow diagram of the basic process of the method of the embodiment of the present invention.

Fig. 2 is the three-dimensional point cloud data of the target object collected in the embodiment of the present invention.

FIG. 3 shows the finally obtained complete 3D point cloud density upsampling result in the embodiment of the present invention.

Detailed ways

As shown in Figure 1, the point cloud density upsampling method based on cross-modal data registration in this embodiment includes:

S101, collecting 3D point cloud data of the target object and images from multiple perspectives;

S102, respectively extracting features of images of multiple viewing angles, registering image features of multiple viewing angles, and calculating a spatial correlation between images and image features at any viewing angle;

S103, constructing a three-dimensional point cloud skeleton model through the spatial association between image features of any two viewing angles;

S104, extracting an original 3D skeleton model from the 3D point cloud data, and overlapping the extracted original 3D skeleton model with the constructed 3D point cloud skeleton model and the original 3D skeleton model to obtain an overlapping skeleton model;

S105. For the points in the overlapping skeleton model, reversely obtain the pixel points in the image corresponding to the viewing angle, copy and expand the obtained pixel points according to a specified multiple, and finally obtain a complete 3D point cloud density upsampling result.

In this embodiment, the point cloud density upsampling method based on cross-modal data registration is the registration process of calculating the spatial association and generating the point cloud skeleton model in steps S102 and S103. The spatial correlation between the three-dimensional point cloud skeleton model is constructed, and the step S104 extracts the original three-dimensional skeleton model from the three-dimensional point cloud data, and two kinds of skeleton models of different modes are obtained. By combining the extracted original three-dimensional skeleton model with the constructed three-dimensional The point cloud skeleton model and the original 3D skeleton model are overlapped to obtain the overlapping skeleton model, which realizes the registration and fusion of cross-modal vector data, and can solve the large amount of calculation and output problems encountered in the two existing methods of point cloud densification. The problem of low model accuracy has the advantages of simple implementation method and low cost, and can realize rapid densification of low-density 3D point clouds based on images.

In this embodiment, the 3D point cloud data of the target object collected in step S101 is a sparse point cloud model _PL .

In this embodiment, when images of multiple viewing angles are collected in step S101 , the angles between adjacent viewing angles are fixed, and all viewing angles cover a range of 360 degrees of the target object, and finally images I _{1 -In} of n viewing angles are obtained.

In this embodiment, calculating the spatial correlation between images and image features at any viewing angle in step S102 refers to calculating the distance between image I _i and image features F _i at any viewing angle _i = x(I _i , F _i ) as a spatial association.

In this embodiment, the distance d _i =d(I _i , F _i ) between the image I _i and the image feature F _i at any viewing angle i refers to the Euclidean distance. The features in the multi-view image are extracted respectively, and the multi-view image features are registered to obtain the pairwise spatial correlation. The collected images I ₁ , I ₂ ,..., _n from different angles of view are used for feature extraction of edge pixels in the image, and the features extracted from each image are denoted as F ₁ , F ₂ ,...,F _n , The image and feature binary data (I ₁ , F ₁ ), (I ₂ , F ₂ ),..., (I _n , F _n ) obtained by extracting the feature and image data between pairs Calculus distance calculation, obtain d _n =d(I _n , F _n ).

In this embodiment, step S103 includes:

S201, taking the distance d _i =d(I _i , F _i ) between the image I _i and the image feature F _i at any viewing angle i as the association weight between the image and the feature at the viewing angle _i The image features F _i are aggregated (features are extracted and combined), and the binary group data (I ₁ ,F ₁ ),(I ₂ ,F ₂ ),...,(I _n ,F _n );

S202. Reconstruct the three-dimensional skeleton model of the binary group data (I ₁ , F ₁ ), (I ₂ , F ₂ ),..., (I _n , F _n ) of images and features of each view angle, wherein any view angle Each point in image I _i and image feature F _i under i is used as a point (x, y, z) in the three-dimensional skeleton, and all points are integrated to obtain the final reconstructed overlapping skeleton model P _I .

In this embodiment, overlapping the extracted original 3D skeleton model with the constructed 3D point cloud skeleton model and the original 3D skeleton model in step S104 refers to combining the extracted original 3D skeleton model with the constructed 3D point cloud skeleton model and The original 3D skeleton model is superimposed in the 3D space to obtain the overlapping skeleton model P ₁ .

In this embodiment, step S105 includes:

S301, for each point (x, y, z) in the overlapping skeleton model, reversely acquire the pixel point (x, y) and the adjacent 8 pixel points (x-1, y- 1), (x-1,y), (x-1,y+1), (x,y-1), (x,y+1), (x+1,y-1,z), ( x+1,y) and (x+1,y+1);

S302, the pixel point (, y) in the image corresponding to the viewing angle and the adjacent 8 pixel points (x-1, y-1), (x-1, y), (x-1, y+1) , (x,y-1), (x,y+1), (x+1,y-1,z), (x+1,y) and (x+1,y+1) a total of nine pixels The coordinate z value of the assigned point (, y, z) is expanded to 9 coordinate points of the point cloud, and finally the overlapping skeleton model is expanded by 9 times to obtain the dense point cloud model P ₂ as the final complete 3D point cloud density Upsampled results.

In this embodiment, the three-dimensional point cloud data of the target object collected in step S101 is shown in Figure 2, and the complete three-dimensional point cloud density upsampling result obtained in step S105 is shown in Figure 3. It can be seen that this embodiment can realize image-based Fast densification of low-density 3D point clouds.

In summary, the point cloud density upsampling method based on cross-modal data registration in this embodiment includes collecting 3D point cloud data of the target object and images from multiple perspectives; extracting image features, registering and calculating Spatial association between images and image features and construction of a 3D point cloud skeleton model; extract the original 3D skeleton model from the 3D point cloud data, and overlap with the 3D point cloud skeleton model to obtain an overlapping skeleton model; for the points in the overlapping skeleton model, reverse Obtain the pixel points in the image corresponding to the viewing angle, copy and expand the obtained pixel points according to the specified multiple, and finally obtain the complete 3D point cloud density upsampling result. The point cloud density upsampling method based on cross-modal data registration in this embodiment can effectively solve the problems of large amount of calculation and low accuracy of the output model encountered in the two existing methods of point cloud densification, and has an implementation method With the advantages of simplicity and low cost, it can realize rapid densification of low-density 3D point clouds based on images.

In addition, this embodiment also provides a system for point cloud density upsampling based on cross-modal data registration, which includes interconnected microprocessors and memories, and the microprocessor is programmed or configured to perform the cross-modal data registration based on Point Cloud Density Upsampling Method for State Data Registration.

In addition, this embodiment also provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program is used to be programmed or configured by a microprocessor to execute the cross-modal data-based Point Cloud Density Upsampling Method for Registration.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram. These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

The above descriptions are only preferred implementations of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, and all technical solutions under the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, some improvements and modifications without departing from the principles of the present invention should also be regarded as the protection scope of the present invention.

Claims (10)

1. A point cloud density up-sampling method based on cross-modal data registration is characterized by comprising the following steps:

s10l, collecting three-dimensional point cloud data of a target object and images of a plurality of visual angles;

s102, respectively extracting the features of the images at multiple visual angles, registering the image features at multiple visual angles, and calculating the spatial correlation between the images and the image features at any visual angles;

s103, constructing a three-dimensional point cloud skeleton model through spatial correlation between image features of any two visual angles;

s104, extracting an original three-dimensional skeleton model from the three-dimensional point cloud data, and overlapping the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model to obtain an overlapped skeleton model;

and S105, reversely acquiring pixel points in the image corresponding to the visual angle aiming at the points in the overlapped skeleton model, copying and expanding the acquired pixel points according to the specified multiple, and finally obtaining a complete sampling result on the density of the three-dimensional point cloud.

2. The point cloud density up-sampling method based on cross-modal data registration according to claim 1, wherein the three-dimensional point cloud data of the target object collected in step S101 is a sparse point cloud model P _L 。

3. The point cloud density up-sampling method based on cross-modal data registration of claim 1, wherein an included angle between adjacent viewing angles is fixed when the images of multiple viewing angles are collected in step S101, and all viewing angles cover a 360-degree range of a target object, and finally an image I of n viewing angles is obtained ₁ ～I _n 。

4. The point cloud density up-sampling method based on cross-modal data registration according to claim 1, wherein the step S102 of calculating the spatial correlation between the image and the image feature at any view angle means calculating the image I at any view angle I _i And image feature F _i A distance d between _i ＝d(I _i ，F _i ) As a spatial association.

5. The method of claim 4, wherein the method comprises using a point cloud density upsampling method based on cross-modal data registrationImage I under the arbitrary view angle I _i And image feature F _i A distance d between _i ＝d(I _i ，F _i ) Referred to as the euclidean distance.

6. The point cloud density up-sampling method based on cross-modal data registration according to claim 5, wherein step S103 comprises:

s201, image I under any view angle I _i And the distance d between the image feature Fi _i ＝d(I _i ，F _i ) Image I at view I as an associated weight between image and feature at view I _i And image feature F _i Carrying out aggregation to construct binary data (I) of each view angle image and characteristic ₁ ，F ₁ )，(I ₂ ，F ₂ )，...，(I _n ，F _n )；

S202, binary data (I) of each view angle image and feature ₁ ，F ₁ )，(I ₂ ，F ₂ )，...，(I _n ，F _n ) Carrying out three-dimensional skeleton model reconstruction, wherein an image I under an arbitrary visual angle I _i And image feature F _i As one point (x, y, z) in the three-dimensional skeleton, to obtain a final reconstructed overlapping skeleton model P _I 。

7. The point cloud density up-sampling method based on cross-modal data registration of claim 1, wherein the step S104 of overlapping the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model means overlapping the extracted original three-dimensional skeleton model with the constructed three-dimensional point cloud skeleton model and the original three-dimensional skeleton model in a three-dimensional space to obtain an overlapped skeleton model P ₁ 。

8. The point cloud density up-sampling method based on cross-modal data registration according to claim 1, wherein step S105 comprises:

s301, aiming at each point (x, y, z) in the overlapped skeleton model, reversely acquiring a pixel point (x, y) in the image of the corresponding view angle and 8 adjacent pixel points (x-1, y-1), (x-1, y + 1), (x, y-1), (x, y + 1), (x +1, y-1, z), (x +1, y) and (x +1, y + 1);

s302, nine pixel points including a pixel point (x, y) in the image corresponding to the visual angle, 8 adjacent pixel points (x-1, y-1), (x-1, y + 1), (x, y-1), (x, y + 1), (x +1, y-1, z), (x +1, y) and (x +1, y + 1) are expanded into coordinate points of 9 point clouds, and finally the overlapped skeleton model is expanded by 9 times to obtain a dense point cloud model P2 as a final complete three-dimensional point cloud density up-sampling result.

9. A point cloud density upsampling system based on cross-modal data registration, comprising a microprocessor and a memory which are connected with each other, wherein the microprocessor is programmed or configured to execute the point cloud density upsampling method based on cross-modal data registration according to any one of claims 1 to 8.

10. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is adapted to be programmed or configured by a microprocessor to perform the method for point cloud density upsampling based on cross-modal data registration according to any one of claims 1 to 8.

Images