CN107240129A - Object and indoor small scene based on RGB D camera datas recover and modeling method - Google Patents

Abstract

The object of the present invention is to provide a kind of object and indoor small scene recovery and modeling method based on RGB-D camera data, be summarized as: this method is in carrying out RGB-D depth data (point cloud) registration modeling, points The constraints of surface-to-surface and point-to-projection are combined and applied to the precise registration of the sequence depth data (point cloud) obtained by the RGB-D camera, and finally the point cloud model (.ply format) of the object or small scene is obtained. The model It can be used for object measurement and further CAD modeling. The registration method in the present invention not only takes into account the speed advantages of the point-to-projection algorithm, but also integrates the accuracy advantages of the point-to-tangent plane algorithm, overcomes the problems of slow speed and low scene accuracy of the traditional point cloud registration method, and can be fast, Accurately find the corresponding point of a point on the source point cloud on the target point cloud to realize multi-view point cloud splicing.

Description

Object and indoor small scene restoration and modeling method based on RGB-D camera data

technical field

The invention belongs to the field of photogrammetry and computer vision, in particular to a method for restoring and modeling objects and small indoor scenes based on RGB-D camera data.

Background technique

In recent years, 3D reconstruction has increasingly become a research hotspot and focus in the field of computer vision, and has been widely used in many fields such as industrial surveying, cultural relics protection, reverse engineering, e-commerce, computer animation, medical anatomy, photomicrography, and virtual reality. Commonly used object and indoor modeling methods mainly use laser scanner LIDAR to obtain point cloud data for modeling, or use cameras to obtain overlapping images based on binocular stereo vision for modeling. The former is expensive and the instrument operation is cumbersome; Or the data processing is complicated and time-consuming, it has requirements on lighting conditions, and special calibration is required for the camera. Portable RGB-D sensors (such as Kinect2.0) are cheap, simple in structure, and easy to operate. They can simultaneously acquire the depth and color data of the scene, and the acquisition of depth data is not interfered by the visible light of the environment. These advantages greatly reduce the difficulty and cost of 3D modeling, enabling ordinary users to digitize objects in daily life and connect the virtual and real worlds.

In the point cloud modeling of object and indoor small scene restoration and modeling using RGB-D camera data, depth data (point cloud) registration is the key technology for 3D object reconstruction and indoor small scene restoration, which involves the initial registration of point cloud. Determination of the quasi-position and precise registration of the point cloud on this basis. At present, in terms of determining the rough registration position of point cloud registration, using the target ball is the most common method, and if there is no target, the method of manually marking the rough position is often used. The point cloud fine registration is based on the rough registration. of. In 1992, Besl and Mckay proposed a registration algorithm based on free-form surfaces - iterative closest point algorithm (Iterative Closest Point, ICP), which is a registration method from point set to point set, without the need to segment the surface and feature extraction, but the convergence speed of the algorithm is slow, and it is easy to iteratively converge to a local extremum, resulting in registration failure. The existing point cloud registration algorithms are basically based on the ICP algorithm and improved to improve the accuracy, efficiency and robustness of point cloud matching to obtain reconstruction results. Microsoft has developed KinectFusion technology for the Kinect camera, the representative of RGB-D cameras, which can perform real-time 3D geometric modeling of objects on machines that support GPU acceleration, and quickly create sufficiently smooth reconstruction scenes. However, this method does not perform loop closure detection and optimization, and the reconstruction model will generate large cumulative errors, resulting in model deformation, and the algorithm is prone to tracking failure when the scene tends to be flat. Since the RGB-D camera can acquire optical and depth data at the same time, people study the use of optical image data to introduce epipolar constraints in ICP registration for RGB-D camera depth data registration, and apply it to Kinect camera data. The robustness of this method It has high reliability and can basically restore the overall model of the scene. Especially in the case of KinectFusion tracking modeling failure, it can still achieve good results, but the model details are sometimes distorted.

Aiming at the problems existing in the existing restoration and modeling methods of objects and small indoor scenes using RGB-D camera data: the estimation accuracy of the moving camera pose is not high, there is systematic error accumulation in the reconstruction model, and the model surface is relatively rough As well as the disadvantages of poor real-time performance of the algorithm, the research uses various constraint information to improve the ICP algorithm to obtain a point cloud registration algorithm with high accuracy and robustness, and realize the restoration and construction of objects and indoor small scenes based on RGB-D camera data. Model, so that the rapid modeling of objects and small scenes can be carried out without the help of expensive and bulky laser scanners, which is one of the research goals of people in the field of computer vision and close-range photogrammetry.

Contents of the invention

The object of the present invention is to provide a method for restoring and modeling objects and indoor small scenes based on RGB-D camera data.

The technical solution to be protected in the present invention is characterized by:

A kind of object and indoor small scene recovery and modeling method based on RGB-D camera data, it is characterized in that, be summarized as: this method is in carrying out RGB-D depth data (point cloud) registration modeling, points to surface Combined with the point-to-projection constraints, it is applied to the precise registration of the sequence depth data (point cloud) obtained by the RGB-D camera, and finally the point cloud model (.ply format) of the object or small scene is obtained. This model can be used for Object measurement and further CAD modeling. The registration method in the present invention not only takes into account the speed advantages of the point-to-projection algorithm, but also integrates the accuracy advantages of the point-to-tangent plane algorithm, overcomes the problems of slow speed and low scene accuracy of the traditional point cloud registration method, and can be fast, Accurately find the corresponding point of a point on the source point cloud on the target point cloud to realize multi-view point cloud splicing.

The technical scheme of the present invention is specifically described as follows:

Let S _P and S _Q be the point cloud data of two adjacent frames obtained by the RGB-D camera from different viewpoints P and Q. Because when RGB-D is used for point cloud acquisition, the sampling frequency is high and the camera moves slowly, so the translation value of the registration value between adjacent frames of point cloud data is close to the 0 vector, and the rotation array is close to the E array. You can use 0, E as the initial value for point cloud registration. S _P , S _Q are local surfaces parameterized by roughly registered point clouds, p ₀ is a point on S _P , is the normal vector of point p ₀ , and q _s is the normal line passing through point p ₀ The intersection point with S _Q is the tangent plane of S _Q through q _s , and q′ _s is the projection of point p ₀ on the tangent plane. Use p ₀ and q' _s as registration points for point cloud registration parameter calculation. The key to the point-to-surface registration method is to obtain the intersection point q _s . The commonly used method for obtaining the intersection point is to find the triangle on the surface of the target point cloud, and use the three vertices of the triangle to interpolate the intersection point. However, the calculation amount of this method is Big and time consuming. In order to avoid this problem, in the present invention, the point-to-projection method is first used to obtain the intersection point, and then the tangent plane passing through the intersection point is obtained based on the obtained intersection point, and the candidate point pairs of registration points are further determined, and then the rigid motion consistency constraint is used to determine the previous The established candidate point pairs are eliminated by erroneous point pairs, and finally the ICP fine registration of _{SP and S Q two} point clouds is performed using the registration point pairs after erroneous point removal to obtain accurate registration parameters. After the fine registration parameters between the continuous frame point clouds are obtained according to the above method, the point cloud model (.ply format) of the object or small scene is obtained by using the fine registration parameters of the point cloud to perform registration modeling on the continuous frame point clouds.

Further provide embodiment technical scheme, process is as follows:

Firstly, the RGB-D data acquisition is carried out (the Kinect2.0 released by Microsoft is used as an example of the depth data acquisition device to illustrate later), and the software and hardware platform environment for RGB-D data acquisition are properly configured to meet the acquisition requirements before data acquisition; When collecting, for a fixed target or scene, within the depth recognition range of the RGB-D camera, the hand-held camera rotates around the object or scene at a relatively stable and low speed to form a closed circle, and large objects should be avoided during the collection process. The jitter to prevent large deviations in frame-to-frame registration.

After obtaining the sequence frame data, perform object scene modeling, as follows:

1. Depth image filtering

In the actual measurement process, the sensor is affected by strong light conditions and the surface material of the object to be measured. The depth data contains a lot of noise, especially at the edge of the object, and invalid data will appear. Therefore, before converting the depth data into a 3D point cloud, Bilateral filtering is used to denoise and smooth the depth image while retaining the upper edge information of the object's depth image. While considering the relationship between pixels in the spatial position distance, the similarity between pixels is also considered.

w _s =w _g ×w _h

Equation (1) is the calculation formula of bilateral filtering. When the standard deviation δ _g of the spatial domain remains unchanged and the weight δ _h of the gray threshold gradually increases, the weights w _h corresponding to different gray values are all very large. Using the gray value of the depth image (depth information) The effect of changing edge information becomes weaker and weaker, and the bilateral filter degenerates into a Gaussian filter. When δ _h is constant and δ _g gradually increases, the blurring of the image becomes more and more obvious. After repeated experiments on objects and small scenes, the template size of bilateral filtering in the present invention is 3×3, and the standard deviations of Gaussian functions in the space domain and range domain are 10 and 15, respectively.

2. Color image feature point detection and image matching

In order to generate a 3D model consistent with the scene and perform texture reconstruction at the same time as geometric reconstruction, it is necessary to use a feature detection operator to find the feature points of the color image and match the colors of two adjacent frames. Due to the large amount of data, ORB (ORiented Brief) algorithm extracts color image feature points. Compared with SIFT (Scale-invariant feature transform) and SURF (Speed-up robust features) operators, the ORB algorithm has a shorter running time, and the accuracy of algorithm matching is high. Binary coding is used to describe the extracted Features greatly save computer memory. After the color image matching is completed, according to the relative pose between the calibrated color lens and the depth lens, the color image is aligned with the depth data, and the colored point cloud under different viewing angles of the scene is generated.

3. Improved ICP fine registration method

A. Use the back-project method to press a point p ₀ on the nth frame to be registered in the direction of the viewpoint of the next frame ((n+1) frame) Project to get its pixel coordinates p _q , the calculation formula is

In formula (2), M _Q is the center projection matrix (the internal reference matrix obtained after camera calibration), and T _Q is the transformation matrix that transforms the point cloud obtained from the Q viewpoint into the same world coordinate system. Since the calculation result of formula (2) is not necessarily an integer, the result of formula (2) is rounded (rounded) to obtain the value p _q on the grid point of the depth image.

B. Using the forward-project method, bring the p and _q coordinates into the right end of the equation (3) to obtain new three-dimensional point coordinates The calculation formula is:

In formula (3), is the internal parameter matrix obtained after camera calibration, Z _c is the depth value of p _q on the (n+1)th depth image, calculated by formula (3)

C. Using formula (4) will the normal vector projected at p ₀ On, get the coordinates of point p ₁

In formula (4), is the unit normal vector at p ₀ .

D. After obtaining p ₁ , use p ₁ as the new p ₀ value to participate in the above three steps of A, B, and C, perform projection, and recursively iterate to obtain p ₂ . Constantly iterate, the point obtained by the i-th projection The iteration converges to the intersection point q _s , and when i tends to infinity, there is In the actual situation, limited by the number of projections, the termination condition of the iteration is:

Among them, ε _c is a given threshold value of 10 ^-6 . Obtain the converged intersection point q _s , determine the tangent plane passing through the point q _s , project p ₀ on the tangent plane, and obtain the projected point q′ _s .

E. Perform the above steps A to D for each point on the depth image preprocessed in step 1 to obtain the q′ _s of each point, and finally form a set of matching point pairs {p ₀ } and {q′ _s }.

F. Affected by the initial value of the iteration and the surface smoothness of the point cloud, there are still wrong corresponding points in the matching point pair {p ₀ } and {q′ _s } established by the above method, which is especially obvious at the initial stage of the iteration. Therefore, in the present invention, rigid motion consistency constraints are used to eliminate wrong point pairs. Assuming (p ₁ ,q ₁ ) and (p ₂ ,q ₂ ) are candidate corresponding point pairs, there are rigid motion consistency constraints:

In formula (6), t is a pre-defined threshold, for all candidate points (p _i , q _i ), i=1, 2,...,n, judge compatibility with each other, and eliminate incompatible points.

G. Use the new corresponding point sequence {p ₀ } and {q′ _s } obtained by eliminating the wrong candidate point pairs as the exact matching point, use the singular value decomposition method to obtain the unit eigenvector corresponding to the largest eigenvalue of the matrix, and calculate the nth frame The rotation matrix R and the translation vector T between the point clouds of the adjacent (n+1) frame can realize fine registration between adjacent frames.

4. The precise registration results between adjacent point cloud frames obtained in step 3 are based on the coordinate system of the first frame point cloud as the reference coordinate system, and the above steps 1 to 3 are used for all other adjacent frame point clouds. processing to obtain the rotation matrix R and translation vector T between adjacent frames, and finally use formula (7) to unify all frame point clouds into the reference coordinate system.

Generate point cloud models of objects and scenes in a unified coordinate system in the form of .ply files. This file can be displayed in a general 3D geometry processing system such as Meshlab or Geomagic Studio for further data processing.

Compared with the existing methods, the fine registration method combining point-to-surface and point-to-projection, and adding rigid motion consistency constraints takes into account the advantages of speed and accuracy of existing algorithms, and the execution time of the algorithm is shorter. The registration accuracy is higher, the structural features of the built model are clearer, the algorithm is more robust, and the accuracy is reliable, which can meet the recovery and modeling of objects and small indoor scenes.

The invention can be applied to the restoration and modeling of objects and indoor small scenes based on RGB-D camera data, and plays a role in the quantitative restoration and modeling of target objects and small scenes.

Description of drawings

Fig. 1 is the overall flowchart of the algorithm of the present invention.

Fig. 2 is a detailed flowchart of the improved ICP algorithm in the present invention.

Fig. 3 is a schematic diagram of a point cloud registration method combined with point-to-plane constraints and point-to-projection constraints proposed by the present invention.

Figure 4 is the 41st frame of data in the conference room scene obtained by the RGB-D sensor (using the Kinect camera), where Figure 4(a) is the corresponding color image (grayscale display), Figure 4(b) is the infrared image, and Figure 4(b) is the infrared image. 4(c) is depth data.

Figure 5 is the data of the next frame (42 frames) adjacent to Figure 4, where Figure 5(a) is the color image (grayscale display), Figure 5(b) is the infrared image, and Figure 5(c) is the depth data .

Figure 6(a) and Figure 6(b) are the results of bilateral filtering for denoising and smoothing the depth data of the conference room scene in Figure 4 and Figure 5, respectively.

Fig. 7 is the matching result of two adjacent color images in Fig. 4 and Fig. 5 (in grayscale display).

Figure 8 is a colored point cloud (grayscale display).

Fig. 9 is an effect diagram of the scene model established after registration by the method of the present invention, wherein the gray triangle cone in the right diagram represents the position and the direction of the visual axis when the Kinect camera acquires the point cloud.

detailed description

The specific implementation method is described below in conjunction with an implementation example, that is, a representative RGB-D camera—Microsoft’s Kinect2.0—obtains sequential color and depth data of objects or small indoor scenes, and generates data that can be used by Meshlab, 3D model data supported by general 3D geometry processing systems such as Geomagic Studio, in the form of .ply files.

Configure the software and hardware platform environment for Kinect data collection before data collection, using a notebook with window 8 (64 bits) and USB 3.0 interface. The software platform environment needs to pre-install the development kit SDK independently developed by Microsoft, and install Configure the open source cross-platform computer vision library OpenCV 3.0. During acquisition, for a fixed target or scene, within the depth recognition range of the Kinect camera, the hand-held camera rotates around the object or scene at a relatively stable and low speed to form a closed circle. During the acquisition process, large shakes should be avoided. Prevent large deviations in frame-to-frame registration.

Example 1:

For the conference room scene in the implementation example, a total of 151 frames of data are collected, and the time interval of data sampling is 3s.

The experimental data obtained by Kinect 2.0 includes the color image, infrared image and depth data of the meeting room scene. The resolution of the color image is 1920×1080, and the resolution of the depth and infrared image is 512×424. The following is the gray value (depth value) of the original depth image of the 41st frame, the range is row number 209~224, and the column number is 192~220.

Table 1

The relative pose parameters between the color lens and the depth lens obtained through calibration are: the rotation matrix is The translation vector is According to the specific process shown in Figure 1, the entire implementation process is as follows:

1. Depth image filtering

Before converting the depth data in the form of an image acquired by the Kinect camera into a 3D point cloud format in the form of (X, Y, Z), considering the actual data acquisition process, the depth sensor is affected by the external lighting conditions and the surface material of the object to be measured. , the obtained depth data contains a lot of noise, especially at the edge of the object, invalid data will appear. Therefore, before the conversion, the depth image is denoised and smoothed using formula (1), while retaining the edge information of the object.

The size of the bilateral filtering template of the present invention is 3×3, and the standard deviations of the Gaussian functions of the space domain and the range domain are 10 and 15 respectively. Then the domain kernel is defined as:

0.9900 0.9950 0.9900 0.9950 1.0000 0.9950 0.9900 0.9950 0.9900

The value range kernel is:

Due to the large amount of depth data, only the original gray value in a 3×3 range of the depth data of the 41st frame is listed here:

177 177 177 177 176 178 175 179 180

After the frame data is bilaterally filtered, the value in the 3×3 range is

175 176 175 176 176 176 173 175 172

Figure 6(a) is the result of bilateral filtering for denoising and smoothing the depth data in Figure 4 (frame 41), and Figure 6(b) is the result of bilateral filtering for the next frame adjacent to Figure 5 (frame 41 (frame 42). )) The result of denoising and smoothing medium depth data.

2. Color image feature point detection and image matching

In order to generate a 3D model that is consistent with the scene, texture reconstruction is carried out at the same time as geometric reconstruction. The feature detection operator is used to find the feature points of the color image and match the colors of two adjacent frames. Due to the large amount of data, the ORB algorithm is used to extract the color image. For feature points, the ORB algorithm takes less time to run than SIFT and SURF operators, and has a high accuracy of algorithm matching. At the same time, it uses binary encoding to describe the extracted features to save computer memory. Match the color images of the 41st frame and the 42nd frame, and the matching is completed, and the result is shown in Figure 7. According to the relative pose between the color lens and the depth lens obtained by calibration, the color image is aligned with the depth data, and the colored point cloud of the object and the scene is generated under different viewing angles. The color image and the 42nd frame are respectively compared The depth data is aligned, and the generated colorized point cloud results are shown in Figure 8(a) and Figure 8(b).

3. Improved ICP fine registration method

First, convert the depth data of the 41st frame after depth filtering to obtain point cloud data, which contains 8475 points. The following is the information of the first 72 points of the point cloud data:

Table 2

The point cloud registration algorithm combining point-to-surface and point-to-projection is used to estimate and optimize the position and attitude of the object or small indoor scene at the exposure moment, and take the coordinate system where the first frame point cloud is located as the reference coordinate system, and the rest The point clouds of each frame are transformed into the reference coordinate system after coordinate transformation.

A. Using the back-project method, take a point p ₀ on the point cloud of the 41st frame, the coordinates are (-0.432926, -0.383050, 1.971088), and follow the viewpoint direction of the point cloud of the 42nd frame according to formula (2) Project to get its pixel coordinate p _q , and the pixel coordinate is (348,270). in,

B. Using the forward-project method, bring the p and _q coordinates into the right end of the equation (3) to obtain a new three-dimensional point The coordinates are (0.261255, 0.179193, 1.000000).

C. Using formula (4) will Project the normal vector at point cloud p ₀ of frame 41 Above, the coordinates of point p ₁ are (-0.308091, -0.357967, 2.012104).

D. After obtaining p ₁ , according to formula (5), get Here, the threshold ε _c is set to be 10 ^-6 . exist When the value is greater than the threshold _εc , take the coordinates of point p ₁ as the new coordinates of p ₀ , return to step A, repeat the above three steps of A, B, and C, perform projection, and obtain p ₂ by recursive iteration; at this time, if Less than ε _c , get q _s . After the converged intersection point q _s is obtained, the tangent plane of point q _s on frame 42 is determined, and p ₀ is projected on the tangent plane to obtain the projected point q′ _s with coordinates of (-0.432665, -0.387157, 1.975511).

E. Perform the above steps A to D for each point on the point cloud generated by the depth image of the frame after preprocessing in step 1 to obtain the q′ _s of each point, and finally form the initial matching point pair set {p ₀ } and {q _'s }. Due to the large number of matching points between the two frames of point clouds, Table 3 only shows part of the corresponding matching points in the initial matching point pair set after the processing of the 41st frame and the adjacent 42nd frame:

table 3

F. Affected by the iterative initial value and the surface smoothness of the point cloud, there are still wrong matching corresponding points in the candidate matching point pairs {p ₀ } and {q′ _s } established by the above method. In the present invention, the rigid motion consistent Sexual constraints eliminate wrong point pairs. Assuming (p ₁ , q ₁ ) and (p ₂ , q ₂ ) are candidate matching point pairs, perform rigid motion consistency constraints on all matching points in the 41st and 42nd frames determined above according to formula (6), Where t is a pre-defined threshold (10% in this method), for all candidate points (p _i , q _i ), i=1,2,...,n, judge the compatibility with each other, and for incompatible points to cull. The following is the result of calculating point pair compatibility for the first two pairs of points in Table 3 above:

This result indicates that these two pairs of points are incompatible and should be rejected. For the above step E, all matching point pairs are obtained, and after using the above method to eliminate the wrong point pairs, more accurate matching point pairs are obtained. Table 4 is a list of part corresponding matching points in the matching point pair collection obtained after this processing:

Table 4

p ₀ -3.5780711 0.54807687 1.6893054 qs ₀ ' -3.5738416 0.547196265 1.6887584 p ₁ -3.6656244 0.51411259 1.7566197 qs ₁ ' -3.6676449 0.51509092 1.7566626 p ₂ -3.7146368 0.50028324 1.8223131 qs ₂ ' -3.7141747 0.50061625 1.8230906 p ₃ -2.2115169 0.99832821 0.29833382 qs ₃ ' -2.2117355 0.9957425 0.29692292 p ₄ -2.2430336 1.0023514 0.41302541 qs ₄ ' -2.2431977 1.0026156 0.41257041 p ₅ -2.2954671 0.98881006 0.49076268 qs ₅ ' -2.2955692 0.9881003 0.50018574 p ₆ -2.4128008 0.94495845 0.58901 qs ₆ ' -2.4125545 0.94472801 0.57922614 p ₇ -3.0111077 0.59995174 0.41161895 qs ₇ ' -3.0111504 0.599762393 0.4117605561 p ₈ -2.7442737 0.80849338 0.79625237 qs ₈ ' -2.7482291 0.79723013 0.80870435 p ₉ -2.8855073 0.77564269 1.0258969 qs ₉ ' -2.8863597 0.77563919 1.0290474 p ₁₀ -3.0237136 0.73981613 1.2300546 qs ₁₀ ' -3.0245329 0.73907477 1.2307017

G. Use the new corresponding point sequence {p ₀ } and {q′ _s } obtained by eliminating the wrong candidate point pair as the matching point and use the singular value decomposition method to obtain the unit eigenvector corresponding to the largest eigenvalue of the matrix to calculate the 41st frame Between the adjacent point cloud of the 42nd frame

The 6 degrees of freedom of the three-dimensional rigid body transformation are expressed as:

For other adjacent frame point clouds, the above steps A to G are used to sequentially obtain the transformation parameters between frames, that is, the rotation matrix R and the translation vector T.

4. Generation of object and scene models

Using the precise registration results R and T between adjacent frame point clouds obtained in step 3, the 151 frames of point cloud data are transformed into a unified coordinate system according to formula (7) (where the first frame point cloud depth camera is located coordinate system), and superimpose the data to obtain the 3D point cloud of the scene in .ply format (displayable in software such as Meshlab and Geomagic Studio). The model of the generated object or scene is shown in Figure 9, and part of the data in the generated .ply file is shown in Table 5:

table 5

Claims (8)

1. An object and indoor small scene recovery and modeling method based on RGB-D camera data is characterized in that a closed loop is formed by acquiring continuous frame depth data (point cloud) of a scene or an object to be modeled by using an RGB-D camera, and the acquired data are processed by the following steps: carrying out bilateral filtering on depth data (point cloud); secondly, detecting the characteristic points of the color image and matching the characteristic points with the image; in the RGB-D depth data (point cloud) registration modeling, the method integrates constraint conditions from point to surface and from point to projection, and determines accurate registration parameters R, T between adjacent depth data (point cloud) by using rigid motion consistency constraint; and fourthly, applying the fine registration parameters between the adjacent depth data (point clouds) to the accurate registration between the sequence depth data (point clouds) obtained by the RGB-D camera to obtain a point cloud model (ply format) of the object or the small scene, wherein the point cloud model can be used for measuring the object and further modeling CAD.

2. The method of claim 1, wherein step (r) preprocesses the depth data (point cloud). Adopting bilateral filtering to reduce noise and smooth the depth image, simultaneously keeping the edge information of the depth image of an object, considering the relation of pixels on the space position distance, and simultaneously considering the similarity degree among the pixels, and taking an expression (1) as a calculation expression of the bilateral filtering:

w_s＝w_g×w_h

<mrow> <msub> <mi>w</mi> <mi>g</mi> </msub> <mrow> <mo>(</mo> <mi>i</mi> <mo>,</mo> <mi>j</mi> <mo>)</mo> </mrow> <mo>=</mo> <mi>exp</mi> <mrow> <mo>(</mo> <mo>-</mo> <mfrac> <mrow> <msup> <mrow> <mo>(</mo> <mi>i</mi> <mo>-</mo> <mi>x</mi> <mo>)</mo> </mrow> <mn>2</mn> </msup> <mo>+</mo> <msup> <mrow> <mo>(</mo> <mi>j</mi> <mo>-</mo> <mi>y</mi> <mo>)</mo> </mrow> <mn>2</mn> </msup> </mrow> <mrow> <mn>2</mn> <msubsup> <mi>&amp;delta;</mi> <mi>g</mi> <mn>2</mn> </msubsup> </mrow> </mfrac> <mo>)</mo> </mrow> </mrow>

<mrow> <msub> <mi>w</mi> <mi>h</mi> </msub> <mrow> <mo>(</mo> <mi>i</mi> <mo>,</mo> <mi>j</mi> <mo>)</mo> </mrow> <mo>=</mo> <mi>exp</mi> <mrow> <mo>(</mo> <mo>-</mo> <mfrac> <msup> <mrow> <mo>(</mo> <mi>I</mi> <mo>(</mo> <mrow> <mi>i</mi> <mo>-</mo> <mi>j</mi> </mrow> <mo>)</mo> <mo>-</mo> <mi>I</mi> <mo>(</mo> <mrow> <mi>x</mi> <mo>-</mo> <mi>y</mi> </mrow> <mo>)</mo> <mo>)</mo> </mrow> <mn>2</mn> </msup> <mrow> <mn>2</mn> <msubsup> <mi>&amp;delta;</mi> <mi>h</mi> <mn>2</mn> </msubsup> </mrow> </mfrac> <mo>)</mo> </mrow> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>1</mn> <mo>)</mo> </mrow> </mrow>

in the formula (1), w_gThe space domain weight is determined by the second expression in the expression (1), w_hThe value domain weight is determined by the third formula in the formula (1), (x, y) is the index value on the depth image to be filtered, (i, j) is the index value in the (x, y) neighborhood range,_gis the standard deviation of the spatial domain and,_his the standard deviation of the gray scale domain, w_sIs a bilateral filtering weight.

Standard deviation of space domain_gInvariant, weight of grey threshold_hIn the process of gradually increasing, the weights w corresponding to different gray values_hThe two-sided filter is a Gaussian filter, and the function of retaining edge information by using the change of the gray level (depth information) of a depth image is weaker and weaker; in that_hThe temperature of the molten steel is not changed,_gin the process of gradual increase, the blurring of the image becomes more and more obvious.

3. The method of claim 2, wherein the size of the bilateral filtering template is 3 x 3, and the standard deviation of the spatial and the value domain gaussian functions are 10 and 15, respectively.

4. The method of claim 1, wherein said step (ii) extracts color image feature points using an orb (ordered brief) algorithm, and describes the extracted features using binary coding. After the color images are matched, according to the relative position and posture between the color lens and the depth lens obtained by calibration, the color images are aligned with the depth data, and colorized point clouds of different viewing angles of the scene are generated.

5. The method of claim 1, wherein in step ③, the precise registration parameter between adjacent depth data (point clouds) is set as S_P、S_QTwo adjacent frames of point cloud data obtained from different viewpoints P, Q for an RGB-D camera. The RGB-D camera moves slowly when acquiring point clouds, the translation amount of a registration value between adjacent frame point cloud data is close to a 0 vector, a rotation array is close to an E array, and 0 and E can be used as initial values of point cloud registration. S_P、S_QIs a coarsely registered point cloud parameterized local surface, p₀Is S_PAt the point of the last point, the first point,is p₀Normal vector of points, q_sIs over p₀Normal to the pointAnd S_QCross point of (2), cross q_sTo S_QTangent plane of (q)_sIs' a p₀The projection of a point on the tangent plane. By p₀And q is_s' registration points as a solution to point cloud registration parameters. The point-to-surface registration method is characterized in that the key point is to obtain an intersection point q_s：

Firstly, the point-to-projection is used to obtain the intersection point, then based on the obtained intersection point the tangent plane passing through the intersection point is obtained and the registration point is further defined, and for S_PAll points on the point cloud are obtained by the method in S_QObtaining a candidate matching point pair by the above registration points, then carrying out error point pair elimination on the previously established candidate matching point pair by utilizing rigid motion consistency constraint, and finally utilizing error point eliminationSubsequent registration point pair S_P、S_QAnd (4) carrying out improved ICP (inductively coupled plasma) fine registration on the two-point cloud to obtain a fine registration parameter.

The process is as follows:

A. using back-project method to register the nth frame S_PAt a certain point p₀Press the (n +1) th frame S of a frame_QDirection of viewpoint ofProjecting to obtain its pixel coordinate p_qThe calculation formula is

<mrow> <msub> <mi>p</mi> <mi>q</mi> </msub> <mo>=</mo> <msub> <mi>M</mi> <mi>Q</mi> </msub> <msubsup> <mi>T</mi> <mi>Q</mi> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msubsup> <msub> <mi>p</mi> <mn>0</mn> </msub> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>2</mn> <mo>)</mo> </mrow> </mrow>

In the formula (2), the reaction mixture is,

M_Qis a central projection matrix (an internal reference matrix obtained after camera calibration),

T_Qthe point cloud obtained from the Q viewpoint is converted into a transformation matrix under the same world coordinate system.

Since the calculation result of the formula (2) is not necessarily an integer, the rounding operation is performed on the result of the formula (2) to obtain the value p on the grid point of the depth image_q。

B. P obtained in step A is subjected to forward-project method_qThe coordinate is brought into the right end of the equation (3) to obtain a new three-dimensional point coordinate The calculation formula of (2) is as follows:

<mrow> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <msub> <mi>X</mi> <mi>c</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>Y</mi> <mi>c</mi> </msub> </mtd> </mtr> <mtr> <mtd> <msub> <mi>Z</mi> <mi>c</mi> </msub> </mtd> </mtr> <mtr> <mtd> <mn>1</mn> </mtd> </mtr> </mtable> </mfenced> <mo>=</mo> <msub> <mi>Z</mi> <mi>c</mi> </msub> <msup> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <msub> <mi>f</mi> <mi>x</mi> </msub> </mtd> <mtd> <mn>0</mn> </mtd> <mtd> <msub> <mi>u</mi> <mn>0</mn> </msub> </mtd> <mtd> <mn>0</mn> </mtd> </mtr> <mtr> <mtd> <mn>0</mn> </mtd> <mtd> <msub> <mi>f</mi> <mi>y</mi> </msub> </mtd> <mtd> <msub> <mi>v</mi> <mn>0</mn> </msub> </mtd> <mtd> <mn>0</mn> </mtd> </mtr> <mtr> <mtd> <mn>0</mn> </mtd> <mtd> <mn>0</mn> </mtd> <mtd> <mn>1</mn> </mtd> <mtd> <mn>0</mn> </mtd> </mtr> </mtable> </mfenced> <mrow> <mo>-</mo> <mn>1</mn> </mrow> </msup> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <mi>u</mi> </mtd> </mtr> <mtr> <mtd> <mi>v</mi> </mtd> </mtr> <mtr> <mtd> <mn>1</mn> </mtd> </mtr> </mtable> </mfenced> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>3</mn> <mo>)</mo> </mrow> </mrow>

in the formula (3), the reaction mixture is,for the internal parameter matrix obtained after camera calibration, f in the matrix_x、f_yPrincipal moments of the depth camera lens in the x and y directions respectively,in order to be the coordinates of the main point like,is p₀Homogeneous coordinate p of point in pixel coordinate system_q(ii) is expressed in terms of (a),are homogeneous coordinates in the depth camera coordinate system. Z_cIs p_qIn the (n +1) th frame S_QThe depth value on the depth image is obtained by calculation of the formula (3)Three-dimensional coordinates of (a).

C. By the formula (4)Projected at p₀Normal vector of (c)To obtain p₁Coordinates of points

<mrow> <mtable> <mtr> <mtd> <mrow> <mi>&amp;alpha;</mi> <mo>=</mo> <mrow> <mo>(</mo> <mrow> <msub> <mi>q</mi> <msub> <mi>p</mi> <mn>0</mn> </msub> </msub> <mo>-</mo> <msub> <mi>p</mi> <mn>0</mn> </msub> </mrow> <mo>)</mo> </mrow> <mo>&amp;CenterDot;</mo> <mover> <mi>p</mi> <mo>&amp;RightArrow;</mo> </mover> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>p</mi> <mn>1</mn> </msub> <mo>=</mo> <msub> <mi>p</mi> <mn>0</mn> </msub> <mo>+</mo> <mi>&amp;alpha;</mi> <mover> <mi>p</mi> <mo>&amp;RightArrow;</mo> </mover> </mrow> </mtd> </mtr> </mtable> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>4</mn> <mo>)</mo> </mrow> </mrow>

In the formula (4), the reaction mixture is,is p₀The unit normal vector of (b) α is a vectorIn excess of p₀Unit normal vector ofThe projected length of (c).

D. To obtain p₁Then p is added₁As new p₀The value participates in the A, B, C steps, projection is carried out, and p is obtained through recursive iteration₂. Continuously iterating, i-th projecting obtained pointConverging the iteration on the intersection point q_sAnd when i tends to infinity, there areThe termination condition of the iteration is:

<mrow> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mrow> <msub> <mi>&amp;epsiv;</mi> <mi>c</mi> </msub> <mo>=</mo> <mo>|</mo> <mo>|</mo> <msub> <mi>p</mi> <mi>i</mi> </msub> <mo>-</mo> <msub> <mi>q</mi> <msub> <mi>p</mi> <mi>i</mi> </msub> </msub> <mo>|</mo> <mo>|</mo> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <msub> <mi>&amp;epsiv;</mi> <mi>c</mi> </msub> <mo>&lt;</mo> <mi>T</mi> </mrow> </mtd> </mtr> </mtable> </mfenced> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>5</mn> <mo>)</mo> </mrow> </mrow>

wherein_cLess than a given threshold T. Obtaining the converged intersection point q_sDetermining a passing point q_sTangent plane of p, i₀Projected on the tangent plane to obtain a projected point q'_s。

E. Performing the steps A to D on each point on the depth image preprocessed in the step ① to obtain q 'of each point'_sFinally, a set of candidate matching point pairs { p }is formed₀Q 'and { q'_s}。

F. The matching point pair { p is established by the method under the influence of the iteration initial value and the smoothness of the cloud surface₀Q 'and { q'_sThere are still wrong corresponding points in the iteration, which is particularly evident at the beginning of the iteration, and thereforeIn the invention, rigid motion consistency constraint is adopted to remove error point pairs. Suppose (p)₁,q₁) And (p)₂,q₂) For candidate corresponding point pairs, there is a rigid motion consistency constraint:

<mrow> <mo>-</mo> <mi>t</mi> <mo>&amp;le;</mo> <mfrac> <mrow> <mo>|</mo> <mo>|</mo> <msub> <mi>p</mi> <mn>1</mn> </msub> <mo>-</mo> <msub> <mi>p</mi> <mn>2</mn> </msub> <mo>|</mo> <mo>|</mo> <mo>-</mo> <mo>|</mo> <mo>|</mo> <msub> <mi>q</mi> <mn>1</mn> </msub> <mo>-</mo> <msub> <mi>q</mi> <mn>2</mn> </msub> <mo>|</mo> <mo>|</mo> </mrow> <mrow> <mi>m</mi> <mi>a</mi> <mi>x</mi> <mrow> <mo>(</mo> <mo>|</mo> <mo>|</mo> <msub> <mi>p</mi> <mn>1</mn> </msub> <mo>-</mo> <msub> <mi>p</mi> <mn>2</mn> </msub> <mo>|</mo> <mo>|</mo> <mo>,</mo> <mo>|</mo> <mo>|</mo> <msub> <mi>q</mi> <mn>1</mn> </msub> <mo>-</mo> <msub> <mi>q</mi> <mn>2</mn> </msub> <mo>|</mo> <mo>|</mo> <mo>)</mo> </mrow> </mrow> </mfrac> <mo>&amp;le;</mo> <mi>t</mi> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>6</mn> <mo>)</mo> </mrow> </mrow>

in the formula (6), t is a predetermined threshold value, and for all candidate points (p)_i,q_i) And i is 1,2, …, n, judging compatibility of each other, and rejecting incompatible points.

G. New corresponding point sequence { p obtained by eliminating wrong candidate point pairs₀And q_s' } as precise matching point, and using singular value decomposition method to obtain matrix maximumCalculating the unit characteristic vector corresponding to the large characteristic value to calculate the n-th frame S_PAnd the adjacent (n +1) th frame S_QAnd (4) realizing accurate registration between adjacent frames by using a rotation matrix R and a translational vector T between the point clouds.

6. The method as claimed in claim 5, wherein the given threshold T-10-⁶。

7. The method of claim 5, wherein the given threshold t in the F point in step (c) is 10%.

8. The method according to claim 1, wherein in the step (iv), after the sequence depth data (point clouds) obtained in the step (iii) are respectively and precisely registered to obtain the precise registration parameters between the continuous frame point clouds, the point cloud precise registration parameters are used to perform registration modeling on the continuous frame point clouds to obtain the point cloud model (in the ply format) of the object or the small scene.

The process is as follows:

and (3) obtaining accurate registration results of each adjacent point cloud frame in the step (c) (the results are all in a reference coordinate system of the coordinate system where the first frame point cloud is located), and unifying all frame point clouds under the reference coordinate system by using a formula (7).

<mrow> <msub> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <mi>X</mi> </mtd> </mtr> <mtr> <mtd> <mi>Y</mi> </mtd> </mtr> <mtr> <mtd> <mi>Z</mi> </mtd> </mtr> </mtable> </mfenced> <mrow> <mi>r</mi> <mi>e</mi> <mi>f</mi> </mrow> </msub> <mo>=</mo> <mi>R</mi> <mfenced open = "[" close = "]"> <mtable> <mtr> <mtd> <mi>X</mi> </mtd> </mtr> <mtr> <mtd> <mi>Y</mi> </mtd> </mtr> <mtr> <mtd> <mi>Z</mi> </mtd> </mtr> </mtable> </mfenced> <mo>+</mo> <mi>T</mi> <mo>-</mo> <mo>-</mo> <mo>-</mo> <mrow> <mo>(</mo> <mn>7</mn> <mo>)</mo> </mrow> </mrow>

And generating an object and scene point cloud model under a unified coordinate system.