CN110751684B - Object three-dimensional reconstruction method based on depth camera module - Google Patents

Abstract

The invention relates to the technical field of three-dimensional reconstruction in the field of computer vision, and relates to a three-dimensional reconstruction method of objects based on a depth camera module. The present invention adopts a new hash method: MD5 in the voxel hash algorithm process, which can greatly improve the speed of data insertion, search and indexing, and reduce collisions; in addition, the present invention proposes a new memory allocation method , which solves the defect of one-time allocation of fixed-size memory in the existing method, so that memory can be automatically allocated during the reconstruction process, and the purpose of dynamically expanding the reconstruction area can be achieved; the present invention adopts a new two-frame alignment method, and calculates Select the orb feature points of the two frames of color images and select the corresponding point pairs. Use the corresponding points to obtain the world coordinates of the previous frame and the camera coordinates of the next frame from the depth map, thereby solving the camera extrinsic parameters of the next frame. By aligning the two frames before and after, this method can more accurately find corresponding points and reduce the drift problem in three-dimensional reconstruction.

Description

Object 3D reconstruction method based on depth camera module

Technical field

The present invention relates to the technical field of three-dimensional reconstruction in the field of computer vision, and more specifically, to a three-dimensional reconstruction method of an object based on a depth camera module.

Background technique

In recent years, computer vision technology has developed rapidly. The use of computer vision technology for three-dimensional reconstruction has become more efficient and convenient, and has many applications in cultural relic protection and restoration, industrial inspection, and 3D printing. It is also a VR and AR application technology. important parts of. Early 3D reconstruction technology usually uses 2D images as input to reconstruct the 3D model in the scene. Due to the data problems of this reconstruction mode itself, the reconstructed 3D model is prone to holes and distortions. In recent years, the invention of various depth cameras, such as kinect, TOF, RealSense, etc., has greatly promoted the development of 3D reconstruction technology, and more technologies have begun to favor the use of depth cameras for real-time 3D reconstruction.

Real-time 3D reconstruction technology uses depth data at different angles and converts it to the same coordinate system to achieve surface reconstruction and rendering. It is very difficult and costly to take into account high reconstruction quality, large reconstruction scale, and fast reconstruction speed. Challenging. The three-dimensional reconstruction technology based on the Kinectfusion algorithm basically meets the real-time requirements, but it can only handle the reconstruction task of small scenes, and its reconstruction process consumes a lot of video memory. The three-dimensional reconstruction method based on the voxel hash algorithm is considered a relatively excellent reconstruction method. It uses the voxel hash method and takes into account the requirements of speed and scene size. However, it still has the following shortcomings: 1. The The important steps on which the technology relies: The hash function used in voxel hashing H (x, y, z) = (x·p ₁ ⊕y·p ₂ ⊕z·p ₃ )%n has a high probability of generating a hash collision , a larger number of hash collisions will seriously slow down the execution efficiency of the algorithm, and increase the risk of memory overflow; 2. This technology requires a fixed-size memory space to be allocated at one time during the voxel hashing process, which limits the three-dimensional Rebuild scalability.

Contents of the invention

In order to overcome the above-mentioned defects in the prior art, the hash function used for voxel hashing has a high probability of hash collision, seriously slows down the execution efficiency of the algorithm, and increases the risk of memory overflow, and provides a depth-based method. The camera module's object three-dimensional reconstruction method uses a new hash function method, which can greatly increase the speed of data insertion, search and indexing, and reduce collisions.

In order to solve the above technical problems, the technical solution adopted by the present invention is: a three-dimensional reconstruction method of objects based on a depth camera module, using an algorithm based on voxel hashing to achieve three-dimensional reconstruction; it is characterized in that, in the voxel hashing process A new hash function method is used to calculate the hash value. The new hash function method includes the following steps:

First, convert the three-dimensional coordinates into one-dimensional indexes through the following formula:

In the formula, Δ is the size of the current device resolution, that is, the size of one voxel;

Then, the calculated one-dimensional index data is converted into a hash value through the MD5 method.

Further, the MD5 conversion process specifically includes the following steps:

S21. Add a 1 and several 0s after the one-dimensional index data, so that the byte length modulo 512 is 448. The length of the data before being padded is shown in the last 64 bits. The data length after padding is an integer multiple of 512; in The length before padding is expressed in the last 64 bits, which means that the last 64 bits of binary numbers can be converted into a decimal number, which represents the length of the data before padding;

S22. Process the padded data in groups of 512 bits. Each group is divided into 16 32-bit sub-blocks. Four 32-bit registers are used to process the sub-blocks in a loop. MD5 is initialized with four link variables as parameters. These four variables They are: a=0x67452301, b=0xefcdab89, c=0x98badcfe, d=0x10325476;

S23. Perform four rounds of cyclic compression operation, each round has 16 steps, and each step uses a message. The message refers to data grouped into 512 bits; use step functions to update variables a, b, c, and d respectively. ; The step function is Q _i+1 =Q _i+1 +((Q _i-3 +f _i (Q _i , Q _i-1 , Q _i-2 ) + w _i +t _i )＜＜＜ s _i );

S24. Concatenate four 32-bit sub-blocks to obtain a 128-bit value, which is the final hash value. Each parameter of abcd has 32 bits. The final compression result obtained after processing all messages is the cascade value of abcd 32*4=128.

Further, the present invention proposes a memory dynamic management method. The existing voxel hash hash table adopts the method of pre-allocating memory. Although this can improve the efficiency of indexing, it limits the scope of the three-dimensional reconstruction scene. This method The invention draws on the hash map (hash matching) method in the standard template library in C++11 to achieve dynamic expansion of memory. When the pre-allocated memory is full, the hash table memory is dynamically expanded to achieve a wider range of Three-dimensional reconstruction. The described dynamic memory management method includes the following steps: adding an integer variable in the hash table, which represents the number of available positions in the current hash table. When the value of the variable is close to 0, stop modifying the hash table. Insert, then open a new hash table of the same size and point the header of the new table to the tail of the old table, and then reconstruct the hash table; assuming that the size of the original hash table is n, the length of the new table at this time It becomes 2n. The hash value of all previously inserted elements is modulo 2n to get the new bucket number, and the elements in the old table are moved to the expanded new table; after the reconstruction is completed, the new bucket number can be obtained Hash table for insertion operations. When calculating, after obtaining the hash value, use modulo N to obtain the bucket in which it should be placed. After expansion, the number of buckets changes from the original N to 2N, so the previous elements must be The hash value modulo 2n.

Further, the object three-dimensional reconstruction method based on the depth camera module specifically includes the following steps:

S1. Set the world view under the current architecture: 3D reconstruction technology essentially establishes a large enough space voxel set to wrap the object to be reconstructed. This parent space voxel set can be divided into three downwards. Layer substructure: chunk is a first-level substructure, and the spatial voxel set contains n*n*n chunk structures; block is a second-level substructure, and each chunk contains m*m*m block structures; voxel is Three-level substructure, each block structure contains t*t*t voxel substructures; the variables m, n, and t mentioned above are all positive integer variables and can be set according to specific circumstances;

S2. Establish a "video stream" based on the TOF module, obtain its depth image, RGB image and point cloud at the current time, and read the TOF module's camera internal parameters K and the camera external parameters T at the current attitude and read them. Store it in relevant parameters;

S3. Based on the color images of the previous and next frames, calculate the ORB feature points of the two color images before and after and select the corresponding point pairs. Use the corresponding points to obtain the world coordinates of the previous frame and the camera coordinates of the next frame from the depth map, thereby Solve the camera extrinsic parameters of the next frame so that the point cloud under the current frame is aligned with the world coordinate system where the standard frame (first frame) is located;

S4. Establish a scalable dynamic hash table on the GPU device side through the voxel hash method based on MD5 encoding; move the blocks within the view cone range from the host side to the hash table on the device side based on the frustum principle;

S5. Based on the depth map of the current frame and combined with the information in the hash table, select the blocks that are valid within the truncation range of the depth area, and create a new dynamically allocated hash on the GPU device side through the memory dynamic management method. table, and move valid blocks to a new hash table. The new hash table records the location information corresponding to all valid blocks and the location of the voxel array it points to;

S6. Traverse all block positions in the new hash table, and use CUDA to launch multi-threads to traverse all voxels in the block. According to the initial position of the starting point of the parent space voxel set in the world coordinate system and the real length of each voxel, Calculate the world coordinate system position of the voxel, use the back-projection formula to project it from the world coordinate system position to the pixel coordinate system position, and determine whether the world coordinate position of the voxel is truly visible within the depth frame range. If it is not visible, do not do it. Process, otherwise use the TSDF-truncated directed distance field formula to update its TSDF value and update the current weight;

S7. The TSDF value is equivalent to a collection of isosurfaces. The position where the TSDF value is 0 is equivalent to the surface of the object. The Marching Cubes technology is used to generate a triangular surface mesh and render it;

S8. Copy all the voxels in the new hash table from the GPU device back to the host, record the content of the voxel at that location on the host, and release the hash table on the GPU device to prevent video memory leaks;

S9. Read the new RGB image, depth image and point cloud at the current moment from the TOF module, and jump to step S3.

Compared with existing technology, the beneficial effects are:

1. The present invention provides a three-dimensional object reconstruction method based on a depth camera module, which uses the MD5 method to encode hash values, effectively reducing the number of conflicts and enabling faster insertion, query, and deletion operations in the hash table;

2. The new memory allocation method proposed by the present invention can allocate new memory during the real-time scanning process, thereby adapting to a wider range of scene reconstruction work and improving the practicability of the algorithm;

3. The present invention adopts a new alignment method of the front and back frames, calculates the orb feature points of the front and back two frame color images and selects the corresponding point pairs, and uses the corresponding points to obtain the world coordinates of the previous frame and the following from the depth map. The camera coordinates of one frame are used to calculate the camera extrinsic parameters of the next frame and align the two frames before and after. This method can find the corresponding points more accurately than using the ICP algorithm and reduce the drift problem in three-dimensional reconstruction.

Description of the drawings

Figure 1 is an overall flow chart of the three-dimensional reconstruction method of the present invention.

Figure 2 is a schematic diagram of the MD5 encoding method of the present invention.

Figure 3 is a schematic diagram of the TSDF calculation of the present invention.

Detailed ways

The drawings are for illustrative purposes only and should not be construed as limitations of the present invention; in order to better illustrate this embodiment, some components of the drawings may be omitted, enlarged or reduced, which do not represent the size of the actual product; for those skilled in the art It is understandable that some well-known structures and their descriptions may be omitted in the drawings. The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limitations of the present invention.

Example 1:

As shown in Figure 1, a three-dimensional object reconstruction method based on a depth camera module specifically includes the following steps:

S1. Set the world view under the current architecture: 3D reconstruction technology essentially establishes a large enough space voxel set to wrap the object to be reconstructed. This parent space voxel set can be divided into three downwards. Layer substructure: chunk is a first-level substructure, and the spatial voxel set contains n*n*n chunk structures; block is a second-level substructure, and each chunk contains m*m*m block structures; voxel is Three-level substructure, each block structure contains t*t*t voxel substructures; the variables m, n, t mentioned above are all positive integer variables and can be set according to the specific situation; m is selected in this embodiment is 128, n is 8, and t is 5. Generally speaking, the larger the variable, the better the reconstructed surface reality will be, but on the contrary, it will reduce a certain calculation speed.

S2. Establish a "video stream" based on the TOF module, obtain its depth image, RGB image and point cloud at the current time, and read the TOF module's camera internal parameters K and the camera external parameters T at the current attitude and read them. stored in relevant parameters.

S3. Based on the color images of the previous and next frames, calculate the ORB feature points of the two color images before and after and select the corresponding point pairs. Use the corresponding points to obtain the world coordinates of the previous frame and the camera coordinates of the next frame from the depth map, thereby Solve the camera extrinsic parameters of the next frame so that the point cloud under the current frame is aligned with the world coordinate system where the standard frame (first frame) is located.

S4. Establish a scalable dynamic hash table on the GPU device side through the voxel hash method based on MD5 encoding; move the blocks within the view cone range from the host side to the hash table on the device side based on the frustum principle.

S5. Based on the depth map of the current frame and combined with the information in the hash table, select the blocks that are valid within the truncation range of the depth area, and create a new dynamically allocated hash on the GPU device side through the memory dynamic management method. table, and moves valid blocks to a new hash table. The new hash table records the location information corresponding to all valid blocks and the location of the voxel array it points to.

S6. Traverse all block positions in the new hash table, and use CUDA to launch multi-threads to traverse all voxels in the block. According to the initial position of the starting point of the parent space voxel set in the world coordinate system and the real length of each voxel, Calculate the world coordinate system position of the voxel, use the back-projection formula to project it from the world coordinate system position to the pixel coordinate system position, and determine whether the world coordinate position of the voxel is truly visible within the depth frame range. If it is not visible, do not do it. Process, otherwise use the TSDF-truncated directed distance field formula to update its TSDF value and update the current weight, as shown in Figure 3.

S7. The TSDF value is equivalent to a collection of isosurfaces. The position where the TSDF value is 0 is equivalent to the object surface. Marching Cubes technology is used to generate a triangular surface mesh and render it.

S8. Copy all the voxels in the new hash table from the GPU device back to the host, record the content of the voxel at that location on the host, and release the hash table on the GPU device to prevent video memory leaks.

S9. Read the new RGB image, depth image and point cloud at the current moment from the TOF module, and jump to step S3.

In this embodiment, in the hash calculation, the key value of the hash table is obtained from the coordinates of the voxel center point through a hash function. In this embodiment, a new hash function method is used to calculate the hash value, the new hash function method includes the following steps:

First, convert the three-dimensional coordinates into one-dimensional indexes through the following formula:

In the formula, Δ is the size of the current device resolution, that is, the size of one voxel;

Then, the calculated one-dimensional index data is converted into a hash value through the MD5 method.

As shown in Figure 2, the MD5 conversion process specifically includes the following steps:

S21. Add a 1 and several 0s after the one-dimensional index data, so that the byte length modulo 512 is 448. The length of the data before being padded is expressed in the last 64 bits. The data length after padding is an integer multiple of 512;

S22. Process the padded data in groups of 512 bits. Each group is divided into 16 32-bit sub-blocks. Four 32-bit registers (4 symbols) are used to process the sub-blocks in a loop; four link variables are used as parameters to initialize MD5. , these four variables are: a=0x67452301, b=0xefcdab89, c=0x98badcfe, d=0x10325476;

S23. Perform four rounds of cyclic compression operation, each round has 16 steps, and each step uses a message. The message refers to data grouped into 512 bits; use step functions to update variables a, b, c, and d respectively. ;, the step function is Q _i+1 =Q _i+1 +((Q _i-3 +f _i (Q _i ,Q _i-1 ,Q _i-2 )+w _i +t _i )＜＜＜s _i );

S24. Concatenate four 32-bit sub-blocks to obtain a 128-bit value, which is the final hash value.

In this embodiment, a dynamic memory management method is proposed. The existing voxel hash hash table adopts the method of pre-allocating memory. Although this can improve the efficiency of indexing, it limits the scope of the three-dimensional reconstruction scene. This invention draws on the hash map (hash matching) method in the standard template library in C++11 to achieve dynamic expansion of memory. When the pre-allocated memory is full, the hash table memory is dynamically expanded to achieve a larger range. 3D reconstruction. The described dynamic memory management method includes the following steps: adding an integer variable in the hash table, which represents the number of available positions in the current hash table. When the value of the variable is close to 0, stop modifying the hash table. Insert, then open a new hash table of the same size and point the header of the new table to the tail of the old table, and then reconstruct the hash table; assuming that the size of the original hash table is n, the length of the new table at this time It becomes 2n. The hash value of all previously inserted elements is modulo 2n to get the new bucket number, and the elements in the old table are moved to the expanded new table; after the reconstruction is completed, the new bucket number can be obtained. Hash table for insertion operations.

Obviously, the above-mentioned embodiments of the present invention are only examples to clearly illustrate the present invention, and are not intended to limit the implementation of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. An exhaustive list of all implementations is neither necessary nor possible. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims (2)

1. A three-dimensional reconstruction method of objects based on a depth camera module, using an algorithm based on voxel hashing to achieve three-dimensional reconstruction; its characteristic is that a new hash function method is used in the voxel hashing process to calculate Hash value, the new hash function method includes the following steps: First, first convert the three-dimensional coordinates into a one-dimensional index through the following formula: In the formula, Δ is the size of the current device resolution, that is, the size of one voxel; Then, the calculated one-dimensional index data is converted into a hash value through the MD5 method; the specific steps include the following: S1. Set the world view under the current architecture: 3D reconstruction technology essentially establishes a large enough space voxel set to wrap the object to be reconstructed. This parent space voxel set is divided into three layers downwards. Substructure: chunk is a first-level substructure, and the space voxel set contains n*n*n chunk structures; block is a second-level substructure, and each chunk contains m*m*m block structures; voxel is three Level substructure, each block structure contains t*t*t voxel substructures; the variables m, n, t in the above are all positive integer variables, which can be set according to the specific situation; S2. Establish a "video stream" based on the TOF module, obtain its depth image, RGB image and point cloud at the current time, and read the TOF module's camera internal parameters K and the camera external parameters T at the current attitude and read them. Store it in relevant parameters; S3. Based on the color images of the previous and next frames, calculate the ORB feature points of the two color images before and after and select the corresponding point pairs. Use the corresponding points to obtain the world coordinates of the previous frame and the camera coordinates of the next frame from the depth map, thereby Solve the camera extrinsic parameters of the next frame so that the point cloud under the current frame is aligned with the world coordinate system where the standard frame is located; S4. Establish a scalable dynamic hash table on the GPU device side through the voxel hash method based on MD5 encoding; move the blocks within the view cone range from the host side to the hash table on the device side based on the frustum principle; S5. Based on the depth map of the current frame and combined with the information in the hash table, select the blocks that are valid within the truncation range of the depth area, and create a new dynamically allocated hash on the GPU device side through the memory dynamic management method. table, and move valid blocks to a new hash table. The new hash table records the location information corresponding to all valid blocks and the location of the voxel array it points to; S6. Traverse all block positions in the new hash table, and use CUDA to launch multi-threads to traverse all voxels in the block. According to the initial position of the starting point of the parent space voxel set in the world coordinate system and the real length of each voxel, Calculate the world coordinate system position of the voxel, use the back-projection formula to project it from the world coordinate system position to the pixel coordinate system position, and determine whether the world coordinate position of the voxel is truly visible within the depth frame range. If it is not visible, do not do it. Process, otherwise use the TSDF-truncated directed distance field formula to update its TSDF value and update the current weight; S7. The TSDF value is equivalent to a collection of isosurfaces. The position where the TSDF value is 0 is equivalent to the surface of the object. MarchingCubes technology is used to generate a triangular surface mesh and render it; S8. Copy all the voxels in the new hash table from the GPU device back to the host, record the content of the voxel at that location on the host, and release the hash table on the GPU device to prevent video memory leaks; S9. Read the new RGB image, depth image and point cloud at the current moment from the TOF module, and jump to step S3. 2. The object three-dimensional reconstruction method based on the depth camera module according to claim 1, characterized in that when the pre-allocated memory of the hash table of voxel hash is full, a memory dynamic management method is used to realize the hash table. Dynamic expansion of memory, the dynamic memory management method includes the following steps: add an integer variable in the hash table, the variable represents the number of available locations in the current hash table, when the value of the variable is close to 0, stop Insert into the hash table, then open a new hash table of the same size and point the head of the new table to the tail of the old table, and then reconstruct the hash table; assuming that the size of the original hash table is n, this When the length of the new table becomes 2n, the hash values of all previously inserted elements are modulo 2n to obtain a new bucket number, and the elements in the old table are moved to the expanded new table; the reconstruction is completed Then you can insert the new hash table.