CN114494589A - Three-dimensional reconstruction method, three-dimensional reconstruction device, electronic equipment and computer-readable storage medium - Google Patents

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method, a device, electronic equipment and a computer readable storage medium, wherein weak texture regions in images to be processed corresponding to a target scene are extracted, the weak texture regions are subjected to mask processing, so that the three-dimensional reconstruction of the weak texture regions is avoided in the three-dimensional reconstruction process, after dense point clouds of regions except the weak texture regions in the target scene are obtained, the weak texture regions are subjected to point cloud filling according to boundary lines and the dense point clouds of the weak texture regions, namely the weak texture regions are not directly subjected to three-dimensional reconstruction, but the weak texture regions are subjected to point cloud filling by utilizing point cloud data related to the boundary lines of the weak texture regions, so that the three-dimensional reconstruction process of the whole target scene is completed, the problem of unsatisfactory reconstruction effect of the weak texture regions in the prior art is effectively solved, and the phenomena of cavities, a large number of noise points and the like in a three-dimensional reconstruction result are avoided, and a better cavity removing effect is realized.

Description

Three-dimensional reconstruction method, apparatus, electronic device and computer-readable storage medium

technical field

The present invention relates to the field of image processing, and in particular, to a three-dimensional reconstruction method, apparatus, electronic device and computer-readable storage medium.

Background technique

The vision-based 3D reconstruction technology is to obtain the 2D image data information of the object through relevant instruments, then analyze and process the obtained data information, and finally reconstruct the contour information of the object surface in the real environment by using the relevant theory of 3D reconstruction. Vision-based 3D reconstruction technology calculates camera pose information and extracts sparse point cloud through image feature matching, but it is difficult to extract feature information for special areas such as weak texture areas and non-textured areas, which causes holes in dense reconstruction. Phenomenon.

At present, most of the solutions are to improve the feature information extraction of weak texture areas by optimizing the 3D reconstruction algorithm based on vision. This method is not ideal for the reconstruction of special areas, and there will still be voids in the 3D reconstruction results. .

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a three-dimensional reconstruction method, device, electronic device and computer-readable storage medium, so as to solve the problem that the reconstruction effect of the weak texture area in the existing three-dimensional reconstruction technology is not ideal, and the phenomenon of voids is prone to occur. The problem.

In order to achieve the above purpose, the technical solutions adopted in the embodiments of the present invention are as follows:

In a first aspect, the present invention provides a three-dimensional reconstruction method, the method comprising:

Extracting weak texture areas in each to-be-processed image corresponding to the target scene;

Perform mask processing on the weak texture area in each image to be processed to obtain each target image;

Performing three-dimensional reconstruction on the target area according to the target images to obtain a dense point cloud of the target area; the target area is an area other than the weak texture area in the target scene;

Point cloud filling is performed on the weak texture area according to the boundary line of the weak texture area and the dense point cloud.

In an optional implementation manner, the extraction of weak texture regions in each image to be processed corresponding to the target scene includes:

Use pixel values in each image to be processed or a pre-trained detection model to extract weak texture areas in each image to be processed, or select weak texture areas in each image to be processed corresponding to the target scene by the user.

In an optional implementation manner, the use of pixel values in each to-be-processed image or a pre-trained detection model to extract weak texture regions in each to-be-processed image includes:

Determine the first weak texture area in each of the to-be-processed images according to pixel values in each of the to-be-processed images;

Input each to-be-processed image into a pre-trained detection model to obtain a second weak texture area in each of the to-be-processed images; wherein the weak texture area includes a first weak texture area and a second weak texture area, and the first weak texture area The reflectivity of a weakly textured region is higher than that of the second weakly textured region.

In an optional implementation manner, performing point cloud filling on the weak texture area according to the boundary line of the weak texture area and the dense point cloud includes:

superimposing the boundary line of the weak texture area with the dense point cloud to obtain point cloud data at the boundary line of the weak texture area;

Perform distribution statistics on the point cloud data at the boundary line of the weak texture area, and select point cloud data within a preset range according to the statistical results;

Point cloud filling is performed on the weak texture area according to the point cloud data within the preset range.

In an optional implementation manner, the performing point cloud filling on the weak texture area according to the point cloud data within the preset range includes:

When the weak texture area is a water area, the water area is filled with a point cloud by taking the average elevation value of the point cloud data within the preset range as the elevation value of the water area.

In an optional implementation manner, the performing point cloud filling on the weak texture area according to the point cloud data within the preset range includes:

When the weak texture area is a non-water area, calculate the normal vector of each point cloud in the point cloud data within the preset range, and use the average value of the normal vectors of each point cloud as the vertical direction of the non-water area direction, and use the point cloud data within the preset range and the vertical direction of the non-water area to fill the non-water area with a point cloud.

In an optional implementation manner, the three-dimensional reconstruction of the target area according to each target image to obtain a dense point cloud of the target area includes:

Perform feature point extraction and feature point matching on each of the target images to obtain matching point pairs between the various target images;

Calculate camera parameters and sparse point cloud corresponding to each target image according to the matching point pair;

A dense point cloud is generated according to the camera parameters and the sparse point cloud.

In a second aspect, the present invention provides a three-dimensional reconstruction device, the device comprising:

A weak texture area extraction module, used to extract weak texture areas in each image to be processed corresponding to the target scene;

a mask processing module, configured to perform mask processing on the weak texture area in each image to be processed to obtain each target image;

a three-dimensional reconstruction module, configured to perform three-dimensional reconstruction on the target area according to the target images to obtain a dense point cloud of the target area; the target area is an area other than the weak texture area in the target scene;

The point cloud filling module is used for filling the weak texture area with point cloud according to the boundary line of the weak texture area and the dense point cloud.

In a third aspect, the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, the computer program being executed by the processor to achieve the following: The steps of the three-dimensional reconstruction method described in any one of the preceding embodiments.

In a fourth aspect, the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, realizes the three-dimensional reconstruction according to any one of the foregoing embodiments steps of the method.

The three-dimensional reconstruction method, device, electronic device, and computer-readable storage medium provided by the embodiments of the present invention extract the weak texture area in each to-be-processed image corresponding to the target scene, and perform mask processing on the weak texture area, so that the In the 3D reconstruction process, the 3D reconstruction of the weak texture area is avoided. After obtaining the dense point cloud of the area other than the weak texture area in the target scene, according to the boundary line and dense point cloud of the weak texture area, the weak texture area is obtained. The texture area is filled with point clouds, that is, the 3D reconstruction of the weak texture area is not directly performed, but the point cloud data related to the boundary line of the weak texture area is used to fill the weak texture area, thus completing the 3D reconstruction of the entire target scene. The reconstruction process effectively solves the problem that the reconstruction effect of the weak texture area in the prior art is not ideal, avoids the phenomenon of holes and a lot of noise in the 3D reconstruction result, and achieves a better hole removal effect.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore do not It should be regarded as a limitation of the scope, and for those of ordinary skill in the art, other related drawings can also be obtained according to these drawings without any creative effort.

Figure 1 shows a schematic diagram of a point cloud void;

FIG. 2 shows a schematic flowchart of a three-dimensional reconstruction method provided by an embodiment of the present invention;

Figure 3 shows a schematic diagram of a weakly textured region that is prone to voids;

FIG. 4 shows another schematic flowchart of a three-dimensional reconstruction method provided by an embodiment of the present invention;

Fig. 5 shows the schematic diagram of the original image of feature point matching;

Fig. 6 shows the result schematic diagram of feature point matching;

Figure 7 shows a schematic diagram of epipolar geometric constraints;

FIG. 8 shows another schematic flowchart of a three-dimensional reconstruction method provided by an embodiment of the present invention;

FIG. 9 shows a schematic diagram of a three-dimensional reconstruction effect for removing voids;

FIG. 10 shows a functional block diagram of a three-dimensional reconstruction device provided by an embodiment of the present invention;

FIG. 11 shows a schematic block diagram of an electronic device provided by an embodiment of the present invention.

Icon: 500-three-dimensional reconstruction device; 700-electronic equipment; 510-weak texture area extraction module; 520-mask processing module; 530-three-dimensional reconstruction module; 540-point cloud filling module; 710-memory; 720-processor; 730 - Communication Module.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. The components of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

Thus, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present invention.

It should be noted that relational terms such as the terms "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Aiming at the problem that holes are prone to appear in 3D reconstruction, most of the current solutions are to improve the feature information extraction of weak texture areas by optimizing the vision-based 3D reconstruction algorithm, so as to achieve the purpose of eliminating holes, but the current improved algorithm has not achieved the ideal. The effect of 3D reconstruction of weak texture areas will appear holes and a lot of noise (as shown in Figure 1), which will affect the accuracy of the point cloud data after 3D reconstruction. In addition to vision-based 3D reconstruction, there is also lidar-based 3D reconstruction. Lidar is an optical sensor technology that irradiates the target object when scanning a laser beam, and measures the distance and position of the object by measuring the time it takes for the irradiated object to bounce back.

Due to the method of improving the reconstruction of weak texture regions by optimizing the vision-based 3D reconstruction algorithm, it has not yet achieved very ideal results. Compared with vision-based 3D reconstruction equipment, lidar equipment is much more expensive, and for large scenes, the accuracy of point positions will also decrease with distance. Therefore, the embodiments of the present invention propose a three-dimensional reconstruction method, apparatus, electronic device, and computer-readable storage medium, which extract weak texture regions in each to-be-processed image corresponding to the target scene, and in the three-dimensional reconstruction process, through The weak texture area is masked to avoid the 3D reconstruction of the weak texture area. After obtaining the dense point cloud of the area except the weak texture area in the target scene, according to the boundary line of the weak texture area and the dense point cloud , fill the weak texture area with point cloud, that is, instead of directly reconstructing the weak texture area, use the point cloud data related to the boundary line of the weak texture area to fill the weak texture area, thus completing the whole goal. The 3D reconstruction process of the scene effectively solves the problem that the reconstruction effect of the weak texture area in the prior art is not ideal, avoids the phenomenon of holes and a lot of noise in the 3D reconstruction results, and achieves a better hole removal effect; The weak texture area is masked in the reconstruction process, so there will be no large amount of noise in the obtained 3D reconstruction result, which effectively improves the accuracy of the point cloud data after 3D reconstruction.

Hereinafter, the three-dimensional reconstruction method provided by the embodiments of the present invention will be described in detail.

Please refer to FIG. 2 , which is a schematic flowchart of a three-dimensional reconstruction method provided by an embodiment of the present invention. It should be noted that the three-dimensional reconstruction method of the embodiment of the present invention is not limited by the specific order of FIG. 2 and the following. It should be understood that in other embodiments, the order of some steps in the three-dimensional reconstruction method of the embodiment of the present invention may be based on It is actually necessary to exchange each other, or some of the steps can also be omitted or deleted. The three-dimensional reconstruction method can be applied in electronic devices, and the specific flow shown in FIG. 2 will be described in detail below.

Step S201, extracting weak texture regions in each to-be-processed image corresponding to the target scene.

In this embodiment, the target scene is a scene to be reconstructed, which may be an outdoor scene or an indoor scene, which is not limited. For example, the flight path of the drone can be planned. During the flight of the drone, the camera carried by the drone is used to capture the target scene, and the captured image is sent to the electronic device. In this way, the electronic device can obtain each image to be processed corresponding to the target scene.

In this embodiment, each image to be processed corresponding to the target scene may include images of the target scene from different viewing angles captured by one or more cameras.

After obtaining each to-be-processed image corresponding to the target scene, the electronic device can use the deep learning framework and combined with the spectral information of ground objects to extract the weak texture area, thereby obtaining the weak texture area in each to-be-processed image corresponding to the target scene. Among them, the weak texture area can be understood as the area with only a small amount of texture or no texture.

Step S202, performing mask processing on weak texture regions in each image to be processed to obtain each target image.

In this embodiment, considering that in the process of 3D reconstruction, it is difficult to extract feature information for weak texture areas, so before performing 3D reconstruction, mask processing is performed on the weak texture areas in each image to be processed, that is, to be processed. The weak texture area on the image is masked, so that the weak texture area does not participate in the subsequent 3D reconstruction process. It can be understood that the target image is an image obtained by masking the weak texture area in the image to be processed.

In step S203, three-dimensional reconstruction is performed on the target area according to each target image to obtain a dense point cloud of the target area; the target area is an area other than the weak texture area in the target scene.

In this embodiment, the target area can be understood as a non-weak texture area. Since the target image is obtained by masking the weak texture area in the image to be processed, the non-weak texture area in the target scene can be realized based on each target image. 3D reconstruction of weakly textured regions, resulting in dense point clouds of non-weakly textured regions.

Step S204, filling the weak texture area with point cloud according to the boundary line and dense point cloud of the weak texture area.

In this embodiment, after completing the three-dimensional reconstruction of the non-weak texture area in the target scene, for the weak texture area in the target scene, the boundary line of the weak texture area and the dense point cloud can be used to obtain the boundary line correlation of the weak texture area. point cloud data, and then complete the point cloud filling of the weak texture area according to the point cloud data related to the boundary line of the weak texture area. In this way, the 3D reconstruction result of the entire target scene can be obtained.

It can be seen that the three-dimensional reconstruction method provided by the embodiment of the present invention avoids the weak texture area in the three-dimensional reconstruction process by extracting the weak texture area in each to-be-processed image corresponding to the target scene, and performing mask processing on the weak texture area. For 3D reconstruction of textured area, after obtaining the dense point cloud of the area except the weakly textured area in the target scene, fill the weakly textured area with point cloud according to the boundary line and dense point cloud of the weakly textured area, that is, it is not directly For the 3D reconstruction of the weak texture area, the point cloud data related to the boundary line of the weak texture area is used to fill the weak texture area with point clouds, thus completing the 3D reconstruction process of the entire target scene, effectively solving the problems in the prior art. The problem of unsatisfactory reconstruction effect in weak texture area avoids the phenomenon of holes and a lot of noise in the 3D reconstruction results, and achieves a better hole removal effect. Since the weak texture area is masked in the process of 3D reconstruction, there will not be a lot of noise in the obtained 3D reconstruction result, which effectively improves the accuracy of the point cloud data after 3D reconstruction.

In practical applications, weak texture regions can be extracted based on different methods. That is, the above-mentioned step S201 may include: extracting the weak texture area in each to-be-processed image by using the pixel value in each to-be-processed image or a pre-trained detection model, or selecting the weak texture in each to-be-processed image corresponding to the target scene by the user area.

That is to say, after the electronic device obtains each image to be processed corresponding to the target scene, the user can select the weak texture area in each image to be processed, or the electronic device can use the pixel value in each The trained detection model extracts weakly textured regions in each image to be processed.

Among them, when using the pixel value in the image to be processed to extract the weak texture area, by traversing the pixels in the image to be processed, when the current pixel value is greater than the preset value, the area growth method is used to determine the pixel value of the adjacent pixel, if the adjacent pixel value is If the pixel value of the domain pixel is greater than the preset value, it continues to diffuse outward until the pixel value is less than the preset value. By this method, the weak texture area in the image to be processed can be obtained. The preset value can be set according to actual needs, for example, set to 240.

The pre-trained detection model can be a deep learning convolutional neural network. When using the pre-trained detection model to extract weak texture areas, each image to be processed can be input into the pre-trained detection model, so as to obtain the weak texture in each image to be processed. texture area. The specific method can be as follows:

(1) Use historical UAV images to label weak texture areas, and non-labeled areas are the background.

(2) Data enhancement is performed on the original data and label data to increase the amount of data for model training; the enhancement methods include cropping, rotating, flipping, scaling, and adding noise.

(3) Divide the prepared data set into three parts according to a certain ratio such as 6:2:2: training set, validation set and test set. First, the training set data is sent to the convolutional neural network. The convolutional neural network generally includes an input layer, a convolutional layer, a pooling layer, and a fully connected layer. The convolutional neural network is used to assign an initial class label to each pixel. Multilayers can effectively capture local features in an image and nest many such modules together in a hierarchical fashion. After the detection model is trained, the detection model is evaluated on the validation set, and a set of hyperparameters with the best performance is selected for testing on the test set. In the process of continuous adjustment of hyperparameters, the detection model performs optimally on the test set.

(4) Use the optimal detection model to complete the classification of weak texture regions in each image to be processed, and obtain the weak texture regions and corresponding categories in each image to be processed. Among them, the boundary line of the weak texture area can be obtained by vectorizing the classification result.

In practical applications, there may be various types of weak texture areas that are likely to cause holes (as shown in Figure 3). Using the above different methods for extracting weak texture areas, different types of weak texture areas in the image to be processed can be extracted. . That is, the above steps use the pixel values in each image to be processed or a pre-trained detection model to extract weak texture regions in each image to be processed, which may include:

Determine the first weak texture area in each to-be-processed image according to the pixel value in each to-be-processed image; input each to-be-processed image into a pre-trained detection model to obtain the second weakly textured area in each to-be-processed image; The texture area includes a first weak texture area and a second weak texture area, and the reflectivity of the first weak texture area is higher than that of the second weak texture area.

In this embodiment, the first weak texture area is a strong reflective area with high reflective rate, such as a strong reflective road surface. The second weak texture area is an area other than the strong reflection area that is prone to voids, and its reflectivity is lower than that of the first weak texture area, such as water, roofs, and solar reflective panels.

For weak texture areas such as strong reflective areas, the pixel intensity value can be used to determine. For weak texture areas such as waters, roofs and solar reflectors, the deep learning convolutional neural network (detection model) is used to complete the extraction, so that different types of weak texture areas can be extracted by different methods.

It can be seen that the embodiment of the present invention adopts different methods to extract weak texture areas for different types of weak texture areas that are likely to cause voids; for strong reflective areas, it is determined according to the pixel intensity value combined with the area growth method; As well as weak texture areas such as solar reflectors, based on the deep learning framework, the pre-trained detection model is used to extract weak texture areas. In this way, the accurate extraction of the weak texture area that is likely to cause voids is achieved, and it is ensured that no voids and a large number of noises appear in the 3D reconstruction result.

In this embodiment, the process of three-dimensional reconstruction of the target area according to each target image may include: feature point extraction and matching, sparse reconstruction and dense reconstruction, and finally obtain a dense point cloud of the target area. Based on this, please refer to FIG. 4 , the above-mentioned step S203 may include the following sub-steps:

In sub-step S2031, feature point extraction and feature point matching are performed on each target image to obtain matching point pairs between each target image.

Among them, the commonly used feature point detections are SIFT (Scale Invariant Feature Transform, scale invariant feature transform), SURF (Speed Up Robust Feature, accelerated robust feature), ORB (Oriented FAST and Rotated BRIEF), Harris, etc. For example, in this embodiment, the SIFT algorithm can be used to extract feature points, and the SIFT feature points remain invariant to rotation, scale scaling, brightness, etc., and are very stable local features. After extracting SIFT feature points, feature point matching is performed to obtain matching point pairs between each target image.

Sub-step S2032: Calculate camera parameters and sparse point clouds corresponding to each target image according to the matched point pairs.

In this embodiment, after the matching point pairs between each target image are obtained, sparse reconstruction can be performed through the Sfm (Structure from motion, motion recovery structure) technology, thereby estimating the camera parameters corresponding to each target image, and according to the triangulation Get a sparse point cloud. Among them, Sfm is a 3D reconstruction algorithm that can recover the camera pose through two or more scenes (pictures) and reconstruct sparse 3D coordinate points. Among them, the camera parameters include camera internal parameters and camera external parameters (ie camera pose, including camera rotation and camera translation).

Sub-step S2033, generate a dense point cloud according to the camera parameters and the sparse point cloud.

In this embodiment, multi-view stereo matching (MVS, Multi View Stereo) can be used to complete the dense reconstruction. consistency) functions, such as sum of squared errors (SSD, Sum of Squared Difference), sum of absolute errors (SAD, Sum of Absolute Differences), and normalized cross-correlation (NCC, Normalized CrossCorrelation), etc., to achieve dense matching between target images, Then, the dense point cloud corresponding to the target area in the target scene is reconstructed.

For example, the dense matching based on depth map fusion may include the following process: (1) Select neighborhood images for each target image to form a stereo image group (candidate set). During the optimal selection of the neighborhood image group, a sufficiently large baseline between images should be considered. (2) Calculate the depth of each pixel of the reference image, the classic framework: calculate the matching cost, cost aggregation, calculate the depth value, and finally optimize the depth, and generate the depth map and normal map. Use gray absolute value difference (AD, Absolute Differences), gray absolute value difference sum, normalized correlation coefficient, mutual information (MI, Mutual Information) method or Census transform (CT) method as the matching cost calculation method of two pixels . (3) According to certain criteria, such as adjacent pixels should have continuous depth values, the cost matrix is optimized. The depth map of each target image is calculated using information transfer strategies, including spatial propagation, view propagation, and temporal propagation. (4) Use the left-right consistency check algorithm to remove the wrong depth caused by occlusion and noise, remove the small connected domain algorithm to remove the isolated abnormal points, and use the median filter, bilateral filter and other smoothing algorithms to smooth the depth map to improve the depth. the quality of the graph. (5) Integrate multiple depth maps into a unified and enhanced scene representation, while reducing left-right inconsistencies, complete the fusion of depth maps, and finally obtain dense point clouds.

In one embodiment, the above sub-step S2032 may specifically include: determining two initial target images from each target image, and calculating the cameras corresponding to the two initial target images according to the matching point pairs between the two initial target images parameters, and according to the camera parameters corresponding to the two initial target images, calculate the sparse point cloud corresponding to the two initial target images; continuously add new target images, according to the currently generated sparse point cloud and the feature points in the new target image The matching relationship is calculated, the camera parameters and sparse point cloud corresponding to the new target image are calculated, and the camera parameters and sparse point cloud corresponding to the new target image are optimized until the camera parameters and sparse point cloud corresponding to all target images are calculated; The camera parameters and sparse point cloud corresponding to each target image are adjusted to minimize the sum of the reprojection errors of all points with the same name projected onto the corresponding target image.

That is to say, the sparse reconstruction in the 3D reconstruction process can include three parts: initial reconstruction, incremental reconstruction and global optimization.

In the initial reconstruction process, two-view images with more visible areas between cameras (two initial target images, as shown in Figure 5) are selected for initialization, and the matching point pairs between the two-view images (as shown in Figure 6) are selected for initialization. ), using the fast approximate nearest neighbor algorithm to get more accurate point pairs. Next, use epipolar geometric constraints (as shown in Figure 7) to complete the initial camera pose estimation: denote P as any point in space, p1 and p2 as the matching point pair in the left and right images, and O1 and O2 as the camera light of the left and right images. The center position, R and t are the rotation matrix and translation matrix of the camera motion (ie, the parameters outside the camera). According to the pinhole camera model, we can get:

s1p1=KP, s2p2=K(RP+t) (1)

Among them, s1 and s2 represent the depth information corresponding to the two matching points, and K is the camera intrinsic parameter matrix (ie, the camera intrinsic parameter). Using the form of homogeneous coordinates, since s1 and s2 are non-zero values, you can get:

p1=KP, p2=K(RP+t) (2)

The point coordinates of p1 and p2 on the normalized plane are marked as x1 and x2. The above formula (2) can be sorted as:

x2=Rx1+t (3)

Multiply both sides of the above formula (3) by t ^∧ and the left by x2 ^T at the same time, we can get:

x2 ^T t ^∧ Rx1=0 (4)

Among them, t ^∧ represents the anti-symmetric matrix of the translation matrix. By introducing the anti-symmetric matrix, the cross product of the matrix can be changed into a point product, that is, a linear transformation, which is beneficial to the calculation; x2 ^T represents the transpose of the coordinate vector x2.

Re-substituting p1 and p2 in formula (4), we can get:

p2 ^T K ^-T t ^∧ RK ^-1 p1=0 (5)

The essential matrix (Essential Matrix) E=t ^∧ R and the Fundamental Matrix (Fundamental Matrix) F=K ^-T EK ^-1 can be obtained by formula (5).

The homography matrix describes the mapping relationship between two planes, and is usually used for the transformation relationship of points on the same plane. The plane equation is written as:

Among them, n represents the normal vector of the plane, d represents the distance from the origin to the plane, and P is the space point.

According to the coplanar relationship of the feature matching points, we can get:

The homography matrix can be obtained by formula (7)

Using the obtained matching point pairs, the fundamental matrix F, essential matrix E, and homography matrix H can be estimated according to the classic eight-point method; then the camera motion R and t information can be recovered by using singular value decomposition (SVD).

According to Figure 7, the equation of motion between point pairs can be obtained:

s1x1=s2Rx2+t (8)

(x1的反对称矩阵)，可得：where s1 and s2 are the depth values of the feature points. Multiply the left and right sides of the above formula (8) by one

(Antisymmetric matrix of x1), we get:

s1x1 ^∧ x1＝0＝s2x1 ^∧ Rx2+x1 ^∧ t (9)

In the above formula (9), the left side is 0, and the right side can be called an equation of s2. Since R and t are already known, the depth value s2 can be solved by using least squares. According to the solved s2, it can also be solved. out s1. In this way, the depth values of the feature points on the two initial target images can be obtained, and then the spatial coordinates of each feature point, that is, the sparse point cloud, can be obtained.

Incremental reconstruction is to reconstruct other images in turn on the basis of the initial reconstruction, and the process is carried out in a cyclic and iterative manner. The loop mainly includes three sub-processes: camera positioning based on Perspective-n-Point (PnP), scene expansion based on triangulation, and camera parameters and scene point cloud based on BA (Bundle Adjustment). Optimization. This iterative process loops until all cameras are successfully positioned or there are no cameras to continue positioning. (1) PnP is a method for solving the motion of 3D to 2D point pairs, which describes how to estimate the pose of the camera when n 3D space points and their projected positions are known. During the initial reconstruction process, we get the 3D coordinates of the feature points in the seed image, and during the incremental reconstruction process, we construct a 3D-to-2D point pair motion problem by adding images. Among them, there are many solutions to the PnP problem, such as P3P, direct linear transformation (DLT), EPnP (Efficient PnP), UPnP, etc., which use 3 pairs of points to estimate the pose. Multiply the problem and solve it iteratively. (2) Triangulation-based scene expansion, taking the third target image as an example, according to the 3D points generated by the initial reconstruction and the 2D matching points of the third image, the PnP problem is constructed, and the third image is estimated by methods such as P3P/EPnP The camera pose corresponding to the target image, and then continue to triangulate to obtain the 3D coordinates of more points, repeat this step, and finally calculate the camera pose and 3D points corresponding to all target images. (3) Optimization of camera parameters and scene point cloud based on BA, in the process of solving camera parameters, generally use P3P/EPnP and other methods to estimate the camera pose, and then construct a least squares optimization problem to adjust the estimated value (ie BA).

The global optimization is to adjust the camera pose and sparse point cloud through the beam method adjustment algorithm after solving the camera parameters and sparse point cloud corresponding to all target images, so that all points with the same name are projected onto the corresponding target image. The sum of the errors is the smallest, and finally the optimized camera parameters and sparse point clouds corresponding to all target images are obtained. Among them, the point with the same name is the image point with the same name, that is, the image point formed by the same object point on different images.

p2=H ₂₁ p1, p1=H ₁₂ p2 (10)

The reprojection error is expressed as:

为估计得到的两幅目标图像中的完全匹配点对，

为重投影误差过程中的估计值。in

To estimate the exact matching point pairs in the two target images,

is the estimated value during the reprojection error process.

In one embodiment, the point cloud filling of the weak texture area may be performed according to the relevant point cloud data at the boundary line of the weak texture area. Please refer to FIG. 8 , the above-mentioned step S204 may include the following sub-steps:

Sub-step S2041, superimpose the boundary line of the weak texture area with the dense point cloud to obtain point cloud data at the boundary line of the weak texture area.

Sub-step S2042, perform distribution statistics on the point cloud data at the boundary line of the weak texture area, and select point cloud data within a preset range according to the statistical result.

For example, normal distribution statistics can be performed on the point cloud data at the boundary line of the weak texture area, and point cloud data within two standard deviation ranges (or other standard deviation ranges) can be extracted according to the statistical results.

Sub-step S2043, fill the weak texture area with point cloud according to the point cloud data within the preset range.

In this embodiment, since the point cloud data within the preset range belongs to the point cloud data at the boundary line of the weak texture area, filling the weak texture area with the point cloud according to the point cloud data within the preset range can effectively guarantee 3D reconstruction effect.

In one embodiment, the above sub-step S2043 may include: when the weak texture area is water, fill the water with a point cloud by using the average elevation of point cloud data within a preset range as the elevation value of the water.

In another embodiment, the above sub-step S2043 may include: when the weak texture area is a non-water area, calculating the normal vector of each point cloud in the point cloud data within a preset range, and calculating the normal vector of each point cloud The average value of is used as the vertical direction of the non-water area, and the non-water area is filled with point clouds using the point cloud data within the preset range and the vertical direction of the non-water area. The color of the point cloud filled by the non-water area is the color of the position of the non-water area in the image to be processed.

That is to say, when filling the point cloud of the weak texture area, it can be processed separately according to the type of the weak texture area (water area, non-water area). For water areas, the average elevation value of the point cloud data within the preset range is calculated as the elevation value of the water area to complete the point cloud filling of the water area; for non-water areas, the method of calculating each point cloud in the point cloud data within the preset range is calculated. vector, obtain the average value of the normal vectors of each point cloud as the vertical direction of the non-water area, and then use the point cloud data in the preset range and the average value of the normal vector to perform plane fitting to complete the point cloud in the non-water area. Fill, the color of the fill area point cloud uses the color of the same location area in the image. As shown in FIG. 9 , it is a schematic diagram of the 3D reconstruction effect of removing voids. In this way, the occurrence of holes and a large number of noises in the 3D reconstruction result is effectively avoided, and a better hole removal effect and a 3D reconstruction effect for weak texture areas are achieved.

In order to perform the corresponding steps in the foregoing embodiments and various possible manners, an implementation manner of a three-dimensional reconstruction apparatus is given below. Please refer to FIG. 10 , which is a functional block diagram of a three-dimensional reconstruction apparatus 500 according to an embodiment of the present invention. It should be noted that the basic principles and technical effects of the three-dimensional reconstruction device 500 provided in this embodiment are the same as those in the above-mentioned embodiments. For the sake of brief description, for the parts not mentioned in this embodiment, reference may be made to the above-mentioned embodiments. corresponding content. The three-dimensional reconstruction device 500 includes a weak texture region extraction module 510 , a mask processing module 520 , a three-dimensional reconstruction module 530 and a point cloud filling module 540 .

The weak texture region extraction module 510 is used for extracting weak texture regions in each image to be processed corresponding to the target scene.

It can be understood that the weak texture region extraction module 510 can perform the above step S201.

The mask processing module 520 is configured to perform mask processing on weak texture regions in each image to be processed to obtain each target image.

It can be understood that the mask processing module 520 can perform the above step S202.

The 3D reconstruction module 530 is used for performing 3D reconstruction on the target area according to each target image to obtain a dense point cloud of the target area; the target area is the area in the target scene except the weak texture area.

It can be understood that the three-dimensional reconstruction module 530 can perform the above step S203.

The point cloud filling module 540 is used to fill the weak texture area with point cloud according to the boundary line and dense point cloud of the weak texture area.

It can be understood that the point cloud filling module 540 can perform the above step S204.

Optionally, the weak texture region extraction module 510 is specifically configured to extract the weak texture region in each to-be-processed image by using the pixel value in each to-be-processed image or a pre-trained detection model, or select each corresponding to the target scene by the user. Weak textured regions in the image to be processed.

Wherein, the weak texture region extraction module 510 is used to determine the first weak texture region in each to-be-processed image according to the pixel value in each to-be-processed image; input each to-be-processed image into a pre-trained detection model to obtain each to-be-processed image The second weak texture area in; wherein, the weak texture area includes a first weak texture area and a second weak texture area, and the reflectivity of the first weak texture area is higher than that of the second weak texture area.

Optionally, the three-dimensional reconstruction module 530 is specifically used to perform feature point extraction and feature point matching on each target image to obtain matching point pairs between each target image; calculate the camera parameters corresponding to each target image according to the matching point pairs and Sparse point cloud; generate dense point cloud based on camera parameters and sparse point cloud.

The three-dimensional reconstruction module 530 is used to determine two initial target images from each target image, calculate the camera parameters corresponding to the two initial target images according to the matching point pairs between the two initial target images, and calculate the corresponding camera parameters according to the two initial target images. The camera parameters corresponding to the initial target images are used to calculate the sparse point clouds corresponding to the two initial target images; new target images are continuously added, and new target images are calculated according to the matching relationship between the currently generated sparse point cloud and the feature points in the new target image. The camera parameters and sparse point cloud corresponding to the target image, and optimize the camera parameters and sparse point cloud corresponding to the new target image until the camera parameters and sparse point cloud corresponding to all target images are calculated; Camera parameters and sparse point cloud to minimize the sum of the reprojection errors of all points with the same name projected onto the corresponding target image.

It can be understood that the three-dimensional reconstruction module 530 can also perform the above sub-steps S2031-S2033.

Optionally, the point cloud filling module 540 is specifically used to superimpose the boundary line of the weak texture area and the dense point cloud to obtain point cloud data at the boundary line of the weak texture area; The point cloud data is distributed and counted, and the point cloud data within a preset range is selected according to the statistical results; the weak texture area is filled with point clouds according to the point cloud data within the preset range.

Wherein, the point cloud filling module 540 is specifically configured to fill the water area with the point cloud by using the average elevation value of the point cloud data within the preset range as the elevation value of the water area when the weak texture area is water. When the weak texture area is a non-water area, calculate the normal vector of each point cloud in the point cloud data within the preset range, take the average value of the normal vectors of each point cloud as the vertical direction of the non-water area, and use the preset The point cloud data in the range and the vertical direction of the non-water area are filled with point clouds. The color of the point cloud filled by the non-water area is the color of the position of the non-water area in the image to be processed.

It can be understood that the point cloud filling module 540 may also perform the above sub-steps S2041 to S2043.

It can be seen that, in the three-dimensional reconstruction device provided by the embodiment of the present invention, the weak texture region extraction module extracts the weak texture regions in the images to be processed corresponding to the target scene, and the mask processing module performs mask processing on the weak texture regions in the images to be processed. , obtain each target image, the 3D reconstruction module reconstructs the target area in 3D according to each target image, and obtains the dense point cloud of the target area; the point cloud filling module performs point cloud filling on the weak texture area according to the boundary line and dense point cloud of the weak texture area. Cloud filling. In this way, by extracting the weak texture area in each image to be processed corresponding to the target scene, and performing mask processing on the weak texture area, the 3D reconstruction of the weak texture area is avoided in the 3D reconstruction process, and the target scene is obtained after the 3D reconstruction of the weak texture area is avoided. After the dense point cloud in the area except the weak texture area, the weak texture area is filled with point cloud according to the boundary line and dense point cloud of the weak texture area, that is, the 3D reconstruction of the weak texture area is not directly performed, but The weak texture area is filled with point cloud data related to the boundary line of the weak texture area, thereby completing the 3D reconstruction process of the entire target scene, and effectively solving the problem of unsatisfactory reconstruction effect for the weak texture area in the prior art. It avoids the phenomenon of holes and a lot of noise in the 3D reconstruction results, and achieves a better hole removal effect. Since the weak texture area is masked in the 3D reconstruction process, a lot of noise will not appear in the obtained 3D reconstruction result, which effectively improves the accuracy of the point cloud data after 3D reconstruction.

Please refer to FIG. 11 , which is a schematic block diagram of an electronic device 700 according to an embodiment of the present invention. The electronic device 700 includes a memory 710 , a processor 720 and a communication module 730 . The elements of the memory 710 , the processor 720 and the communication module 730 are directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, these elements may be electrically connected to each other through one or more communication buses or signal lines.

The memory 710 is used for storing programs or data. The memory 710 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read only memory (Programmable Read-Only Memory, PROM), erasable Read-only memory (ErasableProgrammable Read-Only Memory, EPROM), Electrically Erasable Programmable Read-Only Memory (Electric ErasableProgrammable Read-Only Memory, EEPROM), etc.

The processor 720 is used to read/write data or programs stored in the memory 710, and perform corresponding functions. For example, when the computer program stored in the memory 710 is executed by the processor 720, the three-dimensional reconstruction methods disclosed in the above embodiments can be implemented.

The communication module 730 is used to establish a communication connection between the electronic device 700 and other communication terminals through the network, and to send and receive data through the network.

It should be understood that the structure shown in FIG. 11 is only a schematic diagram of the structure of the electronic device 700 , and the electronic device 700 may further include more or less components than those shown in FIG. 11 , or have different configurations from those shown in FIG. 11 . . Each component shown in FIG. 11 may be implemented in hardware, software, or a combination thereof.

Embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor 720, implements the three-dimensional reconstruction methods disclosed in the foregoing embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may also be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality and possible implementations of apparatuses, methods and computer program products according to various embodiments of the present invention. operate. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

In addition, each functional module in each embodiment of the present invention may be integrated to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A three-dimensional reconstruction method, wherein the method comprises: Extracting weak texture areas in each to-be-processed image corresponding to the target scene; Perform mask processing on the weak texture area in each image to be processed to obtain each target image; Performing three-dimensional reconstruction on the target area according to the target images to obtain a dense point cloud of the target area; the target area is an area other than the weak texture area in the target scene; Point cloud filling is performed on the weak texture area according to the boundary line of the weak texture area and the dense point cloud. 2. The method according to claim 1, wherein the extracting weak texture regions in each to-be-processed image corresponding to the target scene comprises: Use pixel values in each image to be processed or a pre-trained detection model to extract weak texture areas in each image to be processed, or select weak texture areas in each image to be processed corresponding to the target scene by the user. 3. The method according to claim 2, wherein the extraction of weak texture regions in each to-be-processed image by using pixel values in each to-be-processed image or a pre-trained detection model, comprising: Determine the first weak texture area in each of the to-be-processed images according to pixel values in each of the to-be-processed images; Input each to-be-processed image into a pre-trained detection model to obtain a second weak texture area in each of the to-be-processed images; wherein the weak texture area includes a first weak texture area and a second weak texture area, and the first weak texture area The reflectivity of a weakly textured region is higher than that of the second weakly textured region. 4 . The method according to claim 1 , wherein, according to the boundary line of the weak texture area and the dense point cloud, filling the weak texture area with a point cloud, comprising: 5 . superimposing the boundary line of the weak texture area with the dense point cloud to obtain point cloud data at the boundary line of the weak texture area; Perform distribution statistics on the point cloud data at the boundary line of the weak texture area, and select point cloud data within a preset range according to the statistical results; Point cloud filling is performed on the weak texture area according to the point cloud data within the preset range. 5 . The method according to claim 4 , wherein the performing point cloud filling on the weak texture area according to the point cloud data within the preset range, comprising: 5 . When the weak texture area is a water area, the water area is filled with a point cloud by taking the average elevation value of the point cloud data within the preset range as the elevation value of the water area. 6 . The method according to claim 4 , wherein the performing point cloud filling on the weak texture area according to the point cloud data within the preset range, comprising: 6 . When the weak texture area is a non-water area, calculate the normal vector of each point cloud in the point cloud data within the preset range, and use the average value of the normal vectors of each point cloud as the vertical direction of the non-water area direction, and use the point cloud data within the preset range and the vertical direction of the non-water area to fill the non-water area with a point cloud. 7. The method according to claim 1, wherein the three-dimensional reconstruction of the target area according to each target image to obtain a dense point cloud of the target area, comprising: Perform feature point extraction and feature point matching on each of the target images to obtain matching point pairs between the various target images; Calculate camera parameters and sparse point cloud corresponding to each target image according to the matching point pair; A dense point cloud is generated according to the camera parameters and the sparse point cloud. 8. A three-dimensional reconstruction device, wherein the device comprises: A weak texture area extraction module, used to extract weak texture areas in each image to be processed corresponding to the target scene; a mask processing module, configured to perform mask processing on the weak texture area in each image to be processed to obtain each target image; a three-dimensional reconstruction module, configured to perform three-dimensional reconstruction on the target area according to the target images to obtain a dense point cloud of the target area; the target area is an area other than the weak texture area in the target scene; The point cloud filling module is used for filling the weak texture area with point cloud according to the boundary line of the weak texture area and the dense point cloud. 9. An electronic device, characterized in that it comprises a processor, a memory, and a computer program stored on the memory and running on the processor, the computer program being executed by the processor to achieve the right The steps of the three-dimensional reconstruction method of any one of claims 1-7. 10. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the three-dimensional system according to any one of claims 1-7 is realized. Steps of the reconstruction method.

Images