CN116543109A - Hole filling method and system in three-dimensional reconstruction - Google Patents

Abstract

The invention discloses a hole filling method and a hole filling system in three-dimensional reconstruction, which belong to the technical field of three-dimensional reconstruction, and the technical problem to be solved by the invention is how to repair holes in the three-dimensional reconstruction process, keep the human body, the target and the scene in the real world as consistent as possible in the reconstruction process, realize high-fidelity three-dimensional reconstruction, and adopt the following technical scheme: the method comprises the steps of carrying out multi-dimensional hole detection on an input image during primary filling, carrying out accurate screening and smooth filtering, introducing a mean square error, and modifying holes in a target and a scene according to a distance threshold; and on the basis of primary patching, reversely detecting a target in the original image at the hole position, carrying out matching optimization with the primary patching result, calculating texture information of the original image at the hole position, carrying out texture mapping on the basis of primary patching, obtaining an optimized hole patching result, and improving the consistency of the target or scene in the real world and the three-dimensional reconstruction model.

Description

Hole filling method and system in 3D reconstruction

technical field

The invention relates to the technical field of three-dimensional reconstruction, in particular to a hole filling method and system in three-dimensional reconstruction.

Background technique

3D reconstruction, in a broad sense, refers to the restoration and reconstruction of 3D objects or 3D scenes in the physical world. Reconstruction establishes a mathematical model suitable for computer representation and processing. The actual 3D reconstruction process is the inverse process of the image description of the human body, objects, scenes and other objects, so 3D reconstruction technology is the key technology for constructing an expression of the objective world in the digital world.

According to different purposes, 3D reconstruction can be divided into sparse reconstruction and dense reconstruction. Sparse reconstruction focuses on reconstructing the three-dimensional coordinates of image feature points, which are mainly used for positioning. Dense mapping is to reconstruct the entire image or most of the pixels in the image, which plays an important role in navigation and obstacle avoidance.

In terms of modeling methods, 3D reconstruction is divided into traditional modeling methods and methods based on computer vision and computer graphics. The traditional modeling method generally refers to the use of modeling software (such as AutoCAD, 3DMax, Unity, etc.) for forward design and modeling, and at the same time, reverse reconstruction of existing objects and scenes through a 3D scanner. The 3D reconstruction based on computer vision mainly reconstructs the original 3D information from images such as monocular single view, binocular multi-view, RGBD camera multi-view, etc. It is applied in many scenarios because of its low cost, strong sense of reality and high degree of automation. . Among them, the effect of 3D reconstruction based on single view is relatively general, because single view lacks depth and multi-view information; while 3D reconstruction based on multi-view images makes full use of multi-view shooting information, first calibrates the camera, and calculates the camera’s The relationship between the image coordinate system and the real world coordinate system, and then use multiple two-dimensional images to reconstruct three-dimensional information. The method of reconstructing 3D information based on images has gradually become the mainstream method.

For monocular reconstruction, the classic algorithm is SFM reconstruction, which is an offline algorithm for 3D reconstruction based on various collected unordered pictures. It first uses the SIFT algorithm to extract image features, then calculates the Euclidean distance of the corresponding descriptors in the two images, matches the SIFT feature points according to the distance, and calculates the basic matrix according to the epipolar geometry to form a trajectory. Afterwards, the selected image is used to initialize the whole BA process to obtain scene geometry information. Monocular online progressive reconstruction uses the image at the next moment to continuously fuse the previous 3D information. The main algorithm is the REMODE algorithm. Monocular direct reconstruction uses images at several moments to complete the 3D reconstruction of the same scene at one time, that is, deep fusion. It involves fewer images in the calculation and has higher real-time performance. For binocular reconstruction, the two corrected images obtained by the left and right cameras are mainly used to find the matching points of the left and right images, and then restore the three-dimensional information of the environment according to the principle of triangulation. The difficulty of binocular reconstruction lies in the matching of left and right camera images. Currently, the more popular algorithms are: SGBM algorithm and BM algorithm. For RGBD reconstruction, there are two different mapping methods depending on the map form. The first method is to estimate the pose of the camera first, convert the RGBD data into a point cloud, and then stitch them together, and finally get a map composed of discrete points. The resulting point cloud map. On this basis, you can also use triangular meshes (Mesh) and surface patches (Surfel) for mapping. another way. If you want to know the obstacles on the map and navigate on the map, you can also create an occupancy grid map through voxels. There are many mature algorithms for 3D reconstruction based on RGBD cameras, mainly Kinect Fusion, Dynamic Fusion, Elastic Fusion, Fusion 4D, Volumn Deform, etc.

When performing 3D reconstruction on the surface of ground objects, the above methods have produced very good reconstruction results, but when facing small targets such as traffic signs, billboards, fences, street lights, etc., the thickness of the target or scene is less than the limit of point cloud matching accuracy , it is easy to cause the front and back of such objects to be "staggered", resulting in being eliminated during the mesh correctness check, resulting in holes, missing textures, and distortions, or due to problems such as camera angles and scene occlusions, the resulting point cloud image A large area is missing and holes are formed, which seriously affects the fineness, rendering effect and aesthetics of the model, and it is impossible to obtain an accurate model of the target or scene. Common hole filling methods in 3D reconstruction include model completion based on point cloud and hole completion based on dense reconstruction mesh model. The point cloud model of the former is weak in expressing details, textures, etc., while the latter is mainly for a single The model is relatively effective, but the filling of multiple target holes and scene holes is not effective due to mutual interference between the holes. If the hole is filled as a plane, it cannot be well integrated with the spatial information of other surrounding models, resulting in unsatisfactory results.

Therefore, how to repair the holes that appear in the 3D reconstruction process, keep the human body, objects, and scenes in the real world as consistent as possible during the reconstruction process, and realize high-fidelity 3D reconstruction is a technical problem that needs to be solved urgently.

Contents of the invention

The technical task of the present invention is to provide a method and system for filling holes in 3D reconstruction, to solve how to repair the holes that appear in the 3D reconstruction process, and to keep the human body, objects, and scenes in the real world as good as possible during the reconstruction process. Consistently, the problem of achieving high-fidelity 3D reconstruction.

The technical task of the present invention is achieved in the following manner, a method for filling holes in three-dimensional reconstruction, the method is to perform multi-dimensional hole detection on the input image when filling for the first time, perform precise screening, smooth filtering, and introduce mean square Error, and then modify the target and the hole in the scene according to the distance threshold; on the basis of the initial repair, reversely detect the target in the original image at the hole position, match and optimize with the initial filling result, and calculate the original image at the hole position Texture information, and texture mapping on the basis of preliminary filling, to obtain optimized hole filling results, and improve the consistency between the target or scene in the real world and the 3D reconstruction model.

Preferably, the method is as follows:

Hole detection in the 3D model: Use the image and the 3D reconstruction model to calculate the local size invariance feature to roughly screen out the unmatched points; then perform a second fine screening, that is, smooth and denoise the image, introduce the mean square error, and pass the detection The distance threshold between the result and the true value identifies holes;

Judging the filling point and completing the hole filling that meets the conditions: merge the results of all hole filling into the original 3D model to complete the preliminary filling of the holes in the 3D reconstruction model;

Matching and optimization of the 3D reconstruction model: identify the image of the location of the hole in the original image, extract the object or scene in the image, match and optimize it with the initially filled 3D reconstruction model, and put the recognized target in the 3D reconstruction Problem mapping is carried out in the model to improve the accuracy of the 3D reconstruction model.

More preferably, the hole detection in the 3D model is specifically as follows:

Search image positions on all scale spaces, identify potential scale- and rotation-invariant interest points through Gaussian differential functions, delete unstable extreme points, locate key points and determine feature directions, and perform coarse screening of key points for matching; potential Interest points with scale and rotation invariance include corner points, edge points, bright points in dark regions, and dark points in bright regions;

Perform fine screening by cyclically calling the RANSAC algorithm;

Use the non-linear bilateral filtering method to smooth and denoise each image, introduce the mean square error, and detect the symmetrical surface distance between the reconstruction result and the true value. The smaller the distance, the better the reconstruction effect; at the same time, set the critical threshold. When it is below the threshold, it is considered as a hole;

Calculate the angle between the vector formed by the point to be detected and its neighbor points and set the maximum angle threshold:

When the angle between the vector formed by the point to be detected and its neighbor points exceeds the threshold, the point is marked as a boundary feature point, and the detected points are sorted to determine the closed hole;

For smaller holes, in order to prevent their borders from being misconnected, the trend judgment of the vector is introduced to connect the points with smaller deviation angles.

More preferably, the filling point is judged to complete the hole filling that satisfies the conditions, as follows:

Preprocess the hole boundaries, computing the length of each edge:

If it exceeds the set average point distance, add the point to the hole boundary;

Set the uniform boundary direction of the hole, judge the inner and outer boundaries of the hole, and remove the outer boundary contour;

The process of determining the filling point is repeated until no new filling point can be calculated, and the filling of the hole is completed.

More preferably, the matching and optimization of the three-dimensional reconstruction model are as follows:

Input the position of the hole in the original image, and identify the target existing within the area of the position;

Match the identified target with the preliminary filled 3D reconstruction model, calculate the distance between the true value and the preliminary filled 3D reconstruction model, and save the points within the threshold range;

The image of the obtained point set is texture-mapped to the 3D reconstruction model to obtain an optimized hole filling result.

A system for filling holes in three-dimensional reconstruction, the system comprising,

The hole detection unit is used to use the image and the 3D reconstruction model to calculate the local size invariance feature to roughly screen out the unmatched points; and then perform a second fine screening, that is, to smooth and denoise the image, introduce the mean square error, and pass the detection The distance threshold between the result and the true value identifies holes;

The judging and filling unit is used to merge the results of all hole filling into the original 3D model to complete the preliminary filling of holes in the 3D reconstruction model;

The matching and optimization unit is used to identify the image of the position of the hole in the original image, extract the object or scene in the image, match and optimize it with the initially filled 3D reconstruction model, and put the recognized target in the 3D reconstruction model Problem mapping is performed in the 3D reconstruction model to improve the accuracy of the model.

Preferably, the hole detection unit includes,

The coarse screening module is used to search for image positions in all scale spaces, identify potential interest points with scale and rotation invariance through Gaussian differential functions, delete unstable extreme points, locate key points and determine feature directions, and perform key Point coarse sieve matching; potential scale- and rotation-invariant interest points include corner points, edge points, bright spots in dark areas, and dark points in bright areas;

The fine screening module is used to carry out fine screening by calling the RANSAC algorithm in a loop;

The image processing module is used to smooth and denoise each image by using the nonlinear bilateral filtering method, introduce the mean square error, and detect the symmetrical surface distance between the reconstruction result and the true value. The smaller the distance, the better the reconstruction effect; Determine the critical threshold, when it is higher than the threshold, it is considered a hole;

The calculation module is used to calculate the angle between the vectors formed by the point to be detected and its neighbor points and set the maximum angle threshold:

When the angle between the vector formed by the point to be detected and its neighbor points exceeds the threshold, the point is marked as a boundary feature point, and the detected points are sorted to determine the closed hole;

For smaller holes, in order to prevent their borders from being misconnected, the trend judgment of the vector is introduced to connect the points with smaller deviation angles.

More preferably, the judging and filling unit includes,

The preprocessing module is used to preprocess the hole boundary and calculate the length of each side:

If it exceeds the set average point distance, add the point to the hole boundary;

The judging module is used to set the uniform boundary direction of the hole, judge the inner and outer boundaries of the hole, and remove the outer boundary contour;

The cycle module is used to cycle the process of determining the filling point until no new filling point can be calculated, and the filling of the hole is completed;

The matching and optimization unit includes,

A recognition module, configured to input the position of the hole in the original image, and identify the target existing within the area of the position;

The matching module is used to match the identified target with the preliminary filled 3D reconstruction model, calculate the distance between the true value and the preliminary filled 3D reconstruction model, and save the points within the threshold range;

The optimization module is used for texture mapping the obtained image of the point set to the 3D reconstruction model to obtain an optimized hole filling result.

An electronic device comprising: memory and at least one processor;

Wherein, a computer program is stored on the memory;

The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the hole filling method in 3D reconstruction as described above.

A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer program can be executed by a processor to implement the above method for filling holes in three-dimensional reconstruction.

The hole filling method and system in 3D reconstruction of the present invention have the following advantages:

(1) The present invention solves the problem of large-scale missing of the 3D model in the 3D image reconstruction due to the small target, camera angle, scene occlusion, etc., resulting in the formation of holes, which seriously affects the fineness, rendering effect and aesthetics of the model, and the target cannot be obtained. Or the problem of the accurate model of the scene is mainly to output a better 3D reconstruction model through two hole fillings. The first filling is to perform multi-dimensional hole detection on the input image, perform precise screening, smoothing filtering, and introduce mean square error. According to The distance threshold is used to repair the holes on the target and in the scene. On the basis of the first repair, reversely detect the target in the original image at the position of the hole, perform matching optimization with the preliminary filling result, and calculate the original position of the hole. Texture information, perform texture mapping on the initially filled model, obtain optimized hole filling results, and improve the consistency between the target or scene in the real world and the 3D reconstruction model;

(2) The present invention is mainly aimed at the holes that appear in the three-dimensional reconstruction based on images (monocular, binocular, RGBD), and aims to solve the problem of filling holes in models such as objects and scenes in the process of three-dimensional reconstruction. The present invention fills the holes twice Output a better 3D reconstruction model to improve the fineness, presentation and aesthetics of the 3D model;

(3) The present invention performs multi-dimensional hole detection according to the input image, performs precise screening, smoothing filtering, introduces mean square error, repairs holes on the target and in the scene, and reversely detects the original image texture on the hole position and then Texture mapping on the 3D model can better solve the large-area defects and holes generated in the process of actual 3D scene reconstruction and 3D object reconstruction, thereby improving the fineness, rendering effect and aesthetics of the 3D model, and improving the object in the real world. Or the consistency between the scene and the 3D reconstruction model, and output a better 3D reconstruction model.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings.

Accompanying drawing 1 is the flowchart of hole filling method in 3D reconstruction.

Detailed ways

The hole filling method and system in 3D reconstruction of the present invention will be described in detail below with reference to the drawings and specific embodiments of the specification.

Example 1:

As shown in Figure 1, this embodiment provides a method for filling holes in 3D reconstruction. The method is to perform multi-dimensional hole detection on the input image during the initial filling, perform precise screening, smoothing filtering, and introduce averaging square error, and then modify the target and the hole in the scene according to the distance threshold; on the basis of the initial repair, reversely detect the target in the original image at the hole position, and perform matching optimization with the initial filling result to calculate the original image at the hole position Texture information, and texture mapping on the basis of preliminary filling, to obtain optimized hole filling results, and improve the consistency between the target or scene in the real world and the 3D reconstruction model; the details are as follows:

S1. Hole detection in the 3D model: use the image and the 3D reconstruction model to calculate the local size invariance features to roughly screen out the unmatched points; then perform a second fine screening, that is, smooth and denoise the image, introduce the mean square error, and Determine the hole by the distance threshold between the detection result and the true value;

S2. Judging the filling point, and completing the hole filling that satisfies the conditions: merge the results of all hole filling into the original 3D model, and complete the preliminary filling of the holes in the 3D reconstruction model;

S3. Matching and optimization of the 3D reconstruction model: identify the image of the location of the hole in the original image, extract the object or scene in the image, match and optimize it with the initially filled 3D reconstruction model, and place the recognized target in the Problem mapping is carried out in the 3D reconstruction model to improve the accuracy of the 3D reconstruction model.

The hole detection in the three-dimensional model in step S1 of this embodiment is specifically as follows:

S101. Search for image positions in all scale spaces, identify potential scale- and rotation-invariant interest points through Gaussian differential functions, delete unstable extreme points, locate key points and determine feature directions, and perform rough screening of key points for matching ; Potential scale- and rotation-invariant interest points include corner points, edge points, bright points in dark regions, and dark points in bright regions;

S102, perform fine screening by cyclically calling the RANSAC algorithm;

S103. Smooth and denoise each image by using a nonlinear bilateral filter method, introduce a mean square error, and detect the symmetrical surface distance between the reconstruction result and the true value. The smaller the distance, the better the reconstruction effect; at the same time, set a critical threshold, When it is higher than the threshold, it is considered a hole;

S104. Calculate the angle between the vectors formed by the point to be detected and its neighbor points and set the maximum angle threshold:

When the angle between the vector formed by the point to be detected and its neighbor points exceeds the threshold, the point is marked as a boundary feature point, and the detected points are sorted to determine the closed hole;

For smaller holes, in order to prevent their borders from being misconnected, the trend judgment of the vector is introduced to connect the points with smaller deviation angles.

The filling point judgment in the step S2 of this embodiment, and the hole filling that satisfies the conditions are completed are specifically as follows:

S201. Preprocess the boundary of the hole, and calculate the length of each side:

If it exceeds the set average point distance, add the point to the hole boundary;

S202. Set the uniform boundary direction of the hole, judge the inner and outer boundaries of the hole, and remove the outer boundary contour;

S203 , the process of determining the filling point is repeated until no new filling point can be calculated, and the filling of the hole is completed.

The matching and optimization of the three-dimensional reconstruction model in step S3 of this embodiment are as follows:

S301. Input the position of the hole in the original image, and identify the target within the area of the position;

S302. Match the identified target with the preliminary filled 3D reconstruction model, calculate the distance between the true value and the preliminary filled 3D reconstruction model, and save the points within the threshold range;

S303. Perform texture mapping on the obtained image of the point set to the three-dimensional reconstruction model to obtain an optimized hole filling result.

Example 2:

This embodiment provides a hole filling system in 3D reconstruction, the system includes:

The hole detection unit is used to use the image and the 3D reconstruction model to calculate the local size invariance feature to roughly screen out the unmatched points; and then perform a second fine screening, that is, to smooth and denoise the image, introduce the mean square error, and pass the detection The distance threshold between the result and the true value identifies holes;

The judging and filling unit is used to merge the results of all hole filling into the original 3D model to complete the preliminary filling of holes in the 3D reconstruction model;

The matching and optimization unit is used to identify the image of the position of the hole in the original image, extract the object or scene in the image, match and optimize it with the initially filled 3D reconstruction model, and put the recognized target in the 3D reconstruction model Problem mapping is performed in the 3D reconstruction model to improve the accuracy of the model.

The hole detection unit in this embodiment includes,

The coarse screening module is used to search for image positions in all scale spaces, identify potential interest points with scale and rotation invariance through Gaussian differential functions, delete unstable extreme points, locate key points and determine feature directions, and perform key Point coarse sieve matching; potential scale- and rotation-invariant interest points include corner points, edge points, bright spots in dark areas, and dark points in bright areas;

The fine screening module is used to carry out fine screening by calling the RANSAC algorithm in a loop;

The image processing module is used to smooth and denoise each image by using the nonlinear bilateral filtering method, introduce the mean square error, and detect the symmetrical surface distance between the reconstruction result and the true value. The smaller the distance, the better the reconstruction effect; Determine the critical threshold, when it is higher than the threshold, it is considered a hole;

The calculation module is used to calculate the angle between the vectors formed by the point to be detected and its neighbor points and set the maximum angle threshold:

When the angle between the vector formed by the point to be detected and its neighbor points exceeds the threshold, the point is marked as a boundary feature point, and the detected points are sorted to determine the closed hole;

For smaller holes, in order to prevent their borders from being misconnected, the trend judgment of the vector is introduced to connect the points with smaller deviation angles.

The judging and filling unit in this embodiment includes:

The preprocessing module is used to preprocess the hole boundary and calculate the length of each side:

If it exceeds the set average point distance, add the point to the hole boundary;

The judging module is used to set the uniform boundary direction of the hole, judge the inner and outer boundaries of the hole, and remove the outer boundary contour;

The cycle module is used to cycle the process of determining the filling point until no new filling point can be calculated, and the filling of the hole is completed.

The matching and optimization unit in this embodiment includes:

A recognition module, configured to input the position of the hole in the original image, and identify the target existing within the area of the position;

The matching module is used to match the identified target with the preliminary filled 3D reconstruction model, calculate the distance between the true value and the preliminary filled 3D reconstruction model, and save the points within the threshold range;

The optimization module is used for texture mapping the obtained image of the point set to the 3D reconstruction model to obtain an optimized hole filling result.

Example 3:

This embodiment also provides an electronic device, including: a memory and a processor;

Wherein, the memory stores computer-executable instructions;

The processor executes the computer-executable instructions stored in the memory, so that the processor executes the hole filling method in 3D reconstruction in any embodiment of the present invention.

The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs) or other programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory can be used to store computer programs and/or modules, and the processor implements various functions of the electronic device by running or executing the computer programs and/or modules stored in the memory and calling data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function, etc.; the data storage area may store data created according to the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, internal memory, plug-in hard disks, memory stick cards (SMC), secure digital (SD) cards, flash memory cards, at least A disk storage device, flash memory device, or other volatile solid-state storage device.

Example 4:

This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the method for filling holes in three-dimensional reconstruction according to any embodiment of the present invention. Specifically, a system or device equipped with a storage medium may be provided, on which a software program code for realizing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU of the system or device) ) to read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, so the program code and the storage medium storing the program code constitute a part of the present invention.

Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), Tape, non-volatile memory card, and ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

In addition, it should be clear that not only by executing the program code read by the computer, but also by making the operating system on the computer complete part or all of the actual operations through instructions based on the program code, so as to realize the function of any one of the embodiments.

In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or written into the memory provided in the expansion unit connected to the computer, and then based on the program code The instruction causes the CPU installed on the expansion board or the expansion unit to perform some or all of the actual operations, so as to realize the functions of any one of the above-mentioned embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (10)

1. The method is characterized in that in the process of primary filling, the method carries out multi-dimensional hole detection on an input image, carries out accurate screening and smooth filtering, introduces mean square error, and modifies the target and the holes in a scene according to a distance threshold; and on the basis of primary patching, reversely detecting a target in the original image at the hole position, carrying out matching optimization with the primary patching result, calculating texture information of the original image at the hole position, carrying out texture mapping on the basis of primary patching, obtaining an optimized hole patching result, and improving the consistency of the target or scene in the real world and the three-dimensional reconstruction model.

2. The method for filling holes in three-dimensional reconstruction according to claim 1, characterized in that it comprises the following steps:

hole detection in three-dimensional model: calculating local size invariance features by using the image and the three-dimensional reconstruction model to coarsely screen out unmatched points; carrying out secondary fine screening, namely carrying out smooth denoising on the image, introducing a mean square error, and determining holes through a distance threshold between a detection result and a true value;

and (3) judging filling points, and finishing hole filling meeting the conditions: combining all hole filling results into the original three-dimensional model to finish preliminary filling of the holes of the three-dimensional reconstruction model;

matching and optimizing a three-dimensional reconstruction model: identifying the image of the position of the hole in the original image, extracting an object or scene in the image, matching and optimizing the object or scene with the preliminary filled three-dimensional reconstruction model, and performing problem mapping on the identified target in the three-dimensional reconstruction model to improve the accuracy of the three-dimensional reconstruction model.

3. The method for filling holes in three-dimensional reconstruction according to claim 2, wherein the hole detection in the three-dimensional model is specifically as follows:

searching image positions on all scale spaces, identifying potential interest points with scale and rotation invariance through Gaussian differential functions, deleting unstable extreme points, positioning key points and determining characteristic directions, and carrying out key point coarse screening matching; potential interest points with scale and rotation invariance include corner points, edge points, bright points of dark areas and dark points of bright areas;

fine screening is carried out by circularly calling a RANSAC algorithm;

smoothing and denoising each image by using a nonlinear bilateral filtering method, introducing a mean square error, detecting the symmetrical surface distance between a reconstruction result and a true value, and indicating that the reconstruction effect is better as the distance is smaller; setting a critical threshold value at the same time, and considering the hole as the hole when the critical threshold value is higher than the threshold value;

calculating an included angle between vectors formed by the detection point and the neighborhood point thereof, and setting a maximum angle threshold value:

and when the included angle between the vector formed by the detection point and the neighborhood point exceeds a threshold value, marking the point as a boundary characteristic point, and sequencing the detection points to determine a closed hole.

4. The hole filling method in three-dimensional reconstruction according to claim 2, wherein the hole filling for completing the meeting of the condition by filling point judgment is specifically as follows:

preprocessing the hole boundary, and calculating the length of each edge:

if the set average point distance is exceeded, adding the point into the hole boundary;

setting uniform boundary directions of the holes, judging the inner and outer boundaries of the holes, and removing the outline of the outer boundary;

and (5) circulating the process of judging the filling points until no new filling points are calculated, and finishing filling the holes.

5. The hole filling method in three-dimensional reconstruction according to any one of claims 2-4, wherein the matching and optimization of the three-dimensional reconstruction model is specifically as follows:

inputting the position of a hole in an original image, and identifying a target existing in the position area;

matching the identified target with the preliminary filled three-dimensional reconstruction model, calculating the distance between the true value and the preliminary filled three-dimensional reconstruction model, and storing points in a threshold range;

and performing texture mapping on the obtained image of the point set to a three-dimensional reconstruction model to obtain an optimized hole filling result.

6. A hole filling system in three-dimensional reconstruction is characterized in that the system comprises,

the hole detection unit is used for calculating local size invariance characteristics by utilizing the image and the three-dimensional reconstruction model so as to coarsely screen out unmatched points; carrying out secondary fine screening, namely carrying out smooth denoising on the image, introducing a mean square error, and determining holes through a distance threshold between a detection result and a true value;

the judging and filling unit is used for combining the filling results of all holes into the original three-dimensional model to finish the preliminary filling of the holes of the three-dimensional reconstruction model;

the matching and optimizing unit is used for identifying the image of the position of the hole in the original image, extracting an object or scene in the image, matching and optimizing the object or scene with the preliminary filled three-dimensional reconstruction model, and mapping the identified target in the three-dimensional reconstruction model to improve the accuracy of the three-dimensional reconstruction model.

7. The three-dimensional reconstruction mesoporous filler system of claim 6, wherein said hole detection unit comprises,

the coarse screening module is used for searching image positions on all scale spaces, identifying potential interest points with scale and rotation invariance through Gaussian differential functions, deleting unstable extreme points, positioning key points and determining characteristic directions, and performing key point coarse screening matching; potential interest points with scale and rotation invariance include corner points, edge points, bright points of dark areas and dark points of bright areas;

the fine screening module is used for carrying out fine screening by circularly calling the RANSAC algorithm;

the image processing module is used for carrying out smooth denoising on each image by utilizing a nonlinear bilateral filtering method, introducing a mean square error, detecting the symmetrical surface distance between a reconstruction result and a true value, and indicating that the reconstruction effect is better as the distance is smaller; setting a critical threshold value at the same time, and considering the hole as the hole when the critical threshold value is higher than the threshold value;

the calculation module is used for calculating the included angle between the vector formed by the detection point to be detected and the neighborhood point thereof and setting the maximum angle threshold value:

and when the included angle between the vector formed by the detection point and the neighborhood point exceeds a threshold value, marking the point as a boundary characteristic point, and sequencing the detection points to determine a closed hole.

8. The three-dimensional reconstructed hole filling system according to claim 6 or 7, wherein the judging and filling unit comprises,

the preprocessing module is used for preprocessing the hole boundaries and calculating the length of each edge:

if the set average point distance is exceeded, adding the point into the hole boundary;

the judging module is used for setting uniform boundary directions of the holes, judging the inner and outer boundaries of the holes and removing the outline of the outer boundary;

the circulation module is used for circulating the process of judging the filling points until new filling points cannot be calculated, and filling holes is completed;

the matching and optimizing unit comprises a matching and optimizing unit,

the identification module is used for inputting the position of the hole in the original image and identifying the target existing in the position area range;

the matching module is used for matching the identified target with the preliminary filled three-dimensional reconstruction model, calculating the distance between the true value and the preliminary filled three-dimensional reconstruction model, and storing the points in the threshold range;

and the optimizing module is used for performing texture mapping on the obtained image of the point set to a three-dimensional reconstruction model to obtain an optimized hole filling result.

9. An electronic device, comprising: a memory and at least one processor;

wherein the memory has a computer program stored thereon;

the at least one processor executing the computer program stored by the memory, causes the at least one processor to perform the hole filling method in three-dimensional reconstruction according to any one of claims 1 to 5.

10. A computer readable storage medium having stored therein a computer program executable by a processor to implement the hole filling method in three-dimensional reconstruction according to any one of claims 1 to 5.