CN108734728A - A kind of extraterrestrial target three-dimensional reconstruction method based on high-resolution sequence image - Google Patents

Abstract

The invention relates to a method for three-dimensional reconstruction of spatial objects based on high-resolution sequence images, using SIFT feature descriptors to sequentially extract accurate and robust point features of sequence images, and quickly and accurately match adjacent images through hash cascade; joint corresponding cameras The internal parameter matrix, using the motion recovery structure, obtains the relative pose between the image sequences, and obtains the sparse point cloud model of the space object; by matching the sequence images, diffusing interpolation, and iterative filtering, the dense 3D point cloud model of the space object is obtained; using The acquired dense point cloud model is used for surface reconstruction to obtain a dense space target grid model; the acquired precise and dense grid model is used for texture mapping, and finally a visualized 3D model of the space target is obtained. The invention can realize fast and accurate image matching, repair and reconstruct surface highlight holes, and obtain a fine and visualized three-dimensional model of a spatial object.

Description

A Method of 3D Reconstruction of Space Objects Based on High Resolution Sequence Images

technical field

The invention belongs to the technical field of three-dimensional reconstruction of computer vision, and relates to a three-dimensional reconstruction method of a space object based on high-resolution sequence images.

Background technique

With the maturity and development of aerospace science and technology and the military value of outer space, countries around the world pay special attention to the monitoring and situational awareness of space-based space targets. While responding to the threat of foreign space military forces, our country needs to strive to improve its own space military capabilities and gradually meet its own needs in future space military confrontations. New space target monitoring systems and technologies, especially for the acquisition and estimation of 3D models, target attitude and behavior information of cooperative targets and non-cooperative targets, are becoming more and more important. Although my country has made great progress in the fields of aerospace and computer science and some technologies are world-leading, it is limited by the fact that some technical problems have not yet been broken through, and my country has not yet mastered the ability to acquire 3D models of key space targets. How to make full use of the advantages of my country's aerospace technology, combined with computer vision, computer graphics and other professional disciplines, to quickly master the ability to acquire 3D models of key space targets, especially non-cooperative targets, is particularly urgent. The 3D reconstruction technology of space targets based on high-resolution image sequences is the application of 3D reconstruction technology, focusing on breaking through the 3D reconstruction of targets based on high-resolution image sequences and the difficulty of obtaining 3D models.

Image-based 3D reconstruction is a typical computer vision problem, and its core goal can be described as follows: "Given an image set of an object or scene, under the basic assumptions of known materials, viewpoints, and lighting conditions, estimate The most probable three-dimensional shape that can reasonably explain these images". This definition emphasizes the difficulty of this task, that is, the assumed material, viewpoint, and lighting conditions are all known. If these are unknown, then the general problem is that pathological combinations of materials, viewpoints, lighting conditions will produce the exact same image. Therefore, without better assumptions, there is no method that can accurately recover 3D structures relying solely on images. The initial application of 3D reconstruction is mainly structured image sets, and the order of images is affected, such as video sequences. Some MVS (Multi-View Stereo) applications also follow this rule, such as Google Street View and Microsoft Street View, but there are already MVSs that can process unordered image sets on different occasions and hardware, such as 3D maps based on aerial images. Fast and high-quality feature extraction and description have enabled SFM (Structure from Motion) to process unstructured datasets, and high-quality descriptions enable buildings to obtain longer and higher trajectories from images under different poses and illuminations.

At present, in the field of 3D reconstruction based on high-resolution sequence images, there is a lack of targeted and systematic research on space environment objects as the research object in China. The research involved at this stage is in the early stage of discussion, program analysis, and technical demonstration. , or some algorithm tests in a limited simulation experiment environment, a relatively complete theoretical framework has not yet been formed. In addition, there is no actual measured image as a basis in China at present, and it will face some difficulties that may go beyond the theoretical system of 3D reconstruction. In view of the characteristics of space targets in this project, it is necessary to first solve the non-Lambertian structure of 3D reconstruction (mainly In view of the impact of data damage caused by high light and non-uniform illumination of key space targets, these data damages may bring unpredictable challenges to the robustness of the algorithm. Secondly, satellite targets have thin and linear shapes such as sailboards and antennas. These are major technical problems in the field of 3D reconstruction.

Contents of the invention

technical problem to be solved

In order to avoid the deficiencies of the prior art, the present invention proposes a method for three-dimensional reconstruction of a space target based on high-resolution sequence images, which can realize three-dimensional reconstruction of the space target and obtain a three-dimensional model of the space target.

Technical solutions

A method for three-dimensional reconstruction of space targets based on high-resolution sequence images, characterized in that the steps are as follows:

Step 1: Use the SIFT feature descriptor to sequentially extract a set of accurate and robust point features of a set of spatial target high-resolution sequence images I={I ₁ ,I ₂ ,…,I _n }, and perform hash concatenation on each image The set N neighboring image pairs are quickly and accurately matched, and the matching set of N neighboring image pairs of the image I _i is obtained as M _i ={ _mi,iN ,m _i,i-N+1 ,… _mi,j … ,m _i,i+N-1 ,m _i,i+N };

The characteristics of the precise and robust point features are scale invariance and rotation invariance;

Step 2: Use RANSAC estimation to perform stereo matching on the feature-matched image pairs. In each RANSAC cycle, select the image pair with the most feature matching points from the adjacent image pair matching set M _i as the initial matching pair, and combine the image pairs The internal parameter matrix K of the camera is estimated by the eight-point algorithm to obtain the basic matrix F; the normalized singular value decomposition is performed on the basic matrix F to obtain the relative rotation matrix P and translation matrix t of the image pair; the incremental motion recovery structure is used to obtain The space target captures the attitude of the camera to obtain a structured image set and a sparse point cloud model of the space target;

Step 3: For the structured image set and the sparse point cloud model of the spatial target, Harris or DoG corner point fine matching is performed on the sequence image I= _{ I ₁ ,I ₂ ,…,In }, and the sparse feature point cloud model Perform bilinear diffusion interpolation, and use the principle of photometric consistency to constrain iteratively to filter out error points that are scattered outside and inside the actual surface. The diffusion interpolation and filtering process can be iterated 3 times to obtain a 3D point cloud model with dense spatial objects;

Step 4. Perform surface reconstruction based on the spatial floating-scale implicit function for the obtained 3D point cloud model with dense spatial objects: use the zero isosurface {x|implictF(x) of the implicit function defined by the generated regularized octree voxel ＝0} is used as the spatial object to reconstruct the model surface, and the initial dense spatial object mesh model is obtained, and the mesh is in the form of triangular facets;

Step 5: Perform redundancy removal on the initial dense spatial object grid model, set the geometric confidence threshold to remove the envelope from the edge to the inside, and obtain a dense spatial object grid model;

Step 6: Perform large-scale seamless mosaic texture mapping on the dense spatial object grid model, and use the paired Markov random field energy formula E(l) to specify a view l _i as the texture corresponding to the grid surface F _i , The corresponding l _j is used as the texture corresponding to the mesh patch F _j , where F _i and F _j represent the triangular patch of the spatial object mesh model. This process finally obtains a finely visualized 3D model of the spatial object. The energy formula is:

E _date represents the data item constraint for selecting the best view of the corresponding face F _i ∈ Faces, and E _smooth represents the smooth item constraint at the boundary seam of two mesh faces (F _i , F _j ) ∈ Edges;

Step 7: Eliminate the occlusion of the adjacent view of the gradient magnitude through photometric consistency detection, and further use the Euclidean distance weight attenuation to adjust the color of the adjacent surface, reduce the visibility of the spatial object model seam, and obtain a finely visualized 3D model of the spatial object.

In the step 2, the incremental motion recovery structure is used to obtain the acquisition attitude of the space target capture camera, and the method of obtaining the structured image set and the space target sparse point cloud model is as follows: uniformly and randomly extract 8 points from the feature points; use this Find the minimum configuration solution of the basic matrix F for 8 sets of corresponding points; verify the basic matrix F with each set of corresponding points that have not been extracted, and when the distance meets the threshold requirement, it is determined that the correspondence is consistent; after stereo matching for each image Put it into the trajectory to obtain a connected set of matching feature points between multi-viewpoint images; add all image cameras to the trajectory by adding a camera in turn, and finally optimize the result through cluster adjustment to obtain a space target capture camera Acquire poses, structured image sets and sparse point cloud models of spatial objects.

The method of step 7 to obtain a three-dimensional model of a finely visualized space object is as follows: construct a pair of Markov random field energy formulas to specify a view as the texture corresponding to the grid surface, and the energy formula E(l) includes data items E _date and Smoothing item E _smooth ; eliminate the adjacent view occlusion of the gradient magnitude through photometric consistency detection, and further adjust the surface color; set the angle, distance, and color difference as the cost function of the cost value w Select the view set with less cost value for image stitching.

The method for surface color adjustment: using Potts model for image segmentation, querying color discontinuity along the edges of adjacent seams, using Euclidean distance weight attenuation to reduce color differences between patches, and finally using Poisson editing for local optimization, Reduced space object model seam visibility.

Beneficial effect

A three-dimensional reconstruction method for spatial objects based on high-resolution sequence images proposed by the present invention uses SIFT feature descriptors to sequentially extract accurate and robust point features of sequence images, and quickly and accurately matches adjacent images through hash cascade; joint correspondence The internal parameter matrix of the camera, using the motion recovery structure, obtains the relative pose between the image sequences, and obtains the sparse point cloud model of the space object; by matching the sequence images, diffusing interpolation, and iterative filtering, the dense 3D point cloud model of the space object is obtained; The obtained dense point cloud model is used for surface reconstruction to obtain a dense spatial object grid model; the obtained precise dense mesh model is used for texture mapping, and finally a visualized 3D model of the spatial object is obtained.

In the present invention, by performing local sensitive hash encoding on high-dimensional features, using multi-table hash of short codes to perform rough search, and mapping the feedback candidates to high-dimensional Hamming space, the established Hamming distance hash table can realize fast, Accurate image matching; by using the implicit function method to reconstruct the surface of the dense point cloud model, a dense mesh model can be established to fill the highlight holes caused by the spatial object characteristic data during the reconstruction process; by performing large-scale texture mapping, according to the edge The weight of the length is attenuated for color adjustment, eliminating the visibility of seams and enhancing model visualization.

Description of drawings

Fig. 1 overall design of the specific embodiment of the system of the present invention

Fig. 2 The specific embodiment of the present invention obtains the relative pose between the image sequences, and obtains the sparse point cloud model of the space object

Fig. 3 specific embodiment of the present invention octree generation strategy

Fig. 4 Example of the present invention obtains the texture map result

Detailed ways

Now in conjunction with embodiment, accompanying drawing, the present invention will be further described:

The technical scheme that the embodiment of the present invention adopts is:

By inputting a set of spatial target high-resolution sequence image sets I= _{ I ₁ ,I ₂ ,…,In }, the SIFT feature descriptor is used to extract the precise and robust point features of each image, and its features are scale invariance, rotation invariance, and quickly and accurately match the N neighboring image pairs set for each image through hash cascading, where the matching set of N neighboring image pairs of image I _i is M _i ={m _i,iN ,m _i,i-N+1 ,...m _i,j ...,m _i,i+N-1 ,m _i,i+N };

For the pair of matching points X ₁ and X ₂ between two adjacent matching images, they must satisfy the epipolar geometric constraint X ₁ ^T FX ₂ =0, where F is the fundamental matrix, from the matching set M _i of adjacent image pairs The image pair with the most feature matching points is selected as the initial matching pair, and the camera intrinsic parameter matrix K of the image is used to estimate the fundamental matrix F with 8 correspondences, that is, the eight-point algorithm; normalized singular value decomposition is performed on the fundamental matrix F to obtain The relative rotation matrix P and translation matrix t of the image pair; use incremental motion to restore the structure, that is, add an image each time until the acquisition attitude of the space target capture camera is obtained, and a structured image set and a space target sparse point cloud model are obtained;

Using the obtained structured image set and the sparse point cloud model of the space target, by performing Harris or DoG (Difference-of-Gaussian) corner point fine matching on the sequence image I= _{ I ₁ ,I ₂ ,…,In }, After performing bilinear diffusion interpolation on the sparse feature point cloud model, and using the principle of visual consistency to constrain iteratively to filter out the wrong points scattered outside and inside the actual surface, the process of diffusion interpolation and filtering can be iterated 3 times to obtain a dense spatial object. 3D point cloud model;

Use the acquired 3D point cloud model with dense spatial objects to perform surface reconstruction based on spatial floating-scale implicit functions, and use the zero isosurface {x|implictF(x)=0} of the implicit function defined by regularized octree voxels to generate , as the surface of the spatial object reconstruction model, where x is the point position to be reconstructed, impictF(x) is the defined implicit function, and the zero isosurface extraction can use the moving cube algorithm. This process finally obtains the initial dense spatial object network Lattice model, the mesh is in the form of triangular facets;

Use the acquired initial dense spatial object grid model to remove redundancy, and set the geometric confidence threshold to remove the envelope from the edge to the inside to obtain a dense spatial object grid model;

Use the obtained dense spatial target grid model to perform large-scale seamless mosaic texture maps, and use the paired Markov random field energy formula E(l) to designate a view l _i as the texture corresponding to the mesh patch F _i , and the corresponding l _j is used as the texture corresponding to the mesh patch F _j , where F _i and F _j represent the triangular patch of the space object mesh model. This process finally obtains a finely visualized 3D model of the space object. The energy formula is:

E _date represents the data item constraint for selecting the best view of the corresponding face F _i ∈ Faces, and E _smooth represents the smooth item constraint at the boundary seam of two mesh faces (F _i , F _j ) ∈ Edges.

Finally, the adjacent view occlusion of the gradient magnitude is eliminated by photometric consistency detection, and the Euclidean distance weight attenuation is further used to adjust the color of the adjacent surface to reduce the visibility of the spatial object model joints and obtain a finely visualized 3D model of the spatial object.

The SIFT feature of the sequence image, the specific method is as follows: firstly calculate the image difference of adjacent scales for the image, obtain a series of images and find the extreme points in the image space; then filter the extreme points, and traverse the Gaussian difference image successively Comparing the pixel points of its eight neighbors and the nine neighboring pixels of the upper and lower layers of the image, calculate the extreme points; find out the stable feature points, and filter these extreme points to remove the insignificant points. Stable point; select the Gaussian smooth image closest to the scale according to the scale of the key point; select a neighborhood around the key point, and perform Gaussian weighting on the gradient of the midpoint, and generate a descriptor for each sub-region;

The specific method of using the hash cascade to perform fast and accurate matching is as follows: using the local sensitive hash strategy, using the inner product similarity to judge the hash equation of the hash cluster, and selecting a random vector of a hyperplane r from the D-dimensional normal distribution , convert the 128-dimensional feature descriptor into a binary code; use a multi-table hash query with a short code to perform a rough search, remap the returned candidates to a higher-dimensional Hamming space and calculate the Hamming distance, and establish the Hamming The hash table with clear distance is used as the key reference for accurate query, and finally the cascaded hash strategy is used for accurate and fast matching between images;

The method of calculating the fundamental matrix using the eight-point algorithm is as follows: the data set is the corresponding relationship of the feature matching output, and the constraint is the epipolar geometry. The eight-point algorithm can be used to estimate the basic matrix F with 8 corresponding relationships; The matching points X ₁ and X ₂ between images must satisfy the epipolar geometric constraint X ₁ ^T FX ₂ ＝0, this equation is a linear homogeneous equation with 9 unknown parameters about F, because F differs by a constant factor It is unique in meaning, so a non-zero parameter can be normalized into 8 unknown parameters, so if 8 pairs of matching points can be obtained, the fundamental matrix F can be determined linearly; since the fundamental matrix contains the inside and outside of the two images Parameter information, therefore, the joint internal camera parameter K can be obtained from the fundamental matrix to obtain the essential matrix E=K ^T FK, and then the singular value decomposition of the essential matrix can be used to obtain the external parameters of the second image.

Described utilize RANSAC to carry out stereo matching, specific method is as follows: Uniformly randomly extract 8 points from feature point; Utilize these 8 groups of corresponding points to seek the minimum configuration solution of fundamental matrix F; Verify with each group of corresponding points not drawn F, if the distance is small enough, then the correspondence is consistent; otherwise, return;

The relative attitude between the acquired image sequences and the sparse point cloud model of the spatial object are obtained, the specific method is as follows: the next camera is incrementally added, the three-dimensional point in space is obtained by the triangle method, and for a given camera parameter set {P _j } and trajectory set M ^j refers to the three-dimensional coordinates of the trajectory, Refers to the camera whose two-dimensional image coordinates are projected on i _th , and the cluster adjustment requires the minimum nonlinear error E(P,M) to be minimum:

The detailed method of obtaining the dense three-dimensional point cloud model of space objects is as follows: detect feature points by Harris and DoG (Difference-of-Gaussian), and match these feature points between multiple images to obtain a series of sparse patches ; In each image, take Cell(i,j) of β ₂ × β ₂ pixels, which represents a rectangular grid containing pixels, and return η reflecting the local maximum response value of corners and points, through multiple The feature points detected in the two images are used to reconstruct a sparse patch set, which is stored in Cell(i,j) of the overlay image. Assuming image I _i , the corresponding camera optical center is marked as O, for any feature point f in I _i , search for a candidate set with the same type (Harris or DoG) feature point f′ in the adjacent matching image, in the corresponding Within 2 pixels of the epipolar line, use the point pair (f, f′) to triangulate the 3D points in the space;

Given these initial matches, the next two steps are repeated 3 times; diffuse the initial matches to adjacent pixels, resulting in denser patches; in this step, we recursively add new neighborhoods to existing faces In the patch, until the surface that can be observed in the scene is covered, if the two patches p and p' are stored in the continuous Cell(i,j) and Cell(i',j') of the same image I, and its method planes are close, we consider p and p' to be neighbors;

The visual constraints are used to remove those incorrect matches before and after the observed surface. The first step of filtering focuses on eliminating patches that are staggered on the actual surface, that is, considering a patch p ₀ and the set of patches U that it occludes, when satisfied We eliminate p ₀ ; then the filter focuses on the outliers that are scattered inside the actual surface, and uses the depth map of the corresponding image to simply calculate S(p ₀ ) and T(p ₀ ) of each patch p ₀ , if| T(p ₀ )|<γ will filter out p ₀ , S(p ₀ ) means what should be visible, T(p ₀ ) means what is actually visible, γ means the threshold, Represents the mapped normalized mutual light mean;

The specific method of obtaining the initially dense spatial target grid model is as follows: all input sample points are inserted into an octree data structure according to their scale values, and the generated octree has a hierarchical structure, which is given A sampling set, the vertex of the leaf node of the octree is used as the sampling point of the implicit function implictF(x), where x is the point position to be reconstructed, that is, the octree is initialized as an empty octree without any nodes, the first A sample point i is inserted into the newly created root node. The side length of the root node is s _i and the center of the sample point position P _i is; each sample point i has a scale value s _i , and the support domain radius of the sample is 3s _i . make:

Among them, l is the level of the octree where the sample point will be inserted, and S _l is the side length of the node at the lth layer of the octree.

Then estimate the value of the implicit function at these sample points. In order to distinguish these points from the input sample points, the nodes of the octree are divided into two categories: internal nodes and leaf nodes. Internal nodes refer to those with eight child nodes. Node, a leaf node refers to a node without child nodes; in actual situations, the generated octree generally has mixed nodes, that is, nodes with a number of child nodes greater than 0 and less than 8; by assigning child nodes to the mixed node, its child nodes The octree is regularized to 8, the mixed nodes in the octree are eliminated through regularization, new leaf nodes are added, and these points are called voxels; in order to calculate the implicit function value at x, a Efficient search method, only select samples in the octree that affect the value of the implicit function at x; recursively traverse the octree to determine whether each node may contain sample points that affect the value of the implicit function at x; from the above formula It can be seen that the maximum scale value of the sample points contained in node N does not exceed 2S _N , where S _N is the side length of N;

If a node satisfies the relational expression with x, the node can be directly skipped without entering its child nodes to search; otherwise, it is judged whether the distance between all sample points i and x in the node is less than 3s _i . Finally, use all searched sample points that have an influence on x to calculate the value of the implicit function.

The octree determines the position of the voxel, and after calculating the implicit function value at the voxel, it is no longer necessary to input the sample point at this time, and directly extract the zero equivalent value of the implicit function defined by the octree voxel The surface is the final reconstruction surface. The common algorithm for extracting isosurfaces in standard voxels (all voxels have the same size) is the moving cube algorithm. The isosurface of implicit functions is extracted from the octree as the reconstruction surface. :

{x|implictF(x)＝0}

The specific method of obtaining a precise and dense spatial object grid model is as follows: by setting the geometric confidence level of the obtained initial dense spatial object grid model, performing redundancy removal from the edge of the model inwards, and removing large-area surface packets; network;

Use the obtained dense spatial object grid model to carry out large-scale seamless stitching of texture maps, and use the paired Markov random field energy formula E(l) to designate a view l _i as the corresponding mesh patch F _i The corresponding l _j is the texture corresponding to the mesh patch F _j , where F _i and F _j represent the triangle surface of the spatial object mesh model. This process finally obtains a finely visualized 3D model of the spatial object. The energy formula is:

E _date represents the data item constraint for selecting the best view of the corresponding face F _i ∈ Faces, and E _smooth represents the smooth item constraint at the boundary seam of two mesh faces (F _i , F _j ) ∈ Edges.

After the label is obtained by minimizing the equation, the color of the patch is adjusted by first ensuring that each mesh vertex belongs to exactly one texture patch, so each vertex on the seam is copied to 2 vertices, and the vertex v _left belongs to the seam Seam the patch on the left side, v _right belongs to the patch on the right side of the seam; before color adjustment, each vertex has a unique color f _v , and then add a correction g _v to each vertex calculation:

The first item of this formula ensures that the colors on the left and right sides of the seam are as similar as possible, and the second item aims to make the difference between two adjacent vertices v _i and v _j of the same texture map patch smaller, and λ is the adjustment factor; at each vertex, After optimal _gv , the texture is corrected using barycentric coordinates interpolated from _gv of surrounding vertices. Finally, corrections are added to the input image, texture blocks are packed into texture atlases, and texture coordinates are attached to vertices;

For the selection of views, use the graph cut and alpha expansion to optimize the energy formula of the Markov random field, and use the photometric consistency detection principle to replace the original smoothing item; for the data item Compute the corresponding image gradient magnitude of the projected mesh patch F _i by using a Sobel detector And sum the pixels of the gradient magnitude image φ(F _i , l _i ) of all grid faces F _i ;

From the energy equation, the first step is the marking process, the grid of the surface F ₁ , F ₂ ,...F _K , the texture segment V ¹ , V ² ,...V ^N all correspond to the initial view; this is given a Label vector M={m ₁ ,m ₂ ,...,m _K }∈{0..N} ^K , specifying from segment Go to the surface F _i for mapping; set the cost value (The factors involved can be angle, distance, color difference), the smaller the cost value, the more suitable the texture segment is for the corresponding surface, which can be expressed as The seam is the smallest and the quality of the patch segment is the best: E(M)=E _Q (M)+λE _S (M), the first term of this equation is the patch quality The latter term E _s (M) is the distinguishability of a series of adjacent patch gaps P _i (x) is the projection operator, d( , ) is the Euclidean distance of RGB space;

To adjust the surface color and reduce the visibility of seams, the specific method is as follows: Calculate the average color value of the surface projection, all views that can see the surface are the inner layer graph, calculate the average value and covariance of the color mean value of all inner graphs, and pass The multivariate Gaussian function evaluates the degree of view visualization, sets the threshold, iterates the above steps, and uses the covariance or the number of inner layers as the iteration termination condition;

Query along the edge to alleviate the color problem, and attenuate according to the weight of the edge length: vertex v ₁ is adjacent to v ₀ and v ₂ respectively, and its color is searched for the weighted average value on the samples of v ₁ v ₀ and v ₁ v ₂ , and then As the distance from v ₁ increases, the weight decays from 1 to 0;

Global optimization cannot eliminate all visible gaps. Poisson editing needs to be added for local optimization. To limit the Poisson editing of a color to a box between 20 pixels, we assign the value of each outer edge pixel to the sticker The average value of the color of the pixels in the tile's image is compared with the average value of the images assigned to adjacent tiles, and the value of each inner edge pixel is fixed to its current color. If the patch is too small, we omit the inner edge.

After color adjustment, the grid model is spliced and textured to finally obtain a fine, visualized 3D model of the spatial object.

Concrete embodiment: this embodiment selects hardware environment: 2GHz 2*E5 CPU, 128G memory, 12G video memory computer;

The operating system includes Windows7, Ubuntu system;

Adopt C++ to compile and realize this method; Implementation example adopts image resolution: 2048*2048;

Overall scheme design of the present invention is as shown in Figure 1, and concrete implementation is as follows:

Step 1, as shown in Figure 2, use the SIFT feature descriptor to sequentially extract the precise and robust point features of the sequence image, and quickly and accurately match the adjacent images through the hash cascade; combine the camera internal parameter matrix of the image to calculate the image sequence The relative rotation matrix and translation matrix, using the motion recovery structure, obtains the relative pose between the image sequences, and obtains the sparse point cloud model of the space object;

Taking the selected image set as input, first calculate the difference between adjacent scale images, obtain a series of images and find the extreme points in the image space; then filter the extreme points, traverse the pixels in the Gaussian difference image in turn, and its Comparing the eight neighbors and the nine adjacent pixels of the upper and lower layers of the image where they are located, calculating extreme points; finding stable feature points, and screening these extreme points to remove unstable points; Select the Gaussian smooth image closest to the scale for the scale of the point; select a neighborhood around the key point, and perform Gaussian weighting on the gradient of the midpoint, and generate a descriptor for each sub-region;

Use the hash cascade for fast and accurate matching, the specific method is as follows: use the locally sensitive hash strategy, use the inner product similarity to judge the hash equation of the hash cluster, select a random vector of hyperplane r from the D-dimensional normal distribution, and set The 128-dimensional feature descriptor is converted into a binary code; a multi-table hash query with a short code is used to perform a rough search, and the returned candidates are remapped to a higher-dimensional Hamming space and the Hamming distance is calculated to establish the Hamming distance The hash table is used as the key reference for accurate query, and finally the cascaded hash strategy is used for accurate and fast matching between images;

Using the eight-point algorithm to calculate the fundamental matrix, the specific method is as follows: the data set is the corresponding relationship of the feature matching output, and the constraint is the epipolar geometry, and the eight-point algorithm can be used to estimate the fundamental matrix F with eight corresponding relationships; The matching points X ₁ and X ₂ between them must satisfy the epipolar geometric constraint X ₁ ^T FX ₂ ＝0, this equation is a linear homogeneous equation with 9 unknown parameters about F, because F differs in the sense of a constant factor is unique, so a non-zero parameter can be normalized into 8 unknown parameters, so if 8 pairs of matching points can be obtained, the basic matrix F can be determined linearly;

Using RANSAC for stereo matching, the specific method is as follows: uniformly and randomly sample 8 points from the feature points; use these 8 groups of corresponding points to find the minimum configuration solution of the fundamental matrix F; use each group of corresponding points not drawn to verify F, If the distance is small enough, then the correspondence is consistent; otherwise, return;

Through data optimization, the relative pose between the image sequences is obtained, and the sparse point cloud model of the spatial object is obtained. The specific method is as follows: add the next camera incrementally, and obtain the three-dimensional point in space by the triangle method. For a given camera parameter set {P _j } and trajectory set M ^j refers to the three-dimensional coordinates of the trajectory, Refers to the camera whose two-dimensional image coordinates are projected on i _th , and the cluster adjustment requires the minimum nonlinear error E(P,M) to be minimum:

Step 2. Use the result of step 1 as input to obtain a 3D point cloud model with dense spatial objects. The specific method is as follows: detect feature points through Harris and Difference-of-Gaussian, and match these feature points between multiple images to obtain a A series of sparse patches; in each image, take a rectangular grid of β ₂ × β ₂ pixels, and return as the local maximum value η reflecting corners and points; reconstruct a Sparse patch set, stored in the grid covering the image cellC(i,j): Assuming image I, its corresponding camera optical center is marked as O, for any feature point f in I, search in other images with A set F of feature points f' of the same type (Harris or DoG), use point pairs (f, f') to triangulate 3D points in space within 2 pixels of the corresponding epipolar line;

Given these initial matches, the next two steps are repeated 3 times; diffuse the initial matches to adjacent pixels, resulting in denser patches; in this step, we recursively add new neighborhoods to existing faces In the slice, until the observable surface in the scene is covered, if the two facets p and p' are stored in the continuous cells C(i,j) and C(i',j') of the same image I, and their method planes are close, we consider p and p' to be neighbors;

The visual constraints are used to remove those incorrect matches before and after the observed surface. The first step of filtering focuses on eliminating patches that are staggered on the actual surface, that is, considering a patch p ₀ and its occluded patch set U, when satisfied We eliminate p ₀ ; then the filter focuses on the outliers that are scattered inside the actual surface, and uses the depth map of the corresponding image to simply calculate S(p ₀ ) and T(p ₀ ) of each patch p ₀ , if| T(p ₀ )|<γ will filter out p ₀ , S(p ₀ ) means what should be visible, T(p ₀ ) means what is actually visible, γ means the threshold, Represents the mapped normalized mutual light mean;

Step 3. Using the result of step 2 as input, the method of obtaining an initially dense spatial object grid model is as follows: all input sample points are inserted into an octree data structure according to their scale values, and the generated octree The tree has a hierarchical structure, which gives a sampling set, that is, the vertex of the octree leaf node is used as the sampling point of the implicit function implictF(x), where x is the point position to be reconstructed, that is, the vertex of the octree leaf node As the sampling point of the implicit function, the octree is initialized as an empty octree without any nodes, the first sample point i is inserted into the newly created root node, the root node side length is _si and the sample point position P _i is the center;

Each sample point i has a scale value s _i , and the support domain radius of the sample is 3s _i . make:

Among them, l is the level of the octree where the sample point will be inserted, and S _l is the side length of the node at the lth layer of the octree.

As shown in Figure 3, the value of the implicit function is then estimated at these sample points. In order to distinguish these points from the input sample points, the nodes of the octree are divided into two categories: internal nodes and leaf nodes. The internal nodes are Refers to a node with eight child nodes, and a leaf node refers to a node without child nodes; in actual situations, the generated octree generally has mixed nodes, that is, nodes with a number of child nodes greater than 0 and less than 8; by assigning child nodes to mixed nodes The node fills its child nodes to 8 to regularize the octree, eliminates the mixed nodes in the octree through regularization, and adds new leaf nodes, which are called voxels; in order to calculate the hidden value at x Function value, using an efficient search method, only select samples in the octree that affect the value of the implicit function at x: recursively traverse the octree, and determine whether each node may contain a sample that affects the value of the implicit function at x Sample point; From the above formula, it can be seen that the maximum scale value of the sample point contained in node N does not exceed 2S _N , where S _N is the side length of N;

If a node satisfies the relational expression with x, the node can be directly skipped without entering its child nodes to search; otherwise, it is judged whether the distance between all sample points i and x in the node is less than 3s _i . Finally, use all searched sample points that have an influence on x to calculate the value of the implicit function F(x) according to the relational expression.

The octree determines the position of the voxel. After calculating the implicit function value at the voxel, it is no longer necessary to input the sample point at this time, and directly extract the zero equivalent value of the implicit function defined by the octree voxel. The surface is the final reconstruction surface. The common algorithm for extracting isosurfaces in standard voxels (all voxels have the same size) is the moving cube algorithm. The isosurface of implicit functions is extracted from the octree as the reconstruction surface. ;By setting the geometric confidence level of the obtained initial dense space target grid model, the redundancy is removed from the edge of the model inward, and the large-area surface envelope is removed;

Step 4. Use the result of step 3 as input to obtain a finely visualized 3D model of the spatial object. The specific method is as follows: the initial step is to determine the visibility of the input image surface, and use the paired Markov random field energy formula E to calculate a The label l is used to specify a view l _i as the texture corresponding to the mesh patch F _i , and the corresponding l _j as the texture corresponding to the mesh patch F _j , where F _i and F _j represent the triangle surface of the spatial target mesh model piece:

E _date represents the data item constraint for selecting the best view of the corresponding face F _i ∈ Faces, and E _smooth represents the smooth item constraint at the boundary seam of two mesh faces (F _i , F _j ) ∈ Edges.

After the label is obtained by minimizing the equation, the color of the patch is adjusted by first ensuring that each mesh vertex belongs to exactly one texture patch, so each vertex on the seam is copied to 2 vertices, and the vertex v _left belongs to the seam Seam the patch on the left side, v _right belongs to the patch on the right side of the seam; before color adjustment, each vertex has a unique color f _v , and then add a correction g _v to each vertex calculation:

The first item of this formula ensures that the colors on the left and right sides of the seam are as similar as possible, and the second item aims to make the difference between two adjacent vertices v _i and v _j of the same texture map patch smaller, and λ is the adjustment factor; at each vertex, After optimal _gv , the texture is corrected using barycentric coordinates interpolated from _gv of surrounding vertices. Finally, corrections are added to the input image, texture blocks are packed into texture atlases, and texture coordinates are attached to vertices;

For the selection of views, use the graph cut and alpha expansion to optimize the energy formula of the Markov random field, and use the photometric consistency detection principle to replace the original smoothing item; for the data item Compute the corresponding image gradient magnitude of the projected mesh patch F _i by using a Sobel detector And sum the pixels of the gradient magnitude image φ(F _i , l _i ) of all grid faces F _i ;

From the energy equation, the first step is the marking process, the grid of the surface F ₁ , F ₂ ,...F _K , the texture segment V ¹ , V ² ,...V ^N all correspond to the initial view; this is given a Label vector M={m ₁ ,m ₂ ,...,m _K }∈{0..N} ^K , specifying from segment Go to the surface F _i for mapping; set the cost value (The factors involved can be angle, distance, color difference), the smaller the cost value, the more suitable the texture segment is for the corresponding surface, which can be expressed as The seam is the smallest and the quality of the patch segment is the best: E(M)=E _Q (M)+λE _S (M), the first term of this equation is the patch quality The latter term E _s (M) is the distinguishability of a series of adjacent patch gaps P _i (x) is the projection operator, d( , ) is the Euclidean distance of RGB space;

Adjust the surface color to reduce the visibility of the seams. The specific method is as follows: Calculate the average color value of the surface projection. All views that can see the surface are the inner layer image. Calculate the average and covariance of the color mean value of all inner images. Through multivariate Gaussian The function evaluates the degree of visualization of the view, sets the threshold, iterates the above steps, and uses the covariance or the number of inner layers as the iteration termination condition;

Query along the edge to alleviate the color problem, and attenuate according to the weight of the edge length: vertex v ₁ is adjacent to v ₀ and v ₂ respectively, and its color is searched for the weighted average value on the samples of v ₁ v ₀ and v ₁ v ₂ , and then As the distance from v ₁ increases, the weight decays from 1 to 0;

Global optimization cannot eliminate all visible gaps. Poisson editing needs to be added for local optimization. To limit the Poisson editing of a color to a box between 20 pixels, we assign the value of each outer edge pixel to the sticker The average value of the color of the pixels in the tile's image is compared with the average value of the images assigned to adjacent tiles, and the value of each inner edge pixel is fixed to its current color. If the patch is too small, we omit the inner edge;

After color adjustment, the grid model is spliced and textured;

After the above reconstruction steps, a fine and visualized 3D model of the space object is finally obtained.

Claims (4)

1. a kind of extraterrestrial target three-dimensional reconstruction method based on high-resolution sequence image, it is characterised in that steps are as follows：

Step 1：Son, which is described, using SIFT feature extracts one group of extraterrestrial target high-resolution sequence chart image set I={ I successively₁,I₂,…, I_nAccurate steady point feature, and by Hash cascade to N number of neighbour's image of each width image setting to carrying out fast accurate Matching, obtains image I_iN number of neighbour's image to set of matches be M_i={ m_i,i-N,m_i,i-N+1,…m_i,j…,m_i,i+N-1,m_i,i+N}；

The feature of the accurate steady point feature shows as scale invariability, rotational invariance；

Step 2：Using RANSAC estimations to the image Jing Guo characteristic matching to carrying out Stereo matching, recycled in each RANSAC In, by from neighbour's image to set of matches M_iThe most image of middle selected characteristic match point is to as initial matching pair, joint figure The camera Intrinsic Matrix K of picture estimates basis matrix F using 8 algorithms；Singular value point is normalized to basis matrix F Solution, obtains the relative rotation matrices P and translation matrix t of image pair；Using increment type exercise recovery structure, obtains extraterrestrial target and catch Camera acquisition posture is obtained, structured image collection and the sparse point cloud model of extraterrestrial target are obtained；

Step 3：To structured image collection and the sparse point cloud model of extraterrestrial target, by sequence image I={ I₁,I₂,…,I_n} The matching of Harris or DoG angle point essences is carried out, spreads interpolation by carrying out bilinearity to sparse features point cloud model, and utilize luminosity Consistency principle constraint iteration filters out straggly outside actual surface and internal erroneous point, and spreading interpolation and filtering can change In generation 3 times, obtains the intensive three-dimensional point cloud model of extraterrestrial target；

Step 4, the three-dimensional point cloud model intensive to the extraterrestrial target of acquisition carry out the surface based on float in space scale implicit function It rebuilds：Using generating zero contour surface { x | implictF (x)=0 } of the implicit function that defines of regularization Octree voxel as space Object reconstruction model surface, obtains the extraterrestrial target grid model of initial densification, and grid is in triangular facet chip；

Step 5：Redundancy removing is carried out to the extraterrestrial target grid model of initial densification, setting geometry confidence threshold value is carried out by side The envelope removal of edge internally, obtains fine and close extraterrestrial target grid model；

Step 6：The seamless spliced texture mapping of large scale is carried out to fine and close extraterrestrial target grid model, with pairs of Markov Random field energy theorem E (l) specifies a view l_iAs surface mesh F_iCorresponding texture, corresponding l_jAs patch grids F_jCorresponding texture, wherein F_i,F_jRepresentation space target gridding model tri patch, the process finally obtain fine visual Extraterrestrial target threedimensional model, energy theorem are：

E_dateIt represents and chooses corresponding dough sheet F_iThe data item constraint of ∈ Faces optimal views, E_smoothThen characterize two patch grids Boundary seam crossing (F_i,F_j) ∈ Edges smooth item constraint；

Step 7：The neighbouring view that gradient amplitude is eliminated by luminosity consistency detection blocks, and further Euclidean distance is utilized to weigh Decaying carries out neighbour's surface color adjustment again, reduces extraterrestrial target model seam visibility, obtains fine visual space mesh Mark threedimensional model.

2. the extraterrestrial target three-dimensional reconstruction method based on high-resolution sequence image according to claim 1, it is characterised in that：Institute It states and utilizes increment type exercise recovery structure in step 2, obtain extraterrestrial target capture camera and acquire posture, obtain structured image collection Method with the sparse point cloud model of extraterrestrial target is：8 points of uniformly random extraction from characteristic point；It is asked using this 8 groups of corresponding points The minimal configuration solution of basis matrix F；Basis matrix F is verified with each group of corresponding points not being extracted into, is wanted when distance meets threshold value It asks, assert that correspondence is consistent；To be put into track after Stereo matching per piece image, obtain a multi-view image it Between matching characteristic point unicom collection；By way of adding a camera successively, all images camera is added in track, most Result is optimized by boundling adjustment afterwards, extraterrestrial target capture camera is obtained and acquires posture, structured image collection and space Target sparse point cloud model.

3. the extraterrestrial target three-dimensional reconstruction method based on high-resolution sequence image according to claim 1, it is characterised in that：Institute State step 7 obtain fine visual extraterrestrial target threedimensional model method it is as follows：It is built into markov random file energy Formula specifies a view as the corresponding texture of surface mesh, and energy theorem E (l) includes data item E_dateWith smooth item E_smooth；The neighbouring view that gradient amplitude is eliminated by luminosity consistency detection blocks, and further carries out surface color adjustment； It is used as the cost function of cost value w by set angle, distance, color distortionSelect cost The smaller view-set of value carries out image mosaic.

4. the extraterrestrial target three-dimensional reconstruction method based on high-resolution sequence image according to claim 3, it is characterised in that：Institute State the method for carrying out surface color adjustment：Image segmentation is carried out using Potts models, is not connected along adjacent seam edge inquiry color Continuous property, color distortion between reducing dough sheet of being decayed using Euclidean distance weight, is finally carried out local optimum using Poisson editor, reduced Extraterrestrial target model seam visibility.