CN110992263A

CN110992263A - Image stitching method and system

Info

Publication number: CN110992263A
Application number: CN201911183719.2A
Authority: CN
Inventors: 李振宇; 王万国; 王振利; 许玮; 李建祥; 刘广秀; 刘丕玉; 杨月琛; 李猷民; 杨立超; 鉴庆之; 杨波; 孙晓斌; 黄振宁
Original assignee: Electric Power Research Institute of State Grid Shandong Electric Power Co Ltd; State Grid Intelligent Technology Co Ltd
Current assignee: State Grid Intelligent Technology Co Ltd
Priority date: 2019-11-27
Filing date: 2019-11-27
Publication date: 2020-04-10
Anticipated expiration: 2039-11-27
Also published as: CN110992263B

Abstract

The invention discloses an image splicing method and system, comprising: extracting feature points of images to be spliced, describing the features, and obtaining coarse matching feature points; and counting the pixels in the adjacent pixel regions of the coarse matching point set that conform to the matching relationship The number of feature points, by calculating the feature neighborhood score, eliminates the wrong matching points; calculates the homography transformation matrix of the image to be stitched according to the accurately matched feature points; realizes the seamless image stitching through the weighted smoothing algorithm. The invention has the beneficial effects: the feature precision matching speed and registration accuracy are greatly improved, the synthesized image has no obvious geometric dislocation and blurring problems, and the edge transition of the overlapping area is good.

Description

Image splicing method and system

Technical Field

The invention relates to the technical field of digital image processing, in particular to an image splicing method and system based on grid motion statistics and weighted projection transformation.

Background

Image stitching technology has been widely studied in the fields of computer vision and graphics, and means that two or more images having an overlapping region are combined into a complete image with a wide viewing angle and little distortion. Common image splicing algorithms include an image gray scale method, a phase correlation method, an image feature method and the like. The image gray scale method has low complexity but poor robustness; the phase correlation method is fast, but is susceptible to image scale. The image characteristic method has better robustness to changes of illumination, scale and the like of an image, and is the mainstream image splicing technology at present.

Image characterization methods are generally divided into four steps: extracting characteristic points, matching the characteristic points, estimating and fusing a transformation model. The characteristic point matching comprises characteristic point rough matching and characteristic point fine matching. In the traditional image splicing based on the feature points, a RANSAC (random Sample consensus) algorithm is adopted to eliminate wrong rough matching, matching point pairs are randomly extracted by the method, the quality of the rough matching point pairs is not considered, the running time is in direct proportion to the iteration times, and the overall fine matching speed is slow. The problems of low feature matching speed, distortion of panoramic images and the like can be caused by high complexity of feature fine matching and low image registration precision in the traditional image splicing.

Such as: when the remote unmanned aerial vehicle is adopted to patrol the overhead transmission line, the remote unmanned aerial vehicle has the characteristics of large patrol range, multiple images, wide coverage and the like, so that the obtained image data volume is large, the splicing difficulty is high, and the operation speed of the splicing algorithm needs to be focused. In an image splicing algorithm based on an image feature method, such as an SIFT feature algorithm, although the splicing quality is high and the robustness is good, the algorithm is very complex and consumes much time, so that an algorithm with high splicing speed and splicing precision is required to realize image splicing of an overhead transmission line shot by an unmanned aerial vehicle.

Disclosure of Invention

The invention aims to solve the problems and provides an image splicing method and an image splicing system, which can realize rapid and high-quality image splicing.

In order to achieve the purpose, the invention adopts the following specific scheme:

an image stitching method disclosed in one or more embodiments includes:

extracting the feature points of the images to be spliced, and describing and matching the feature points to obtain rough matched feature points;

counting the number of feature points which accord with the matching relation in the adjacent pixel regions of the rough matching point set, and eliminating wrong matching points by calculating feature neighborhood scores;

calculating a homography transformation matrix of the images to be spliced according to the accurately matched characteristic points, and carrying out image registration; and realizing smooth image splicing by a weighted smoothing algorithm.

Further, the image feature extraction and description are performed by using an ORB algorithm.

Further, carrying out rough matching on the feature points by using a violent matching algorithm, and filtering wrong rough matching by using a cross matching method.

Further, dividing the image into a plurality of non-overlapping grids, counting the feature score of each grid, and meanwhile, counting the feature neighborhood scores of the grids adjacent to the grids in a set number; rotating the grids and the adjacent grids to obtain the maximum grid characteristic score; when the maximum grid feature score is larger than the grid feature score threshold value, judging that the matching is correct; otherwise, it is an error match.

Further, counting the feature neighborhood scores of eight grids adjacent to the current grid to obtain a Sudoku feature score.

Further, the average value of the number of the rough matching features of the grids in the current nine-square grid is counted, and the grid feature score threshold is determined according to the average value.

Further, a global homography matrix between the image sequences is constructed through the matching point pairs; calculating a homography matrix of local dependence by adding weight coefficients; the weight coefficient is determined according to the Gaussian distances from the current feature point to all feature points on the image.

Further, according to the obtained homography matrix of the local dependence, the corresponding images are transformed to determine the overlapping area between the images, and the image to be fused is mapped to a new blank image to form a splicing map.

In one or more embodiments, an image stitching system includes a server including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the image stitching method when executing the computer program.

A computer-readable storage medium is disclosed in one or more embodiments, on which a computer program is stored, which when executed by a processor performs the image stitching method described above.

The invention has the beneficial effects that:

(1) the invention provides a coarse matching feature point screening method for grid neighborhood feature statistics, which is characterized in that the number of feature points meeting the matching relation is counted in adjacent pixel regions of a coarse matching point set, mismatching feature points are removed, and the precise matching of the feature points is realized.

(2) The invention provides an image registration method based on a weighted projection transformation model, wherein a locally dependent homography transformation matrix is calculated by adding a Gaussian weight coefficient, and the ghost effect and parallax error caused by registration of a global homography matrix are solved.

(3) The invention greatly improves the precision matching speed and the registration precision of the characteristics, the synthesized image has no obvious problems of geometric dislocation and blurring, and the edge transition of the overlapping area is good.

Drawings

FIG. 1 is a flow chart of an embodiment of the present invention;

FIG. 2 is a schematic diagram of distribution characteristics of correct matching and incorrect matching;

FIG. 3 is a schematic diagram of grid division and Sudoku grid neighborhood;

fig. 4(a) - (c) are schematic diagrams of the squared figure, squared figure rotated 1 time and squared figure rotated 4 times respectively.

The specific implementation mode is as follows:

the invention is described in detail below with reference to the accompanying drawings:

it is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.

Example one

In one or more embodiments, an image stitching method is disclosed, as shown in fig. 1, a feature extraction ORB algorithm is applied to image stitching, correct matching and incorrect matching are distinguished by counting the number of features meeting a matching relationship in pixel regions adjacent to coarse matching feature points, and the fine matching speed of the feature points is improved; and then, a weighted projection transformation method is adopted, the pixel distance relation is introduced to optimize the weight coefficient of the transformation model, and the registration precision of the overlapped region is improved. The method comprises the following specific implementation processes:

1. feature extraction

The method is realized by the following steps:

(1) feature point extraction

ORB (organized FAST and rotaed BRIEF) is an algorithm for FAST feature point extraction and description. The ORB algorithm is divided into two parts, namely feature point extraction and feature point description. The feature extraction is developed by fast (features from acquired Segment test) algorithm, and the feature point description is improved according to brief (binary robustindendementelementary features) feature description algorithm.

A point P is selected from the image, and whether the point is a characteristic point is judged by the FAST algorithm, namely, a circle with the radius of r is drawn by taking the point P as the center of the circle. If the gray value of n continuous pixel points on the circumference is larger or smaller than the gray value of the P point, the P point is considered as the characteristic point. In order to solve the problem of a plurality of characteristic points at adjacent positions, the invention sorts the detected FAST characteristic points according to Harris corner response values, and selects the points with the response values of the first 80% (if 100 FAST characteristic points are detected in total, the 80 points with the Harris corner response values ranked in the front are extracted) as the extracted characteristic points.

In order to realize multi-scale invariance of the characteristic points, the invention adopts a form of building a pyramid image. By setting a scaling factor scaleFactor (the value of the invention is 1.2) and the number of pyramid layers nlevels (the value of the invention is 8). And then the original image is reduced into n levels of images according to the scale factor. The scaled image is: i ═ I/scalefactor (k ═ 1,2, …, nlevels). And (4) extracting the sum of the characteristic points of the n images with different proportions as the characteristic point of the image.

Since FAST features do not have directionality, the ORB algorithm uses moment (moment) methods to determine the direction of FAST feature points. And calculating the centroid of the feature point in a radius range by using the r as the moment, wherein a vector is formed from the coordinates of the feature point to the centroid as the direction of the feature point. Moments are defined as follows:

wherein, I (x, y) is the pixel value of the image pixel (x, y). The centroid of this moment is:

a direction vector is formed between the image feature point and the centroid, so that the direction of the feature point can be characterized by the angle of the direction vector, and the calculation formula is as follows:

(2) description of characteristic points

The ORB algorithm is characterized by adding twiddle factor improvement on the basis of BRIEF characterization. The BRIEF algorithm calculates a binary string of feature descriptors. It selects n pairs of pixel points pi, qi (i ═ 1,2, …, n) in the neighborhood of a feature point. The magnitude of the gray value of each point pair is then compared. If I (pi) > I (qi), the value of the corresponding position in the generated binary string is 1, otherwise, the value is 0. All the point pairs are compared, and a binary string with the length of n is generated. In order to increase the noise resistance of the feature descriptors, the method firstly performs Gaussian smoothing on the image, replaces the value of a certain point pair with mxm neighborhood gray level average value of a certain point in the neighborhood, and then compares the sizes of the point pair, so that the feature values have higher noise resistance.

Because the BRIEF descriptor lacks rotation invariance, the capability of resisting image plane rotation is poor, and the rotation invariance needs to be added to enhance the capability of resisting noise. For a feature point, the BRIEF descriptor is formed by comparing n point pairs, combining n point pairs (2n points) together to form a matrix B, which is in the form:

and then constructing a rotation matrix by using the directions of the characteristic points, wherein the rotation matrix is in the form of:

correcting the matrix B through the rotation matrix to enable the matrix B to have rotation invariance, wherein a correction formula is as follows:

S_θ＝R_θS

and a rotation invariant descriptor is extracted according to the direction of the feature point, so that the sensitivity of a BRIEF operator to the direction is overcome.

2. Coarse matching of features

The invention utilizes a Brute Force (BF) algorithm to carry out rough matching on the characteristic points to obtain N groups of rough matching characteristic point pairs which are marked as { F_a,F_bIn which F_a＝{f_a1,f_a2,...,f_aNAnd F_b＝{f_b1,f_b2,...,f_bN}. The violence matching algorithm compares descriptors of feature points one by one in a vector space, and selects a pair with a smaller distance (Hamming distance is adopted by the invention) as a matching point. After violence matching is performed, the invention uses a cross matching method to roughly filter the wrong matching. The idea of cross-filtering is to use the matched points to match in reverse after a match is made, and to consider a correct match if the matched point is still the first matched point. For example, the first-time feature point a uses a violent matching method, and the matched feature point is the feature point B; and in turn, using feature point B for matching, if feature point a is still matched,this is considered a correct match, otherwise it is an incorrect match.

3. Fine matching of features

(1) Feature neighborhood score definition

For image matching, a proper match has a certain number of matching feature points in the neighboring regions on the two images, while the neighboring regions that are mistakenly matched on the two images are different, so the number of feature point matches in the neighboring regions is usually zero, as shown in fig. 2. Therefore, the number of the feature points with the matching relation can be counted in the neighborhood of the rough matching feature point set, and the elimination of the error matching is realized. Therefore, the invention distinguishes correct matching and wrong matching by counting the number of feature points which accord with matching relation in the pixel areas adjacent to the rough matching point set.

Will { I_a,I_bThe neighborhood of the matching feature point set in { N } is denoted as { N_a,N_bIn which N is_a＝{N_a1,N_a2,...,N_aN}，N_b＝{N_b1,N_b2,...,N_bN}. For the ith set of matching point pairs { f_ai,f_biGet statistics of f_aiNeighborhood N_aiFeature point set of (1) { f)_a1,f_a2,...,f_aMiAnd total number of feature points M_iAnd counting the feature point set corresponding to the coarse matching of the feature points

And the corresponding feature point sets are located at f_biNeighborhood N_biS of_i. According to M_iAnd S_iIs set as a score threshold S_TTo determine the ith matching point pair { f_ai,f_biWhether it is a correct match. Finally, all matched feature point sets are traversed to eliminate error matching, and a fine matched feature point set { F is obtained_a',F_b'}. For convenience of description, S will be_iCalled feature neighborhood score, the calculation formula is as follows:

in the formula s_ikRepresents and N_aiWhether the characteristic point of the k-th characteristic point rough matching is positioned in N_biIf in N_biIf so, the value is determined as 1, otherwise, the value is determined as 0.

(2) Grid feature statistics

To speed up statistics, the invention divides an image into grids that do not overlap G-P × Q, i.e., { I ═ Q }_a,I_bDivide it into a set of grid blocks { a, B }, where a ═ a }₁,a₂,...,a_i,...,a_G},B＝{b₁,b₂,...,b_j,...,b_GIs a and a_iIs represented by_aThe ith grid, b_jIs represented by_bThe jth grid in (1), as shown in fig. 3. In order to increase robustness, the feature neighborhood score of each grid is counted, and the grid feature neighborhood scores of 8 grids adjacent to the grid are counted, and the count is called as a Sudoku feature neighborhood score S:

in the formula, S_i,jIs the j-th grid feature score in the nine-square grid in which the ith grid is located.

To avoid the effect of inter-image rotation on statistics, pair I_bThe upper nine-square grid is rotated clockwise as shown in fig. 4(a) - (c). Wherein the grid G₅The rotation does not change its position, but the adjacent grid moves clockwise, e.g. grid G in the upper left corner₁After the 1 st rotation, grid G₄After 4 th rotation is grid G₉When the rotation is performed for the 9 th time in sequence, the distribution situation of the grid features is consistent with that of the grid features shown in the graph (a) in FIG. 4, and the maximum grid feature neighborhood fraction under the condition of 8 rotations is counted

In the formula (I), the compound is shown in the specification,

and (4) rotating the nine-square lattice where the ith grid is positioned for k times to obtain the characteristic score of the jth grid. Then, the average value of the number of the rough matching features of the grid in the current nine-square grid is counted:

in the formula, M_i,jAnd the number of the rough matching feature points in the jth grid in the nine-square grid in which the ith grid is positioned is shown. When grid feature scores

Greater than a grid feature score threshold S_TWhen it is, determine { f_ai,f_biIt is a correct match, otherwise it is an incorrect match.

In the formula, α is the weight of the feature point number average.

4. Image registration stitching

The image registration is a technology for determining an overlapping area and an overlapping position between images to be spliced, and the image registration method based on characteristic points is adopted, namely a transformation matrix between image sequences is constructed through matching point pairs, so that the splicing of panoramic images is completed. According to the inter-image transformation matrix H, the corresponding images can be transformed to determine the overlapping area between the images, and the images to be fused are mapped to a new blank image to form a splicing map.

If p is_a＝[x_a,y_a]^TAnd p_b＝[x_b,y_b]^TRepresenting an image { I_a,I_bA pair of matching points in (c). N' groups of fine matching point pairs obtained by the steps

The global homography matrix H can be solved:

in the formula (I), the compound is shown in the specification,

and

are each p_aAnd p_bOf (d) homogeneous coordinates of (d), H ∈ e^3×3。

When I is_aAnd I_bWhen the image is not obtained by rotating the camera around the optical center of the camera or the image background cannot approximate a plane scene, the global homography matrix H is used as a transformation model, and a ghost effect or parallax error is caused after registration. To solve this problem, the present invention calculates the locally dependent homography matrix H by adding weight coefficients_*Then to I_bEach point p in_b*The transformation is carried out, and the transformation is carried out,

wherein H_*Calculated from the following equation:

in the formula, a weight matrix

A∈^2N×9Is a matrix of direct linear transformation equations, the vector H is a variant of the matrix H, H ═ H₀₀h₀₁h₀₂h₁₀h₁₁h₁₂h₂₀h₂₁h₂₂]Coefficient of weight

Is based on the current point p_b*To I_bAll the characteristic points of

Is determined by the Gaussian distance of (1), set to p_b*The closer the neighborhood is, the larger the pixel weight is, the relatively far pixel weight takes a corresponding smaller value:

where σ is a scale parameter. To prevent the weighting coefficients from being too sparse, a default compensation value γ ∈ [0,1] is introduced.

In the fusion process, the overlapped area has suture lines, so that processing is needed, and a weighted smoothing algorithm is adopted, wherein the main idea of the algorithm is as follows: the gray value Pixel of the Pixel point in the image overlapping region is obtained by weighted average of the gray values Pixel _ L and Pixel _ R of the corresponding points in the two images, namely, Pixel is k × Pixel _ L + (1-k) × Pixel _ R, wherein k is an adjustable factor, and in the overlapping region, along the image splicing direction, k gradually changes from 1 to 0, thereby realizing smooth splicing of the overlapping region.

Example two

The embodiment discloses an image splicing system, which comprises a server, wherein the server comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, and the image splicing method of the embodiment is realized when the processor executes the program.

EXAMPLE III

The present embodiment discloses a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, performs the image stitching method described in the first embodiment.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, it is not intended to limit the scope of the present invention, and it should be understood by those skilled in the art that various modifications and variations can be made without inventive efforts by those skilled in the art based on the technical solution of the present invention.

Claims

1. an image stitching method, is characterized in that, comprises:

Extract the feature points of the image to be spliced, describe and match the feature points, and obtain the coarse matching feature points;

Count the number of feature points that match the matching relationship in the adjacent pixel area of the coarse matching point set, and eliminate the wrong matching points by calculating the feature neighborhood score;

According to the accurately matched feature points, the homography transformation matrix of the image to be spliced is calculated, and the image is registered; the weighted smoothing algorithm is used to realize the seamless image splicing.

2. An image stitching method as claimed in claim 1, characterized in that, ORB algorithm is used to extract and describe image features.

3. An image stitching method according to claim 1, characterized in that, a brute force matching algorithm is used to perform rough matching of feature points, and a cross-matching method is used to filter mismatched feature points.

4. a kind of image stitching method as claimed in claim 1 is characterized in that, divides the image into a plurality of non-overlapping grids, counts the number of feature points in each grid, and counts the setting adjacent to it simultaneously The feature neighborhood score of the number grid; the grid and its adjacent grids are rotated to obtain the maximum grid feature neighborhood score; when the maximum grid feature neighborhood score is greater than the grid feature score threshold, It is judged as a correct match; otherwise, it is a false match.

Denote the neighborhood of the matching feature point set in {I _a ,I _b } as {N _a ,N _b }, where _{Na ={N a1} _, N _a2 ,...,N _aN }, N _b ={N _b1 ,N _b2 ,...,N _bN }. For the i-th matched point pair {f _ai ,f _bi }, count the feature point set in the f _ai neighborhood _Nai

and the total number of feature points M _i , and count the feature point sets corresponding to the rough matching of these feature points

and the number S _i of these corresponding feature point sets located in the neighborhood N _bi of _f _bi , define Si as the feature neighborhood score.

5. a kind of image stitching method as claimed in claim 4, it is characterized in that, while the grid feature score where the feature point is located, count the grid feature neighborhood scores of eight adjacent grids with it, and obtain the nine-square grid feature Neighborhood Score.

In the formula, S _i,j is the feature neighborhood score of the i-th grid in the image I _a corresponding to the j-th grid of I _b .

6. a kind of image stitching method as claimed in claim 5 is characterized in that, the mean value Mi _,j of the rough matching feature quantity in the current nine-square grid grid is counted, and the grid feature score threshold S _T is defined as:

Rotate the nine-square grid to get the largest grid feature neighborhood score

when

When it is greater than the threshold value S _T , it is determined that {f _ai , f _bi } is a correct match, otherwise, it is an incorrect match.

7. A kind of image stitching method as claimed in claim 1, it is characterized in that, build the global homography matrix between image sequences by matching point pairs; Calculate the homography matrix of local dependence by adding weight coefficient; Wherein, The weight coefficient is determined according to the Gaussian distance from the current feature point to all feature points on the image.

8. A kind of image stitching method as claimed in claim 7, it is characterized in that, according to the homography matrix of obtained local dependence, transform corresponding image to determine the overlapping area between the images, and map the image to be fused to . to a new blank image to form a mosaic.

9. An image stitching system, characterized in that, comprising a server, the server comprising a memory, a processor and a computer program stored on the memory and running on the processor, and the processor implements the rights when executing the program. The image stitching method described in any one of requirements 1-8.

10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the image stitching method according to any one of claims 1-8 is executed.