CN104021568B - Automatic registering method of visible lights and infrared images based on polygon approximation of contour - Google Patents

Abstract

The invention relates to an automatic registering method of visible lights and infrared images based on polygon approximation of a contour, belonging to the field of multi-source image registration. The method mainly comprises the following five steps: I, applying an edge detection operator and contour tracing algorithm to extract a main contour of an image; II, carrying out polygon approximation on the extracted contour; III, matching the fitted contour and selecting control points; IV, selecting an image transformation model and estimating transformation parameters; and V, carrying out corresponding transformation and interpolation operation to input images according to the transformation parameters. The method provided by the invention is high in registering precision and fast in speed and can effectively solve the automatic registering problem of visible lights and infrared images under rigid body transformation.

Description

Automatic Registration Method of Visible Light and Infrared Images Based on Contour Polygon Fitting

technical field

The invention relates to an automatic registration method of visible light and infrared images based on contour polygon fitting, and belongs to the field of multi-source image registration.

Background technique

Different from single-sensor images, visible light and infrared images from different sensors have large differences in gray value, image contrast, sensitive targets, etc., which increases the difficulty and complexity of image registration. Common image registration methods are divided into two categories: methods based on image grayscale and methods based on image features. Because the grayscale features of images acquired by different sensors are not consistent, it is difficult to register such images using image grayscale-based registration methods. The feature-based registration method can extract the salient features of the image, such as contours, corners, etc., compresses the amount of image information, has a fast registration speed, and is robust to image grayscale transformation. With the development of multi-source image fusion technology, registration methods based on image features have been widely used in the field of image registration. This kind of method mainly includes four aspects: feature extraction, feature matching, selection of transformation model and calculation of parameters, coordinate transformation and interpolation.

The most widely used line feature is the outline feature. Paper: Dai X L, Khorram S.A Feature-based Image Registration Algorithm Using Improved Chain-code Representation Combined with Invariant Moments[J].IEEE Transactions on Geoscience and RemoteSensing,1999,37(5):2351-2362. A multi-source image based on The contour image registration method only matches the closed contour of the input image, and estimates the registration parameters according to the centroid of the matched closed contour. This method has high accuracy of image registration, but it needs to be able to detect better closed contours from the input image. This registration method is not suitable when matching closed contours cannot be detected from the input image. Paper: Li H, Manjunath B S, Mitra S K.A Contour-based Approach to Multisensor Image Registration[J].IEEE Transactions on Imaging Processing,1995,4(3):320-334. Proposed a contour-based multisensor image registration method, by matching the open contour and the closed contour respectively, the corner points of the matching open contour and the centroid of the matching closed contour are selected as control points. This method makes full use of the open contour and closed contour information of the image, and the registration accuracy is relatively high. However, the algorithm complexity of this method is high, and the corner detection has a great influence on the registration accuracy.

Contents of the invention

The purpose of the present invention is to propose an automatic registration method of visible light and infrared images based on contour polygon fitting to address the shortcomings of existing multi-source image registration methods.

To achieve the above object, the present invention adopts the following technical solutions:

A method for automatic registration of visible light and infrared images based on contour polygon fitting, comprising the following steps:

1). Extract the main contours of the reference image and the image to be registered respectively;

2). After contour extraction, there are many redundant points and noises in the contour, which will not only increase the difficulty of matching, but also for complex contours, errors are more likely to occur during the registration process; polygon fitting is required for the extracted contours , simplify the complex outline, and remove redundant points and noise in the outline;

3). Contour matching and control point selection;

4). Select the image transformation model and estimate the image transformation parameters according to the control points;

5). Perform resampling and interpolation operations on the image to be registered according to the estimated transformation parameters. For each pixel coordinate in the registration result image, its coordinates in the image to be registered are calculated one by one according to the transformation parameters, which can easily cause the coordinates of the calculated pixels to be not all integers, and interpolation operations can be used to solve this problem .

The contour extraction in the step 1) is mainly divided into two parts: edge detection and contour tracking. The infrared image is used as the reference image, and the visible light image is used as the image to be registered. Although the gray similarity of infrared and visible light images is very low, they have common target information, and the contour correlation of common target information is relatively large. Canny operator can be used to detect the contours of the reference image and the contour of the image to be registered, which are highly correlated. Freeman chain code is used to describe the contour, and the coordinates and chain code of each boundary point are stored through contour tracking. The chain code way to describe the contour needs to record the starting point of the contour and the chain code value list of each point on the contour relative to the previous point. Take the 8-domain chain code as an example, first search the starting point of the contour in the order from left to right and from top to bottom; then search for the second contour point in the order of left bottom, bottom, right bottom and right in the counterclockwise direction, and then follow Search for other contour points in the order of lower left, lower, lower right, right, upper right, upper, upper left, and left until the starting point of the next contour is found. The traced outlines are of varying lengths and contain a lot of messy texture edges. In order to improve operating efficiency and obtain useful contours that are most conducive to matching, in the contour tracking process, combined with the contour extraction effect, a contour length threshold T _C can be set, and only contours exceeding this threshold are retained.

In the step 2), what polygon fitting is carried out to the contour adopts a fitting method of iterative endpoints: see the paper for details: Zhang Fan, Zhai Zhihua, Zhang Xinhong. A fast algorithm for polygon fitting in image processing [J]. Computer Development and Application, 2001,4(10):474-478. The schematic diagram of curve fitting is shown in Figure 1. In this fitting algorithm, the more iterations, the higher the fitting accuracy of the contour, and the closer the fitted contour is to the original contour. The main steps of the fitting algorithm to fit a contour are as follows:

(1) Set a distance threshold T;

(2) Select the starting point A and the ending point B of the contour line as two endpoints of the fitted polygon;

(3) Calculate the distance between all points between A and B on the contour line AB to the line connecting A and B, select the point C with the largest distance among them, and set the maximum distance as H;

(4) compare H and T, if H＞T, illustrate that C is an end point of the fitting polygon, continue step (5); If H＜T, then jump out of algorithm, illustrate that there is no end point on this segment contour line;

(5) The endpoint C divides the contour line AB into two parts AC and BC, and finds out the endpoints of the two parts of the contour line according to (2), (3), (4), and (5) respectively; Get all the endpoints (A, B, C, D...) on the curve AB, and connect them in order to obtain the final fitted polygon. The endpoints A, B, C, D... are the vertices of the polygon.

The contour matching in the step 3) adopts a chain code matching method. A digitized contour curve can be represented by an eight-direction Freeman chain code. As shown in Figure 2, P _i represents the current pixel, i is the pixel index value, C _i represents the chain code value of P _i , and C _i ∈{0,1,…,7}. If the next pixel of P _i is at position b ₆ , then C _i is 6. Set the reference image and the image to be registered as the first image and the second image respectively, and according to the encoding method of Freeman chain code, the chain code of each contour in the first image and the second image can be obtained, and the two images can be polygon The fitted contour is represented by chain code.

Suppose the chain code {a _i } represents a contour A of length N _A in the first image, and the chain code { _bi } represents a contour B of length N _B in the second image; the (k+1)th pixel on A is the starting point, and the (l+1)th pixel on B is the starting point, respectively intercepting the contour segments with length n; among them, the chain code value corresponding to the (k+1)th pixel on A is a _k , and the ( The chain code value corresponding to l+1) pixels is b _l ; then the matching degree of two contour segments is defined as:

In the formula, Among them, i,j∈[0,n-1], D _kl ⁿ represents the matching degree of the two contour segments of A and B with length n, a _k+j represents the (k+j+1)th pixel on A chain code value, b _l+j represents the chain code value of the (l+j+1)th pixel on B; a matching degree threshold T _D is set, when It shows that the two intercepted contour segments are matched.

To obtain the control points, it is necessary to select the matching feature points on the matching contour. The feature of each contour is a collection of several small contour features centered on the feature point. Therefore, a contour segment of a certain length near the feature point on the contour can be selected as a matching unit to find a matching contour segment in another image. That is, take the feature point on the contour as the center, select T _L pixel points before and after to form a feature chain code segment with a length of (2T _L +1), traverse another image, continuously calculate the matching degree, and select the largest value greater than T _D The feature contour segment corresponding to the matching degree is the matching contour segment, and the corresponding feature point is the matching feature point.

The detection of feature points has a great influence on the registration accuracy of images. Observing the extracted contour image, it is found that the contour feature points mainly include corner points, tangent points and inflection points. Corner points have large curvatures and are relatively easy to extract; tangent points and inflection points have small curvatures and are difficult to extract. Using the corner detection method based on Freeman chain code, there will always be missed detection, which increases the difficulty of setting the matching threshold parameters. In addition, when the number of matched corners is small, the registration accuracy of the image will be reduced. For the corner detection algorithm, see: Yu Bo, Guo Lei, Zhao Tianyun, etc. Curve matching method described by Freeman chain code [J]. Computer Engineering and Application, 2012, 48(4): 5-8. From the polygon fitting process we can see , all corner points, tangent points, and inflection points are included in the polygon vertices. Therefore, the vertices of the polygon can be selected as the feature points, and there will be no missed detection, and the registration effect will be better. The specific operation flow chart is shown in Figure 3.

Described step 4) transformation model adopts rigid body transformation model in:

in, is the coordinates of the control points in the image to be registered, Δx is the horizontal displacement in pixels; Δy is the vertical displacement in pixels; θ is the rotation angle in degrees.

Define the transformation parameter matrix: The image control point matrix to be matched: Reference image control point matrix: Where m is the number of matched control point pairs, m>2, and and and ..., and for matching control point pairs. may have: According to this formula, the transformation parameter matrix M can be calculated.

A clipped least squares algorithm is used to estimate the transformation parameter matrix. The main steps of the algorithm are as follows:

(1) Assuming that m pairs of control points form a set P, the least squares solution to calculate the transformation matrix M of all points in P is:

(2) Use the transformation matrix M and matrix X obtained in (1) to obtain the estimated value Calculate estimates with actual value The error between the estimated and actual values for each pair of control points in :

in, is the actual value of the control point i in the image to be registered, is the estimated value of the control point i in the image to be registered, i∈{1,2,…,m}.

(3) Delete the pair of matching control points corresponding to the above-mentioned Error _max , and update the set P′. Then use the updated P' to calculate a new transformation matrix M' using the least squares method.

(4) Set an error threshold T _E , and repeat (2) and (3) continuously. Until Error _max < T _E , the final transformation matrix is obtained.

When the number of outliers is less than half, the clipped least squares method has a strong error tolerance, can continuously eliminate outliers, and can correctly estimate the transformation parameters.

The interpolation algorithm in the step 5) is bilinear interpolation. After obtaining the transformation parameters, it is necessary to use the transformation parameters to perform corresponding coordinate transformation on the image to be registered. For each pixel coordinate in the registration result image, its coordinates in the image to be registered are calculated one by one according to the transformation parameters, which can easily cause the coordinates of the calculated pixel points to not all be integers, and interpolation method can be used to solve this problem . Bilinear interpolation is an interpolation method with a small amount of calculation, can eliminate aliasing to a large extent, and has good interpolation effect. The essence of this method is to use the pixel value weights of 4 neighborhood integer points to estimate the pixel value of the current non-integer point. The schematic diagram of the algorithm is shown in Figure 4.

Table 1

Table 1 is the method of the present invention and literature Li H, Manjunath B S, Mitra S K.A Contour-basedApproach to Multisensor Image Registration [J]. IEEE Transactions on Imaging Processing, 1995, 4 (3): 320-334. and manual registration method Experimental comparison results. Experimental results show that the parameter estimation of the method of the present invention is closest to the actual value of the parameter. Besides, the RMSE of the method of the present invention is minimal. It can be seen that compared with the other two methods, the registration accuracy of the method of the present invention is the highest. The registration results of the above three images to be registered using the method of the present invention are shown in FIGS. 9-11 .

The invention has high registration precision and fast speed, and can effectively solve the problem of automatic registration of visible light images and infrared images under rigid body transformation.

Description of drawings

Fig. 1 is a schematic diagram of curve polygon fitting;

Fig. 2 is a schematic diagram of Freeman chain code direction value and chain code direction;

Fig. 3 is the polygon vertex flowchart of selection match;

Fig. 4 is a schematic diagram of bilinear interpolation (Denotes integer pixels in the image);

Fig. 5 is the input infrared image;

Figure 6 is the input visible light image;

Figure 7 is the first image to be registered;

Figure 8 is the second image to be registered;

Figure 9 is the third image to be registered;

Fig. 10 is a reference image and an image to be registered-contour matching diagram (the left is the reference image, the right is the image to be registered);

Fig. 11 is a two-contour matching diagram of a reference image and an image to be registered (the left is a reference image, and the right is an image to be registered);

Fig. 12 is a three-contour matching diagram of the reference image and the image to be registered (the left is the reference image, and the right is the image to be registered);

Fig. 13 is a registration result diagram of image 1 to be registered;

Fig. 14 is a registration result diagram of image 2 to be registered;

FIG. 15 is a registration result diagram of image three to be registered.

detailed description

The present invention will be further described below in conjunction with accompanying drawings and examples.

Step 1: Input the reference image and the image to be registered. As shown in Figure 5-6, there are two known registered 734×473 infrared images and visible light images. Taking the infrared image as a reference image, the following three geometric transformations are performed on the visible light image (1) counterclockwise rotation 3 degrees; (2) horizontal displacement 5, vertical displacement 5; (3) horizontal displacement 5, vertical displacement 5, counterclockwise Rotate 3°; obtain three images to be registered, as shown in Figure 7-9.

Step 2: Extract the contour information of the reference image and the three images to be registered respectively. Firstly, use the Canny operator to detect the edge of each image, and obtain the contour image with relatively large relative information between the reference image and the image to be registered. Then, track the detected contour and store the coordinates of each boundary point. During this process, the short contours and isolated points in the image are ignored, and only the contours exceeding a certain threshold length T _C are kept. Here T _C =80, see Figure 10-12 contour matching diagram. If the input image is noisy, it is necessary to filter and denoise the corresponding image before edge detection.

Step 3: Perform polygon fitting on the extracted contour. Comparing the fitting effect at different thresholds, it can be seen that the larger T is, the more serious the deformation of the fitted curve segment is, especially in the area where the original contour is not flat. Here, T=2 is selected, which can not only eliminate redundant points on the contour but also maintain the shape of the original contour curve to a greater extent. At the same time, the vertices fitted by each outline polygon are stored. Here the polygon vertices are selected as the feature points of the contour. The polygon vertices cover several common feature points such as corner points, tangent points, and inflection points, and there is no problem of missed detection. The feature points can be stored during the polygon fitting process, which reduces the complexity of the algorithm.

Step 4: Use the Freeman chain code to encode the contour after polygon fitting, and select the feature point as the center, and the length of T _L pixels before and after is (2T _L +1) feature chain code segment as the matching unit to perform contour matching . Set the reference image and the image to be registered as the first image and the second image respectively, assuming that the chain code {a _i } represents a contour A of length N _A in the first image, and the chain code {b _i } represents the contour A in the second image A contour B of length N _B ; with the (k+1)th pixel on A as the starting point and the (l+1)th pixel on B as the starting point, the contour segments with a length of n are intercepted respectively; wherein, the ( The chain code value corresponding to k+1) pixels is a _k , and the chain code value corresponding to the (l+1)th pixel on B is b _l ; then the matching degree of the two contour segments is defined as:

In the formula, Among them, i,j∈[0,n-1], D _kl ⁿ represents the matching degree of the two contour segments of A and B with length n, a _k+j represents the (k+j+1)th pixel on A chain code value, b _l+j represents the chain code value of the (l+j+1)th pixel on B; a matching degree threshold T _D is set, when It shows that the two intercepted contour segments are matched. The corresponding feature points on the matching contour segment are the matching feature points.

Step 5: Select the image transformation model and estimate the transformation parameters. Here it is assumed that there is a rigid body transformation between the reference image and the image to be registered, and the rigid body transformation model is:

in, is the coordinates of the control points in the image to be registered, Δx is the horizontal displacement in pixels; Δy is the vertical displacement in pixels; θ is the rotation angle in degrees.

Define the transformation parameter matrix: The image control point matrix to be matched: Reference image control point matrix: Where m is the number of matched control point pairs, m>2, and and and ..., and for matching control point pairs. may have:

Use the cropped least squares method to continuously eliminate the mismatched feature points, and estimate the transformation parameter matrix of the image according to the selected control points. Figure 10-12 shows the contour matching results of the three images to be registered. Use "■" to indicate the matched control points after filtering.

Step 6: Perform resampling and interpolation operations on the three images to be registered according to the calculated transformation parameters. Figure 13-15 shows the registration results of the three images to be registered.

Step 7: Describe the registration accuracy. The registration accuracy is described by root mean square error (RMSE). The smaller the RMSE value, the higher the registration accuracy. The calculation formula of RMSE is as follows:

In the formula, Among them, n′ is the logarithm of the final control points. In order to ensure the correct solution of the transformation parameters in the experiment, it is necessary to ensure that n′>2; ( _xi , y _i ) are the coordinates of the control points in the reference image; is the coordinate of the control point in the image to be registered; Δx is the horizontal displacement, the unit is pixel; Δy is the vertical displacement, the unit is pixel; θ is the rotation angle, the unit is degree.

The content of the present invention is limited to the registration of visible light images and infrared images under rigid body transformation, and other transformation models are not within the spirit and principle of the present invention.

Table 1

Table 1 is the method of the present invention and literature Li H, Manjunath B S, Mitra S K.A Contour-basedApproach to Multisensor Image Registration [J]. IEEE Transactions on Imaging Processing, 1995, 4 (3): 320-334. and manual registration method Experimental comparison results. Experimental results show that the parameter estimation of the method of the present invention is closest to the actual value of the parameter. Besides, the RMSE of the method of the present invention is minimal. It can be seen that compared with the other two methods, the registration accuracy of the method of the present invention is the highest. The registration results of the above three images to be registered using the method of the present invention are shown in FIGS. 9-11 . Besides, the present invention has no requirement on the number of closed contour extractions of the input image.

Although the specific implementation of the present invention has been described above in conjunction with the accompanying drawings, it does not limit the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, those skilled in the art do not need to pay creative work Various modifications or variations that can be made are still within the protection scope of the present invention.

Claims (5)

1. A method for automatically registering visible light and infrared images based on contour polygon fitting is characterized by comprising the following steps:

1) respectively extracting main outlines of the reference image and the image to be registered;

extracting a main contour, dividing the main contour into an edge detection part and a contour tracking part, carrying out edge detection by adopting a Canny operator which is best in noise suppression and edge positioning compromise, and tracking all contours in a counterclockwise direction;

2) performing polygon fitting on the main profile extracted in the step 1) by adopting an iterative endpoint fitting algorithm, simplifying the complex profile on the premise of keeping the profile characteristics, and removing redundant points and noise in the profile;

3) matching the fitted contours and selecting control points; in order to obtain the control points, matching feature points on the matching contour need to be selected, and polygon vertexes are selected as the feature points;

4) selecting an image transformation model, and estimating image transformation parameters according to the control points;

5) resampling the image to be registered and carrying out interpolation operation;

and (4) calculating coordinates of each pixel in the registration result image one by one according to the transformation parameters obtained in the step 4), and obtaining the gray value of each pixel in the registration result image by using an interpolation algorithm.

2. The method for automatic registration of visible and infrared images based on contour polygon fitting according to claim 1, characterized in that: the extraction quantity of the closed contours of the input images in the step 1) is not required, and the automatic registration of the visible light images and the infrared images under rigid body transformation can be realized under the condition of obtaining the clear main contours of the images.

3. The method for automatic registration of visible and infrared images based on contour polygon fitting according to claim 1, characterized in that: performing polygon fitting on the extracted main profile by adopting an iterative endpoint fitting algorithm in the step 2), wherein the method for fitting one profile mainly comprises the following steps:

(1) setting a distance threshold T;

(2) selecting a starting point A and an end point B of a contour line as two end points of a fitting polygon;

(3) calculating the distance from all points A, B on the contour line AB to the connecting line A, B, selecting the point C with the maximum distance, and setting the maximum distance value as H;

(4) comparing H and T, if H > T, indicating that C is an end point of the fitting polygon, and continuing to the step (5); if H is less than T, jumping out of the algorithm, and showing that no end point exists on the section of contour line;

(5) the end point C divides the contour line AB into an AC part and a BC part, and the end points on the contour lines of the two parts are respectively found out according to the steps (2), (3), (4) and (5); all the endpoints on the curve AB are found out (A, B, C, D … …), and are connected in sequence to obtain the final fitting polygon, and the endpoint A, B, C, D … … is the polygon vertex.

4. The method for automatic registration of visible and infrared images based on contour polygon fitting according to claim 1, characterized in that: the outline matching in the step 3) adopts a chain code matching mode;

firstly, according to the coding mode of the Freeman chain code, representing each fitted outline by the Freeman chain code; setting a reference image and an image to be registered as a first image and a second image respectively, and assuming a chain code { a }_iRepresents a length N in the first image_AProfile A of (1), chain code { b }_iRepresents a length N in the second image_BThe profile B of (A); respectively intercepting a contour segment with the length of n by taking the (k +1) th pixel on A as a starting point and the (l +1) th pixel on B as a starting point; wherein, the chain code value corresponding to the (k +1) th pixel on A is a_kAnd the (l +1) th pixel on B corresponds to the chain code value B_l(ii) a The matching degree of the two contour segments is defined as:

in the formula,wherein i, j ∈ [0, n-1 ]]，D_kl ⁿDenotes the degree of match, a, of A, B two contour segments of length n_k+jDenotes the chain code value of the (k + j +1) th pixel on A, b_l+jA chain code value representing the (l + j +1) th pixel on B; setting a matching degree threshold T_DWhen is coming into contact withDescription interceptionThe two profile segments of (a) are matched;

the feature of each contour is a set of several sections of small contour features taking a feature point as a center, so that a contour section near the feature point can be selected as a matching unit to search for a matching contour section in another image; and (3) taking the polygon vertex in the step 2) as the center, selecting the same number of pixels from the front and the back of the contour to form a feature contour segment, and searching a matched contour segment in the other image through contour matching degree calculation, wherein the corresponding feature point is the matched feature point.

5. The method for automatic registration of visible and infrared images based on contour polygon fitting according to claim 1, characterized in that: in the step 4), the forward transformation from the pixel points of the image to be registered to the pixel points of the registration result image cannot ensure that each pixel point of the image to be matched has a corresponding pixel point in the registration result image, namely, an unassigned pixel point may appear in the registration result image; in order to avoid the situation, inverse transformation is adopted, namely, for each pixel point in the reference image, the pixel point corresponding to the pixel point in the image to be registered is solved reversely;

the rigid body transformation model is:

wherein,for the coordinates of the control points in the image to be registered,coordinates of control points in the reference image; Δ x is the horizontal displacement in pixels; Δ y is the vertical displacement in pixels; theta is a rotation angle in degrees;

defining a transformation parameter matrix:controlling a point matrix of the image to be matched:reference picture control point matrix:wherein m is the logarithm of matched control points, m > 2, andand andandthe control points are matched control point pairs;

estimating a transformation parameter matrix by using a cutting least square algorithm, wherein the algorithm mainly comprises the following steps:

(1) assuming that M pairs of control points form a set P, calculating the least square solution of the transformation matrix M of all the points in P as:

(2) obtaining an estimated value by using the transformation matrix M and the matrix X obtained in (1)Calculating an estimateAnd the actual valueError between the estimated value and the actual value of each control point:

wherein,for the actual value of the control point i in the image to be registered,an estimated value of a control point i in the image to be registered, i ∈ {1,2, …, m };

(3) the Error is maximized Error_maxDeleting the corresponding matching control point pair, and updating the set P'; then, calculating a new transformation matrix M 'by using the updated P' and a least square method;

(4) setting an error threshold T_EContinuously repeating the steps (2) and (3); up to Error_max＜T_EObtaining a final transformation matrix;

under the condition that the number of the outliers is not more than half, the least square algorithm of cutting has stronger fault-tolerant capability, can continuously remove the outliers, and correctly estimates the transformation parameters.