CN1545064A - Infrared and visible light image fusion method - Google Patents

Abstract

The invention relates to a fusion method of infrared and visible light images. On the basis of multi-resolution decomposition of infrared images and visible light images, the relative "target" is obtained from the original image by using the different imaging characteristics of infrared images and visible light images. Information and "background" information, in order to divide the image into background area, target area and the edge part between them, three different fusion rules are used for these three parts to determine the multi-resolution representation of the fused image, and finally through The multi-resolution inverse transform is used to obtain the fused image. The invention adopts the method based on the target area to fuse the infrared and visible light images of the same scene, while retaining the target information to the greatest extent, it enhances the edge and detail information in the background information, and the quality of the fused image is greatly improved.

Description

Infrared and visible light image fusion method

Technical field:

The invention relates to an image fusion method based on a target area, which is used to fuse infrared and visible light images of the same scene, and can be widely used in optical imaging, target monitoring, security inspection and other systems.

Background technique:

Image fusion technology is the fusion of visual information in multi-sensor information fusion. It uses different imaging methods of various imaging sensors to provide complementary information for different images, increase the amount of image information, reduce the amount of original image data, and improve adaptability to the environment. In order to obtain more reliable and accurate useful information for observation or further processing. Image fusion technology is an emerging technology that integrates sensors, signal processing, image processing and artificial intelligence. In recent years, it has become a very important and useful image analysis and computer vision technology. , robotics, medical image processing, and military applications have broad application prospects.

Generally speaking, due to the difference in the infrared radiation characteristics of the target and the background, the infrared image can provide relatively complete target information, but the background information it contains is blurred; on the contrary, the visible light image can provide more comprehensive background information, but Target information is not obvious. Through image fusion technology, accurate and comprehensive fused images with target and background information can be obtained. In the processing of infrared and visible light image fusion methods, the representative method is the multi-resolution image fusion method. The basic idea is to decompose the input original image to obtain different resolution representations, and then fuse them respectively based on fusion rules. After calculation, the fused image is obtained through multi-resolution reconstruction. Multi-resolution image fusion methods include Laplacian pyramid algorithm, contrast pyramid algorithm, gradient pyramid algorithm and other pyramid methods. With the development and in-depth study of wavelet theory, discrete wavelet transform has become another very useful tool in multi-resolution image fusion. Fusion methods based on discrete wavelet transform and wavelet frame have been widely used in the field of fusion.

For the previous fusion algorithm research, most of the algorithms did not consider the use of information such as the imaging characteristics of the original image to improve the performance of the algorithm when performing fusion operations. There is no targeted fusion operation for the different information contained in the original image.

Invention content:

The purpose of the present invention is to propose a fusion method of infrared and visible light images aiming at the deficiencies in the prior art, which can improve the quality of the fusion image and achieve a more ideal fusion effect.

In order to achieve such a purpose, the method of the present invention, on the basis of multi-resolution decomposition of the infrared image and the visible light image, uses the different imaging characteristics of the infrared image and the visible light image to obtain the "target" information and the "background" information from the original image. "Information (relatively speaking) divides the image into three parts: the background area, the target area, and the edge part between them. For these three parts, three different fusion rules are used to determine the multi-resolution of the fused image. Indicates that the fused image is finally obtained through multi-resolution inverse transformation.

Method of the present invention comprises following specific steps:

1. On the basis of the registration of the infrared image and the visible light image of the same scene, the àtrous (porous) wavelet algorithm is used to decompose the image, and the two-dimensional convolution operator obtained by the B ₃ spline scaling function is used for specific operations, and the image Decomposed into detailed information on different frequency bands and approximate information on the lowest frequency band.

2. Use the threshold method to segment the infrared image, and then determine the target area according to the imaging characteristics and target image features (including gray information, texture information, contour information, etc.). Generally speaking, the target appears in the infrared image with relatively high pixel gray value. After determining the target area of the infrared image, the original image is divided into three parts: the background area, the target area and the edge part between them.

3. During fusion processing, for the target area, directly select the wavelet coefficient of the corresponding area of the infrared image as the fused wavelet coefficient; for the edge part between the target area and the background area, compare the absolute value of the wavelet coefficient of the infrared image and the wavelet coefficient of the visible light image Size, select the wavelet coefficient with a large absolute value as the wavelet coefficient after fusion; for the background area, consider the wavelet coefficient itself and its correlation with the wavelet coefficient in the neighborhood at the same time, determine the fusion measurement index, and measure the wavelet coefficient by the size of the fusion measurement index How much information is included. By comparing the size of the fusion measurement index of the infrared image and the visible light image, it is determined that the wavelet coefficient with the largest index is the wavelet coefficient after fusion.

4. Perform wavelet inverse transform on the wavelet coefficients obtained in each area to obtain the fused image.

Method of the present invention has following beneficial effect:

When the àtrous (porous) wavelet algorithm is used to decompose the image, due to its translation invariance, it can reduce the wrong selection of fusion coefficients and the influence of registration errors on the fusion results during fusion; in the process of àtrous wavelet transformation, the obtained wavelet The surfaces have the same size, so it is easier to find the corresponding relationship between the coefficients of each wavelet surface, which is beneficial to the fusion operation; since the àtrous wavelet algorithm does not involve convolution operations during reconstruction, it is useful for region-based fusion operations. It is beneficial to reduce the influence on the edge part between regions. Using the fusion rules based on the target area, the edge and detail information in the background information can be enhanced while maintaining the target information to the greatest extent. The image fusion method of the invention greatly improves the image quality after fusion, and has great significance and practical value for the subsequent processing and image display of the application system.

Description of drawings:

FIG. 1 is a schematic diagram of an infrared and visible light image fusion method based on a target area in the present invention.

As shown in the figure, the wavelet transform (àtrous porous wavelet algorithm) is performed on the infrared image and the visible light image respectively to obtain multiple wavelet coefficient representations with different resolutions. Through the "target" information and "background" information obtained from the original image ( Relatively speaking) to guide the fusion decision to obtain fused representations of wavelet coefficients at different resolutions. Finally, inverse wavelet transform is performed on the fused wavelet coefficients to obtain the fused image.

Fig. 2 is a flow chart of fusion decision-making in the present invention.

As shown in the figure, on the basis of regional segmentation of the infrared image, after the target area is determined through the imaging characteristics of the image, the infrared image and the visible light image are divided into three parts: the background area, the target area and the edge between them part. Different fusion rules are implemented on these three regions to obtain a fusion decision map.

Fig. 3 is the original infrared and visible light images of the present invention and the comparison with the fusion results based on the pixel fusion method and the window fusion method.

Among them, (a) is the original visible light image; (b) is the original infrared image; (c) is the fusion result based on the pixel fusion method; (d), (e) is the fusion result of two window fusion methods; (f ) is the fusion result based on the target region fusion method of the present invention.

Detailed ways:

In order to better understand the technical solutions of the present invention, the implementation manners of the present invention will be further described below in conjunction with the accompanying drawings.

The process flow of the method of the present invention is shown in Figure 1. First, the infrared image and the visible light image are respectively subjected to wavelet transformation to obtain a plurality of wavelet coefficient representations with different resolutions, and the "target" information and "background" information obtained from the original image ( Relatively speaking, the background information in the visible light image is richer, and the target information in the infrared image is richer) to guide the fusion decision to obtain the fusion of wavelet coefficients with different resolutions. Finally, inverse wavelet transform is performed on the fused wavelet coefficients to obtain the fused image.

The fusion decision-making process of the present invention is shown in Figure 2. On the basis of regional segmentation of the infrared image, after the target area is determined through the imaging characteristics of the image, the infrared image and the visible light image are divided into three parts: background area, target area regions and the margins between them. Different fusion rules are implemented on these three regions to obtain a fusion decision map.

The concrete implementation of the present invention carries out as follows:

1. On the basis of the registration of the infrared image and the visible light image of the same scene, the àtrous (porous) wavelet algorithm is used to decompose the original visible light image and the infrared image respectively, and the image is decomposed into detailed information on different frequency bands and the approximation of the lowest frequency band information.

The basic idea of the àtrous wavelet algorithm is to decompose a signal or image into detailed information on different frequency bands and approximate information on the lowest frequency band. This detailed information is called a wavelet surface, and its size is the same as that of the original image. For the image f(x, y), the following image sequence can be obtained step by step

Among them, f _k (x, y) is the approximate image at scale k, and L _k is the low-pass filter. k=1, 2, . . . , N.

The difference between approximate images of adjacent scales constitutes the coefficients of the wavelet transform, that is, the wavelet surface

ω _k (x, y) = f _k (x, y) - f _k-1 (x, y) k = 1, 2, . . . , N (2)

Using the _B3 -spline scaling function, the obtained two-dimensional convolution operator is shown below.

$\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 256</span>}}\left[\begin{array}{ccccc}\mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 4</span>}& \mathrm{<span\; class="notranslate">\; 6</span>}& \mathrm{<span\; class="notranslate">\; 4</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 4</span>}& \mathrm{<span\; class="notranslate">\; 16</span>}& \mathrm{<span\; class="notranslate">\; twenty\; four</span>}& \mathrm{<span\; class="notranslate">\; 16</span>}& \mathrm{<span\; class="notranslate">\; 4</span>}\\ \mathrm{<span\; class="notranslate">\; 6</span>}& \mathrm{<span\; class="notranslate">\; twenty\; four</span>}& \mathrm{<span\; class="notranslate">\; 36</span>}& \mathrm{<span\; class="notranslate">\; twenty\; four</span>}& \mathrm{<span\; class="notranslate">\; 6</span>}\\ \mathrm{<span\; class="notranslate">\; 4</span>}& \mathrm{<span\; class="notranslate">\; 16</span>}& \mathrm{<span\; class="notranslate">\; twenty\; four</span>}& \mathrm{<span\; class="notranslate">\; 16</span>}& \mathrm{<span\; class="notranslate">\; 4</span>}\\ \mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 4</span>}& \mathrm{<span\; class="notranslate">\; 6</span>}& \mathrm{<span\; class="notranslate">\; 4</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

2. Use the threshold method to segment the infrared image, and then determine the target area according to the imaging characteristics and target image features (including gray information, texture information, contour information, etc.).

For the target, it appears in the infrared image that the gray value of the pixel is relatively high, and the target area is determined according to the difference in the gray value of the pixel in the image and the motion information of the target. Of course, in the actual division, it is not excluded that other non-target areas are classified as target areas. However, because it has some characteristics of the target area, it is still regarded as a target area if it cannot be determined whether it is a target area. The target area is determined. Here, the selection of the target area takes precedence over the selection of any other area.

After determining the target area of the target image, the infrared image and the visible light image can be divided into three parts: the background area, the target area and the edge part between them.

3. Use different image fusion rules to perform fusion operations on the three parts respectively. Let the infrared image be A, and the visible light image be B; X represents the target area, X represents the background area, and E represents the edge between the two areas. The specific fusion rules are as follows:

● Fusion of "high frequency" coefficients

(1) Target area; generally speaking, the pixels in the target area have basically the same gray value or texture information, what needs to be done is not to compare the size of the "high frequency", the purpose is to keep the target information of the area to the greatest extent. Therefore, the fusion coefficient of this area is selected as:

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}}^{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

(2) The edge part between the two regions; for this region, the position of the coefficient is known. In order to better express the edge information, the fusion coefficient of this part is selected as:

(3) Background area; compared to the target area, the background area contains rich edge and detail information. In order to better express the detail information of the area, and consider that some detail information of the target area has not been extracted into the target area , the present invention uses a new fusion measurement index to determine fusion coefficients to highlight detailed information. This fused metric considers both the coefficient itself and its correlation with coefficients in the neighborhood:

PI _X (m, n) = C _p (m, n)·I _t (m, n) (6)

Among them, PI _X (m, n) measures the information contained in the coefficient; C _p (m, n) reflects the characteristic information of the coefficient; I _t (m, n) reflects the correlation of coefficients in the neighborhood.

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

Wherein, C _X ^H (m, n) is the high-frequency coefficient of image;

For the correlation of coefficients in the neighborhood, first determine the sign value of the high-frequency coefficients of the image

$\mathrm{<span\; class="notranslate">\; sign</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 8</span>}<span\; class="notranslate">\; )</span>$

If the value of the high-frequency coefficient is greater than or equal to zero, then sign is 1; otherwise sign is 0. For I _t (m, n), it can be expressed as:

I _t (m, n) = p _X (m, n) · (1-p _X (m, n)) (9)

Wherein, p _X (m, n) is the probability value that the coefficient of point (m, n) has the same sign value as the coefficient in the neighborhood. If the coefficients of (m, n) points are the same as those in the neighborhood, p _X =1; if they are different from the coefficients in the neighborhood, p _X =0.

The fusion coefficient of this part is selected as:

● Fusion of low frequency coefficients

4. Perform wavelet inverse transform on the "high-frequency" fusion coefficients and low-frequency fusion coefficients obtained after fusion to obtain a fused image.

The reconstruction of the image is:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; )</span>$

The fusion results obtained by the present invention are compared with the fusion results obtained by other fusion methods, and the evaluation results are compared as shown in Table 1, and Fig. 3 is the image obtained by fusion. In Figure 3, (a) is the original visible light image; (b) is the original infrared image; (c) is the fusion result based on the pixel fusion method; (d), (e) are the fusion results of two window fusion methods; (f) is the fusion result based on the target region fusion method of the present invention. The results show that the present invention greatly improves the quality of the fused image, which is better than other fused methods.

Table 1 Index evaluation of image fusion results

                                                                                              based

Mutual information 3.5953 1.4415 1.4468 1.4479

Edge information 0.3228 0.3091 0.3106 0.3193

Claims (1)

1. An infrared and visible light image fusion method is characterized in that comprising the following specific steps: 1) On the basis of the registration of the infrared image and the visible light image of the same scene, the àtrous porous wavelet algorithm is used to decompose the image, and the two-dimensional convolution operator obtained by the _B3- spline scaling function is used for specific operations, and the image is decomposed into Detailed information on different frequency bands and approximate information on the lowest frequency band; 2) The threshold method is used to segment the infrared image, and then the target area is determined according to the imaging characteristics and feature information of the target image. After the target area of the infrared image is determined, the original image is divided into the background area, the target area and the area between them. the edge part; 3) During the fusion process, for the target area, directly select the wavelet coefficient of the corresponding area of the infrared image as the fused wavelet coefficient; for the edge part between the target area and the background area, compare the absolute value of the wavelet coefficient of the infrared image and the wavelet coefficient of the visible light image Size, select the wavelet coefficient with a large absolute value as the wavelet coefficient after fusion; for the background area, consider the coefficient itself and its correlation with the coefficients in the neighborhood at the same time, determine the fusion measurement index, the size of the fusion measurement index measures the wavelet coefficient contained The amount of information, by comparing the size of the infrared image and visible light image fusion measurement index, determine the wavelet coefficient with the largest index as the wavelet coefficient after fusion; 4) Perform wavelet inverse transform on the wavelet coefficients obtained in each area to obtain the fused image.

Images