CN104751432A - Image reconstruction based visible light and infrared image fusion method - Google Patents

Abstract

The invention discloses an image reconstruction based visible light and infrared image fusion method and belongs to the field of image processing. The image reconstruction based visible light and infrared image fusion method sequentially comprises the steps of calculating the gradient of a registered infrared image and the gradient of a visible light brightness image, estimating noise strength of the brightness image, calculating the weighted sum of the gradient of the brightness image, obtaining a reconstructed image, performing image normalization and reconstruction, combining the reconstructed brightness image and an original visible light color component, and the problem of remarkable difference of image boundaries in the existing image splicing and fusion process is solved. The image reconstruction based visible light and infrared image fusion method can be applied to fusion processing of multiple registered images.

Description

A Fusion Method of Visible Light and Infrared Images Based on Image Reconstruction

technical field

The invention belongs to the field of image processing, and in particular relates to a fusion method of visible light and infrared images based on image reconstruction.

Background technique

Image fusion refers to the comprehensive processing of image information detected by multiple sensors to achieve a more comprehensive and reliable description of the detection scene. By integrating the complementary information of different image information sources, image fusion can not only overcome the limitations and differences of single sensor images in terms of geometry, spectrum and spatial resolution, but also remove the redundancy of multi-source image information. The efficiency of image understanding and recognition improves image quality at the same time, which is conducive to the positioning, identification and interpretation of physical phenomena and events. Image fusion is generally divided into three levels: pixel-level fusion, feature-level fusion and decision-level fusion. The present invention belongs to pixel-level fusion. The input and output data of pixel-level fusion processing are based on image data, and its processing goal is to extract the interesting information in the image to be fused and integrate it into the fused image.

With the development of multi-scale analysis tools, represented by DWT and a series of improved schemes such as non-subsampled contourlet (NSCT), multi-scale analysis tools have been widely used in the field of image fusion [Kong W, Zhang L, Lei Y.Novel fusionmethod for visible light and infrared images based on NSST–SF–PCNN[J].Infrared Physics&Technology,2014,65:103-112.]. Fusion rules are another crucial factor in fusion algorithms based on multi-scale analysis. Fusion rules can generally be divided into three categories: pixel-based fusion rules, window-based fusion rules, and region-based fusion rules [Ye Chuanqi. Based on Research on Multi-Scale Decomposition Multi-Sensor Image Fusion Algorithm [D].[Doctoral Dissertation]. Xidian University, 2009]. Fusion images based on pixel-based fusion rules such as traditional weighted average have low contrast; fusion rules based on windows, such as fusion algorithms based on statistical characteristics of window regions [Zhang Qiang, Guo Baolong. A non-sampled Contourlet transform infrared image and visible light image Fusion algorithm [J]. Journal of Infrared and Millimeter Waves, 2007, 06:476-480.], which considers the correlation between adjacent pixels, improves the fusion effect to a certain extent; Multiple pixels in a region participate in the fusion process as a whole, such as the image fusion algorithm based on region segmentation [Liu Kun, Guo Lei, Li Hui-hua, et al. Fusion of infrared and visible light images based on region segmentation[ J].Chinese Journal of Aeronautics,2009,22(1):75-80], the overall visual effect of the fusion image is better, and the fusion trace can be better suppressed. However, when the above image fusion method performs fusion processing on images of different sizes, or when there is a significant gradient change in the image, there are obvious fusion and splicing traces in the fusion image. The use of boundary filtering can reduce the fusion trace, but it also reduces the sharpness of the image. The invention adopts an image reconstruction method, has few adjustment parameters, and can realize image fusion without any traces without filtering the fusion image.

Contents of the invention

Aiming at the deficiencies in the prior art, the present invention provides a fusion method of visible light and infrared images based on image reconstruction, which solves the problem of fusion traces in the fusion image after splicing and fusion of the existing images. The process flow is shown in Figure 1 It mainly includes the following steps:

Step 01. Read m infrared images IR _i (x, y) and n visible light images IM _j (x, y) that need to be fused, where i=1...m, j=1...n; Transform a visible light image IM _j (x, y) into YUV space, and the three components of YUV space are {IV _j , Cb _j , Cr _j }, where IV _j is the brightness component, Cb _j and Cr _j are the chrominance components ;

Step 02. Perform operations from step 03 to step 06 for each infrared image and visible light image;

Step 03. Approximately obtain the gradient image GR _i (x, y) of the infrared image IR _i (x, y) and the gradient of the brightness component IV _j (x, y) corresponding to the visible light image by first-order forward or backward difference Image GV _j (x,y); the following formula is approximated by the first-order forward difference:

GR _i (x,y)＝▽IR _i (x,y)≈(IR _i (x+1,y)-IR _i (x,y),IR _i (x,y+1)-IR _i (x ,y));

GV _j (x,y)＝▽IV _j (x,y)≈(IV _j (x+1,y)-IV _j (x,y),IV _j (x,y+1)-IV _j (x ,y));

Step 04. Perform mean filtering on the gradient image IR _i (x, y) to obtain the mean image IRM _i (x, y), and perform mean filtering on the gradient image IV _j (x, y) to obtain the mean image IVM _j (x, y) ;

Step 05. Subtract the mean image IRM _i (x, y) from the gradient image IR _i (x, y) to obtain the error image IRE _i (x, y), and subtract the mean from the gradient image IV _j (x, y) The image IVM _j (x, y) gets the error image IVE _j (x, y);

Step 06. Calculate the noise standard deviation of the error images IRE _i (x, y) and IVE _j (x, y) respectively;

The noise standard deviation of the error image IRE _i (x, y) is specifically obtained by the following method:

(06-1) The standard deviation σ _R0,i of the statistical error image IRE _i (x,y);

(06-2) Delete the error points in the error image IRE _i (x, y) that are distributed outside the triple variance 3σ _R0,i ;

(06-3) Repeat steps (06-1) to (06-2) to perform iterative operations until the relative error of the standard deviation obtained by two adjacent iterative operations is less than 10%, that is, the result of the p+1th iterative operation When the relative error of the standard deviation σ _Rp+1,i relative to the standard deviation σ _Rp+1,i obtained in the pth iteration is less than 10%, record the standard deviation σ _Rp+1 obtained in the p+1th iteration _,i is the noise standard deviation σ _R,i of the error image IRE _i (x,y);

The calculation method of the noise standard deviation σ _V,j of the error image IVE _j (x,y) is the same as that of the noise standard deviation σ _R,i ;

Step 07. Calculate the weighting coefficients μ _R,i and μ _V,j :

μ _R,i =σ _V,i /(σ _Rs +σ _Vs ), μ _V,j =σ _R,j /(σ _Rs +σ _Vs )

Among them: σ _Rs = σ _R,1 +σ _R,2 +...+σ _R,m ; σ _Vs =σ _V,1 +σ _V,2 +...+σ _V,n ;

Step 08. Obtain the gradient image weighted sum G(x, y)=[Gx(x, y), Gy(x, y)], which is specifically obtained in the following manner:

Gx(x,y)＝μ _R,1 *GR _1,x (x,y)*[sign[|GR _1,x (x,y)|-σ _R,1 ]+1]/2

+μ _R,2 *GR _2,x (x,y)*[sign[|GR _2,x (x,y)|-σ _R,2 ]+1]/2

...

+μ _R,m *GR _m,x (x,y)*[sign[|GR _m,x (x,y)|-σ _R,m ]+1]/2

+μ _V,1 *GV _1,x (x,y)*[sign[|GV _1,x (x,y)|-σ _V,1 ]+1]/2

+μ _V,2 *GV _2,x (x,y)*[sign[|GV _2,x (x,y)|-σ _V,2 ]+1]/2

...

+μ _V,n *GV _n,x (x,y)*[sign[|GV _m,x (x,y)|-σ _V,m ]+1]/2;

Gy(x,y)＝μ _R,1 *GR _1,y (x,y)*[sign[|GR _1,y (x,y)|-σ _R,1 ]+1]/2

+μ _R,2 *GR _2,y (x,y)*[sign[|GR _2,y (x,y)|-σ _R,2 ]+1]/2

...

+μ _R,m *GR _m,y (x,y)*[sign[|GR _m,y (x,y)|-σ _R,m ]+1]/2

+μ _V,1 *GV _1,y (x,y)*[sign[|GV _1,y (x,y)|-σ _V,1 ]+1]/2

+μ _V,2 *GV _2,y (x,y)*[sign[|GV _2,y (x,y)|-σ _V,2 ]+1]/2

...

+μ _V,n *GV _n,y (x,y)*[sign[|GV _m,y (x,y)|-σ _V,m ]+1]/2;

Among them, GR _i,x (x,y) and GR _i,y (x,y) are the x and y components of the gradient image GR _i (x,y) respectively, and GV _i,x (x,y) and GV _{i , y} (x, y) are respectively the x and y components of the gradient image GV _j (x, y), that is, GR _i (x, y) = [GR _{i, x} (x, y), GR _{i, y} (x , y)], GV _j (x, y) = [GV _{i, x} (x, y), GV _{i, y} (x, y)];

Step 09. Solve the Poisson equation ▽ ² I _Re (x, y) = div (G (x, y)), and obtain the reconstructed brightness image I _Re (x, y);

Step 10. Read the pixel maximum value I _maxRe and the pixel minimum value I _minRe of the image I _Re (x, y), and normalize the image I _Re (x, y): I _Reunify (x, y)=(I _Re (x,y)-I _minRe )/(I _maxRe -I _minRe );

Step 11. Randomly select an image from the n visible light images, and combine the chromaticity components Cb and Cr of the visible light image with the reconstructed normalized brightness image I _Reunify (x, y) to obtain the fused Image {I _Reunify , Cb, Cr}.

The beneficial effects of the present invention are:

By adopting the image fusion method of the present invention, the problem of fusion traces existing in fusion images obtained by existing image splicing and fusion algorithms is solved. At the same time, during the fusion process, the noise in flat areas is suppressed.

Description of drawings

Fig. 1 is the flow chart of the visible light and infrared image fusion method based on image reconstruction provided by the present invention;

Fig. 2 is the RGB space visible light image that is used for embodiment, and image width is 1024, and height is 1024, and its field of view is one-third of infrared image, and is positioned at the middle of infrared image;

Fig. 3 is the infrared image used in the embodiment, the image width is 320, the height is 240, and its field of view is three times that of the visible light image;

Fig. 4 is the image after merging Fig. 2 and Fig. 3 by using the traditional multi-scale image fusion method;

Fig. 5 is a fused image using the image merging method provided by the present invention in the embodiment.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

Example

The purpose of this embodiment is to fuse an infrared image with a visible light image, which specifically includes the following steps:

Step 01. Read the visible light image IM(x,y), as shown in Figure 2; transform the visible light image into YUV space, and the three components are {IV,Cb,Cr}, where IV(x,y) is the brightness Components, Cb and Cr are chrominance components, the image size is 1024×1024, the field of view is one-third of the field of view of the infrared image, and is located in the very center of the infrared image; read the infrared image IR(x,y), such as As shown in Figure 3, the image size is 320×240, and the field of view is three times that of the visible light image;

Step 02. According to the field of view ratio of the visible light image and the infrared image, complete the pixel scaling conversion between the visible light image and the infrared image: linearly compress the visible light image to 1/3 of its image in both horizontal and vertical directions, and the compressed size is 341×341, The center of the image coincides with the center of the infrared image, and the floating-point coordinates are rounded to integer grid coordinates;

Step 03. Use the first-order forward difference to obtain the gradient images GR(x,y) and GV(x,y) of the infrared image IR(x,y) and the brightness component image IV(x,y):

GR(x,y)=▽IR(x,y)≈(IR(x+1,y)-IR(x,y),IR(x,y+1)-IR(x,y)),

GV(x,y)=▽IV(x,y)≈(IV(x+1,y)-IV(x,y),IV(x,y+1)-IV(x,y));

Step 04. Perform mean filtering on the gradient image IR(x,y) to obtain the mean image IRM(x,y), and perform mean filtering on the gradient image IV(x,y) to obtain the mean image IVM(x,y);

Step 05. Subtract the mean image IRM(x,y) from the gradient image IR(x,y) to obtain the error image IRE(x,y), and subtract the mean image IVM(x ,y) to get the error image IVE(x,y);

Step 06. Calculate the standard deviation σ _R,i and σ _V,j of the error images IRE(x,y) and IVE(x,y) respectively;

Step 07. Calculate the weighting coefficients μ _R,i and μ _V,j ;

Step 08. Obtain the gradient image weighted sum G(x, y)=[Gx(x, y), Gy(x, y)];

Step 09. Solve the Poisson equation ▽ ² I _Re (x, y) = div (G (x, y)), and obtain the reconstructed brightness image I _Re (x, y);

Step 10. Read the maximum pixel value I _maxRe and the minimum pixel value I _minRe of the image I _Re (x, y), and perform a normalization operation on the image I _Re (x, y): I _Reunify (x, y)=( I _Re (x,y)-I _minRe )/(I _maxRe -I _minRe );

Step 11. Combine the reconstructed normalized brightness image I _Reunify (x, y) with the color components Cb and Cr of the original visible light image to obtain the fused image {I _Reunify , Cb, Cr}, as shown in Figure 5, The image output size is 320×240 color image.

Fig. 4 is the fused image after merging Fig. 2 and Fig. 3 by adopting the traditional multi-scale image fusion method; as can be seen from comparison, the fused image obtained by adopting the method provided by the present invention in this embodiment case can be better than other methods The boundary of the fusion image is processed, and the boundary of the fusion image is natural and smooth.

Claims (3)

1., based on visible ray and the infrared image fusion method of Image Reconstruction, specifically comprise the following steps:

Step 01. reads the m width infrared image IR needing to carry out merging _i(x, y) and n width visible images IM _j(x, y), wherein, i=1 ... m, j=1 ... n; By each width visible images IM _j(x, y) is converted into yuv space, and three components of yuv space are respectively { IV _j, Cb _j, Cr _j, wherein IV _jluminance component, Cb _jand Cr _jit is chromatic component;

Step 02. operates to step 06 by step 03 each width infrared image and visible images;

Step 03. is similar to single order forward direction or backward difference and obtains infrared image IR _ithe gradient image GR of (x, y) _ithe luminance component IV that (x, y) and visible images are corresponding _jthe gradient image GV of (x, y) _j(x, y);

Step 04. couple gradient image IR _i(x, y) carries out mean filter and obtains average image IRM _i(x, y), to gradient image IV _j(x, y) carries out mean filter and obtains average image IVM _j(x, y);

Step 05. is from gradient image IR _iaverage image IRM is deducted in (x, y) _i(x, y) obtains error image IRE _i(x, y), from gradient image IV _javerage image IVM is deducted in (x, y) _j(x, y) obtains error image IVE _j(x, y);

Step 06. is error of calculation image IRE respectively _i(x, y) and IVE _jthe noise criteria of (x, y) is poor;

Error image IRE _ithe noise criteria difference of (x, y) obtains especially by following methods:

(06-1) statistical error image IRE _ithe standard deviation sigma of (x, y) _{r0, i};

(06-2) error image IRE is deleted _ithree times of variance 3 σ are distributed in (x, y) _{r0, i}outside error point;

(06-3) repeat step (06-1) and carry out interative computation to (06-2), until stop when the relative error of adjacent twice interative computation gained standard deviation is less than 10%, i.e. the standard deviation sigma of the p+1 time interative computation gained _{rp+1, i}relative to the standard deviation sigma of the p time iteration acquisition _{rp+1, i}relative error when being less than 10% only, remember the standard deviation sigma of the p+1 time interative computation gained _{rp+1, i}for error image IRE _ithe noise criteria difference σ of (x, y) _r,i;

Error image IVE _jthe noise criteria difference σ of (x, y) _v,jcomputing method and noise criteria difference σ _r,icomputing method identical;

Step 07. calculates weighting coefficient μ _r,iand μ _v,j:

μ _R,i＝σ _V,i/(σ _Rs+σ _Vs)，μ _V,j＝σ _R,j/(σ _Rs+σ _Vs)

Wherein: σ _rs=σ _{r, 1}+ σ _{r, 2}++ σ _r,m; σ _vs=σ _{v, 1}+ σ _{v, 2}++ σ _v,n;

Step 08. obtains gradient image weighted sum G (x, y)=[Gx (x, y), Gy (x, y)];

Step 09. solves Poisson equation ▽ ²i _re(x, y)=div (G (x, y)), obtains the luminance picture I of reconstruct _re(x, y);

Step 10. reading images I _rethe pixel maximal value I of (x, y) _maxRewith pixel minimum I _minRe, to image I _re(x, y) normalization: I _reunify(x, y)=(I _re(x, y)-I _minRe)/(I _maxRe-I _minRe);

Step 11. is random selecting piece image from described n width visible images, by the normalization luminance picture I of chromatic component Cb, Cr of this visible images and reconstruct _reunify(x, y) combines, and obtains the image { I after merging _reunify, Cb, Cr}.

2. the visible ray based on Image Reconstruction according to claim 1 and infrared image fusion method, is characterized in that, the GR of gradient image described in step 03 _i(x, y) and GV _j(x, y) specifically adopts and tries to achieve so that single order forward difference is approximate:

GR _i(x,y)＝▽IR _i(x,y)≈(IR _i(x+1,y)-IR _i(x,y),IR _i(x,y+1)-IR _i(x,y))，

GV _j(x,y)＝▽IV _j(x,y)≈(IV _j(x+1,y)-IV _j(x,y),IV _j(x,y+1)-IV _j(x,y))。

3. the visible ray based on Image Reconstruction according to claim 1 and infrared image fusion method, is characterized in that, described gradient image weighted sum G (x, y)=[Gx (x, y), Gy (x, y)] obtain especially by with under type:

Gx(x,y)＝μ _R,1*GR _1,x(x,y)*[sign[|GR _1,x(x,y)|-σ _R,1]+1]/2

+μ _R,2*GR _2,x(x,y)*[sign[|GR _2,x(x,y)|-σ _R,2]+1]/2

······

+μ _R,m*GR _m,x(x,y)*[sign[|GR _m,x(x,y)|-σ _R,m]+1]/2

+μ _V,1*GV _1,x(x,y)*[sign[|GV _1,x(x,y)|-σ _V,1]+1]/2

+μ _V,2*GV _2,x(x,y)*[sign[|GV _2,x(x,y)|-σ _V,2]+1]/2

······

+μ _V,n*GV _n,x(x,y)*[sign[|GV _m,x(x,y)|-σ _V,m]+1]/2；

Gy(x,y)＝μ _R,1*GR _1,y(x,y)*[sign[|GR _1,y(x,y)|-σ _R,1]+1]/2

+μ _R,2*GR _2,y(x,y)*[sign[|GR _2,y(x,y)|-σ _R,2]+1]/2

······

+μ _R,m*GR _m,y(x,y)*[sign[|GR _m,y(x,y)|-σ _R,m]+1]/2

+μ _V,1*GV _1,y(x,y)*[sign[|GV _1,y(x,y)|-σ _V,1]+1]/2

+μ _V,2*GV _2,y(x,y)*[sign[|GV _2,y(x,y)|-σ _V,2]+1]/2

······

+μ _V,n*GV _n,y(x,y)*[sign[|GV _m,y(x,y)|-σ _V,m]+1]/2；

Wherein, GR _i,x(x, y) and GR _i,y(x, y) is gradient image GR respectively _ix, y component of (x, y), GV _i,x(x, y) and GV _i,y(x, y) is gradient image GV respectively _jx, y component of (x, y), i.e. GR _i(x, y)=[GR _i,x(x, y), GR _i,y(x, y)], GV _j(x, y)=[GV _i,x(x, y), GV _i,y(x, y)].