CN104881854B - High dynamic range images fusion method based on gradient and monochrome information - Google Patents

Abstract

The invention relates to digital image processing technology, which is a fast and effective multi-exposure image fusion method. This method does not require complex image decomposition and reconstruction process, and uses pixel weighting to complete the fusion, so that the image can display more detailed information and improve image quality. For this reason, the technical solution adopted by the present invention is that the high dynamic range image fusion method based on gradient and brightness information includes the following steps: 1) For each input multi-exposure image, calculate its local contrast weight factor L _i (x, y); 2) Convert the original image to the color space of hue H, color saturation S and brightness I, extract the brightness component I, and calculate its brightness weight factor H _i (x, y); 3) Weight estimation 4) Use recursive filtering The weight map is improved to obtain the final weight function of each input image: 5) Weighted fusion of multi-exposure images. The invention is mainly applied to digital image processing.

Description

High dynamic range image fusion method based on gradient and brightness information

technical field

The invention relates to digital image processing technology and high dynamic range image processing technology. Specifically, it relates to a high dynamic range image fusion method based on gradient and brightness information.

Background technique

Natural scenes have a very large dynamic range, and it is difficult for ordinary camera equipment to capture all brightness levels in the same scene. In scenes with strong light, the images captured by ordinary cameras are often under-exposed or over-exposed, so details will be lost in areas that are too dark or too bright in the image. In order to solve this problem, people usually take a series of images of the same scene with different exposures, and generate a high dynamic range (High Dynamic Range, HDR) image by processing the image sequence. High dynamic range images can provide more dynamic range and image details, and better reflect the visual effects in the real environment.

At present, some multi-exposure image fusion algorithms have been proposed. Goshtasby[1] proposed a multi-exposure image fusion technique for image segmentation. This method divides each multi-exposure image into a series of blocks of the same size, and uses the information entropy index to measure the quality of the image block to find the image block with the most ideal exposure corresponding to each region. Afterwards, the selected blocks are fused together using a monotonically decreasing fusion function. In order to make the fused image have the maximum amount of information, the optimal block size and the width of the fusion function are iteratively optimized by the gradient ascending algorithm. This algorithm is not efficient due to the large number of complex calculations required. In addition, this method cannot get a good fusion effect at the object boundary. Mertens et al. [2] used contrast, saturation, and exposure as fusion guidance indicators to generate weight maps in RGB space, and used Laplacian pyramids to complete image fusion. The algorithm first decomposes the multi-exposure image by Laplace and decomposes the weight map by Gaussian. After that, each layer image of the Laplacian pyramid is multiplied by each layer image of the corresponding Gaussian pyramid. Finally, the fused image can be generated by pyramid reconstruction. The result of this fusion algorithm is not clear in the dark area of the image, and there are more underexposed pixels. Moreover, as the number of pyramid decomposition layers increases and the size of the image increases, the running time of the algorithm will increase significantly. Wei Zhang et al. [3] used the gradient information of the input image for multi-exposure image fusion. The algorithm uses the gradient size and direction to construct the consistency evaluation factor and the visibility evaluation factor respectively, and then multiplies the two factors to obtain the weight map. The final fused image is obtained by combining the original image sequence with the weight map. Shen Jianbing et al. [4] proposed a novel enhanced Laplacian pyramid fusion algorithm. The algorithm measures image exposure quality based on a combination of local and global weight factors as weights, and the enhancement process is guided by this weight map. Images generated by this method have better color and texture information. However, this algorithm also uses the process of pyramid decomposition and reconstruction, and the calculation efficiency is not high.

It can be seen that the existing multi-exposure image fusion algorithm often needs to perform filtering or image decomposition and reconstruction in order to obtain a better fusion effect, and the calculation amount is relatively large. The result obtained by the simple algorithm of the fusion process is not ideal. Therefore, the present invention proposes a simple direct fusion method of multi-exposure images, which utilizes the gradient size and the brightness component to guide the weight map to complete the fusion process and expand the dynamic range of the image.

references

[1]Goshtasby A A.Fusion of multi-exposure images[J].Image and Vision Computing,2005,23(6):611-618.

[2]Mertens T, Kautz J, Van Reeth F. Exposure fusion[C]//Computer Graphics and Applications, 2007.PG'07.15th Pacific Conference on.IEEE, 2007:382-390.

[3] Zhang W, Cham W K. Gradient-directed composition of multi-exposure images [C]//Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010:530-536.

[4] Shen J, Zhao Y, Yan S, et al. Exposure fusion using boosting Laplacian pyramid [J]. IEEE Trans. Cybern, 2014, 44(9): 1579-1590.

Contents of the invention

To overcome the deficiencies in technology, the present invention aims to develop a fast and effective multi-exposure image fusion method. This method does not require complex image decomposition and reconstruction process, and uses pixel weighting to complete the fusion, so that the image can display more detailed information and improve image quality. For this reason, the technical scheme that the present invention takes is, the high dynamic range image fusion method based on gradient and brightness information, comprises the following steps:

1) For each input multi-exposure image, calculate its local contrast weight factor L _i (x, y);

2) Convert the original image to the color space of hue H, color saturation S and brightness I, extract the brightness component I, and calculate its brightness weight factor H _i (x, y);

3) Weight estimation: the two image quality measurement factors are combined to calculate the weight function for image fusion After normalization, the normalized weight function W _i ′(x,y) is obtained:

ε<10 ^-12 ;

4) Use a recursive filter to improve the weight map to obtain the final weight function W _i (x, y) of each input image:

W _i (x, y) = RF (W _i '(x, y), sigma_s, sigma_r) (9)

Among them, RF represents the recursive filter operator, and sigma_s and sigma_r are filter parameters;

5) Perform weighted fusion of multi-exposure images, using W _i (x, y) as the weight to obtain the final fused image F(x, y):

I _i (x, y) is the intensity of the pixel at the position (x, y) of the i-th input image.

The local contrast weighting factor L _i (x,y) is calculated by the following formula:

Among them, g _i (x, y) is the image gradient value obtained by applying the Sobel gradient operator to the grayscale image of the input i-th image, is the unnormalized local contrast weighting factor.

The calculation of the brightness weighting factor uses the following formula to eliminate these pixels with poor exposure, and obtain the input image B _i (x,y) with pixels with good exposure:

Among them, l _i (x, y) is the brightness value of the pixel point of the i-th input image at the position (x, y), t is a threshold value defining the exposure range, and the weight of the pixel point with a brightness value of 0.5 is set to is 1, the weight values of other pixels decrease sequentially according to the increase or decrease of the brightness value, and the initial brightness weight map is obtained

To avoid singularities, the brightness weight is added to a very small value to get the improved brightness weight factor

Normalized to get the final brightness weight factor H _i (x,y):

Compared with prior art, technical characteristic and effect of the present invention:

The fused image obtained by adopting the technical solution of the present invention has a better overall light and shade effect from a subjective point of view, especially for the fusion of dark areas in the image, and has a better color reproduction degree. Since the fusion is performed by directly multiplying the weight map and the original multi-exposure image sequence, complex image decomposition calculation is not required, and the calculation complexity of the algorithm of the present invention is low. In terms of objective index experiments.

Description of drawings

Figure 1 "Garage" multi-exposure image sequence and fusion results, in the figure, (a) "Garage" LDR image sequence; (b) algorithm of Mertens et al.; (c) algorithm of Wei Zhang et al.; (d) algorithm of the present invention.

Fig. 2 is a flowchart of the scheme of the present invention.

detailed description

The dynamic range of the image sensor is much smaller than the natural brightness variation range, which makes it impossible to obtain a clear image with a single shot when the brightness changes drastically. Multi-exposure image fusion is an effective way to expand the dynamic range of images. It can significantly improve image clarity and contrast for scenes with large brightness changes and no fast moving objects, so that more image details can be displayed. High dynamic range technology can be widely used in traffic video, biomedical, satellite remote sensing, games and other industries that need to display high dynamic detail images. Therefore, developing a method for extending the dynamic range of an image has high research and application value.

The invention directly weights and fuses multi-exposure images based on two image quality measurement factors. Through the weight function composed of two fusion indexes, a weight map can be calculated for each multi-exposure image. The two weighting factors are as follows:

1. Local contrast weighting factor. For the input multi-exposure image, its gradient size can reflect the clarity of the image and the amount of information it contains. In underexposed or overexposed areas of the image, the gradient values of the pixels will be very small. On the contrary, in well-exposed areas, the gradient value of pixels will be relatively large. Therefore, the gradient size can be used as an index to measure the image exposure quality. The local contrast weighting factor L _i (x, y) of the multi-exposure image can be calculated by the following formula:

where ε is a very small value to avoid singularities. g _i (x, y) is the image gradient value obtained by applying the Sobel gradient operator to the grayscale image of the input i-th image, is the unnormalized local contrast weighting factor.

2. Brightness weighting factor. The brightness value of a pixel can often reflect the quality of the image exposure. Pixels with appropriate brightness values in the image have better color effects and clearer details. Therefore, the present invention selects the luminance component as another image quality measurement factor. In order to avoid pixels with too small or too large brightness values causing interference during fusion, the following formula can be used to eliminate these pixels with poor exposure, and obtain an input image B _i (x,y) with pixels with good exposure :

Among them, l _i (x, y) is the luminance value of the pixel at the (x, y) position of the i-th input image, and t is a threshold defining the exposure range. In the present invention, the weight of a pixel with a brightness value of 0.5 is set to 1 when constructing the brightness weight factor. The weight values of other pixels decrease in turn according to the increase or decrease of the brightness value, and the initial brightness weight map is obtained

To avoid singularities, the brightness weight is added to a very small value to get the improved brightness weight factor

Normalized to get the final brightness weight factor H _i (x,y):

The steps of the multi-exposure image sequence fusion algorithm are as follows:

1) For each input multi-exposure image, calculate its local contrast weighting factor.

2) Convert the original image to Hue, Saturation and Brightness (HIS) color space, extract the brightness component I, and calculate its brightness weight factor.

3) Weight estimation. Combines two image quality measures to compute a weighting function for image fusion After normalization, the normalized weight function W _i ′(x,y) is obtained:

4) Use a recursive filter to improve the weight map, so that the weight values in areas with similar image exposures are also similar, avoiding gap artifacts in the fused image, and obtaining the final weight function W _i (x, y):

W _i (x, y) = RF (W _i '(x, y), sigma_s, sigma_r) (9)

Among them, RF represents a recursive filter operator, and sigma_s and sigma_r are filter parameters.

5) Perform weighted fusion of multi-exposure images, using W _i (x, y) as the weight to obtain the final fused image F(x, y):

In order to verify the effect of the algorithm, the above-mentioned algorithm is applied to fuse the multi-exposure image sequence and the fusion effect is analyzed. For color images, the three channels of R, G, and B are calculated separately during fusion, and only the brightness component is operated when calculating the brightness factor. In order to compare the quality of the fused images, the results of the algorithm of the present invention are compared with the algorithm of Mertens et al. [2] and the algorithm of Wei Zhang et al. [3]. The experimental results were analyzed from both subjective and objective aspects.

The objective experiment uses image mean, color entropy, average gradient, information entropy and standard deviation to measure the experimental results. The image mean reflects the average lightness and darkness of the image. Color entropy reflects the sum of the amount of information contained in the three channels of the image R, G, and B. The average gradient is the sharpness of the image, which reflects the ability of the image to express the contrast of details. Information entropy reflects the amount of information contained in an image, and is an important indicator to measure the richness of image information. The standard deviation reflects the degree of dispersion of the gray mean.

The experimental results of the “Garage” image sequence are shown in Figure 1 and Table 1.

Table 1 Experimental results of objective indicators

From the experimental results in Figure 1, it can be seen that the Mertens et al. algorithm has a good effect in terms of color reproduction, but the details in the dark area of the garage are not displayed clearly, there are many underexposed pixels, and the overall image is dark. Moreover, due to the need to decompose and reconstruct the Laplacian pyramid, the method of Mertens et al. is too computationally intensive. Similarly, the algorithm of Wei Zhang et al. shows darker garage area and lacks detailed information. The fused image obtained by adopting the technical solution of the present invention has a better overall light and shade effect from a subjective point of view, especially for the fusion of dark areas in the image, and has a better color reproduction degree. Since the fusion is performed by directly multiplying the weight map and the original multi-exposure image sequence, complex image decomposition calculation is not required, and the calculation complexity of the algorithm of the present invention is low. In terms of objective index experiment, it can be seen from Table 1 that the average value, color entropy, information entropy, and standard deviation index of the fusion image generated by the algorithm of the present invention are higher than those of the other two algorithms, which is consistent with the conclusion of subjective visual effect evaluation. This shows that the algorithm of the present invention is superior to the other two algorithms in the extraction and processing of detailed information.

In practical applications, in order to obtain the best fusion result, the parameters involved in the algorithm of the present invention are set as follows: the value range of ε is less than or equal to 10 ⁻¹² , the value range of t is 0.05 to 0.2, sigma_s=60, sigma_r =2. In the above experiment, ε=10 ^-12 , t=0.1. Taking the multi-exposure image sequence as the experimental object, adopting the steps described in the present invention and setting the above parameter values, a fused image with better visual effect can be obtained. Experimental results show that the algorithm of the present invention has better effects in terms of subjective vision and objective quantitative indicators.

Claims (3)

1. a high dynamic range image fusion method based on gradient and brightness information, is characterized in that, comprises the steps: 1) For each input multi-exposure image, calculate its local contrast weight factor L _i (x, y); 2) Convert the original image to the color space of hue H, color saturation S and brightness I, extract the brightness component I, and calculate its brightness weight factor H _i (x, y); 3) Weight estimation: the two image quality measurement factors are combined to calculate the weight function for image fusion After normalization, the normalized weight function W _i ′(x,y) is obtained: <mrow><msubsup><mi>W</mi><mi>i</mi><mo>&amp;prime;</mo></msubsup><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow><mo>=</mo><mfrac><mrow><msub><mover><mi>W</mi><mo>^</mo></mover><mi>i</mi></msub><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow></mrow><mrow><munderover><mo>&amp;Sigma;</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><msub><mover><mi>W</mi><mo>^</mo></mover><mi>i</mi></msub><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow></mrow></mfrac><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>8</mn><mo>)</mo></mrow></mrow> ε<10 ^-12 ; 4) Use a recursive filter to improve the weight map to obtain the final weight function W _i (x, y) of each input image: W _i (x, y) = RF (W _i '(x, y), sigma_s, sigma_r) (9) Among them, RF represents the recursive filter operator, and sigma_s and sigma_r are filter parameters; 5) Perform weighted fusion of multi-exposure images, using W _i (x, y) as the weight to obtain the final fused image F(x, y): <mrow><mi>F</mi><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow><mo>=</mo><msubsup><mi>&amp;Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></msubsup><msub><mi>W</mi><mi>i</mi></msub><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow><mo>&amp;times;</mo><msub><mi>I</mi><mi>i</mi></msub><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>10</mn><mo>)</mo></mrow></mrow> I _i (x, y) is the intensity of the pixel at the position (x, y) of the i-th input image. 2. the high dynamic range image fusion method based on gradient and brightness information as claimed in claim 1, is characterized in that, local contrast weighting factor L _i (x, y) is calculated by following formula: <mrow><msub><mover><mi>L</mi><mo>^</mo></mover><mi>i</mi></msub><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow><mo>=</mo><msub><mi>g</mi><mrow><mi>i</mi><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></mrow></msub><mo>+</mo><mi>&amp;epsiv;</mi><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>1</mn><mo>)</mo></mrow></mrow> <mrow><msub><mi>L</mi><mi>i</mi></msub><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow><mo>=</mo><mfrac><mrow><msub><mover><mi>L</mi><mo>^</mo></mover><mi>i</mi></msub><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></mrow><mrow><msubsup><mi>&amp;Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></msubsup><msub><mover><mi>L</mi><mo>^</mo></mover><mi>i</mi></msub><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi></mrow><mo>)</mo></mrow></mrow></mfrac><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>2</mn><mo>)</mo></mrow></mrow> Among them, g _i (x, y) is the image gradient value obtained by applying the Sobel gradient operator to the grayscale image of the input i-th image, is the unnormalized local contrast weighting factor. 3. the high dynamic range image fusion method based on gradient and luminance information as claimed in claim 1, is characterized in that, the calculation of luminance weight factor uses following formula to get rid of these pixels with poor exposure, and obtains a pixel with good exposure. Input image B _i (x,y) of pixels: Among them, l _i (x, y) is the brightness value of the pixel point of the i-th input image at the position (x, y), t is a threshold value defining the exposure range, and the weight of the pixel point with a brightness value of 0.5 is set to is 1, the weight values of other pixels decrease sequentially according to the increase or decrease of the brightness value, and the initial brightness weight map is obtained To avoid singularities, the brightness weight is added to a very small value to get the improved brightness weight factor <mrow><msub><mover><mi>H</mi><mo>^</mo></mover><mi>i</mi></msub><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow><mo>=</mo><msub><mover><mi>H</mi><mo>~</mo></mover><mi>i</mi></msub><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow><mo>+</mo><mi>&amp;epsiv;</mi><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>5</mn><mo>)</mo></mrow></mrow> 1 Normalized to get the final brightness weight factor H _i (x,y): <mrow><msub><mi>H</mi><mi>i</mi></msub><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow><mo>=</mo><mfrac><mrow><msub><mover><mi>H</mi><mo>^</mo></mover><mi>i</mi></msub><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow></mrow><mrow><msubsup><mi>&amp;Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></msubsup><msub><mover><mi>H</mi><mo>^</mo></mover><mi>i</mi></msub><mrow><mo>(</mo><mi>x</mi><mo>,</mo><mi>y</mi><mo>)</mo></mrow></mrow></mfrac><mo>-</mo><mo>-</mo><mo>-</mo><mrow><mo>(</mo><mn>6</mn><mo>)</mo></mrow><mo>.</mo></mrow> 2