CN104616273B - A kind of many exposure image fusion methods based on Laplacian pyramid - Google Patents

Abstract

A kind of many exposure image fusion methods based on Laplacian pyramid of the disclosure of the invention, are related to computer vision field and image/video process field, and research biases toward image procossing.The method is for traditional fusion method based on laplacian pyramid the drawbacks of have lost the information such as the details of image during carrying out multi-resolution decomposition and reconstruct, a certain degree of details enhancing is carried out to input picture using image detail enhancement method, a part is offset because pyramid amplifies and reduces the loss of detail that causes, so as to include the effect of abundant details and definition higher with the generation image after being merged to many exposure images.

Description

A Multi-exposure Image Fusion Method Based on Laplacian Pyramid Decomposition

technical field

The invention relates to the fields of computer vision and image/video processing, and the research focuses on image processing.

Background technique

Dynamic range refers to the ratio between the maximum and minimum values of a physical quantity. For an actual scene, the dynamic range refers to the ratio of the light radiance of the brightest point to the darkest point in the real scene. The dynamic range covered by the high dynamic range image is very large, which can well reproduce the natural scene, retain the detailed information in the scene, and bring people a realistic visual experience. High dynamic range imaging technology plays an important role in the field of image and video. When a real scene has a high dynamic range, the resulting image will always contain overexposed and underexposed areas, even if the camera is set to the correct exposure. High dynamic range imaging technology can obtain high dynamic range images and make these high dynamic range images with rich details displayed on low dynamic range devices.

At present, the commonly used high dynamic range imaging technology is multi-exposure image fusion method, including: direct fusion method, region-based fusion method and layer-based fusion method. The direct fusion method is to directly fuse the input images according to the obtained weight map of the input images. In this method, weight map quality is the key to obtain high-quality images. In order to obtain high-quality weight maps, this method can become very complicated. The region-based fusion method first divides the input image into several regions, then selects the optimal blocks in the same region of all input images, and finally fuses the selected optimal regions. Since this method requires pixel-level operations to select the optimal block and pixel-level fusion, the algorithm takes a long time to run. The layer-based fusion method is a fusion method that uses a layered framework to decompose the input image, perform certain processing, and then reconstruct it. Commonly used layer-based fusion methods are: Laplacian-based fusion methods and sub-band-based fusion methods. Subband-based fusion methods can obtain high-quality high-dynamic-range images with rich details. Because the acquisition of the gain control chart of this method is relatively complicated, the complexity of this method is very high, and the real-time performance is not high. The multi-exposure image fusion method based on Laplacian pyramid is currently the most used method to obtain high dynamic range images. The traditional fusion method based on the Laplacian pyramid loses information such as image details in the process of multi-scale decomposition and reconstruction, and does not compensate for the lost information, so that the final high dynamic range image lacks details, etc. information. Therefore, it is very important to propose a new multi-exposure image fusion method based on Laplacian pyramid to obtain high dynamic range images with rich details and high definition. Figure 1 shows the traditional fusion framework based on Laplacian pyramid.

The main purpose of the present invention is to realize a high real-time fusion method and be able to generate high dynamic range images with rich details and high definition. To achieve this goal, a new multi-exposure image fusion method based on the Laplacian pyramid decomposition framework is proposed.

Contents of the invention

The present invention improves and designs a multi-exposure image fusion method based on Laplacian pyramid decomposition in view of the deficiency of the background technology, so as to achieve that the generated image after multi-exposure image fusion contains rich details and high definition.

The technical solution of the present invention is a multi-exposure image fusion method based on Laplacian pyramid decomposition, which aims at image loss in the process of multi-scale decomposition and reconstruction in the traditional fusion method based on Laplacian pyramid The drawbacks of information such as details, etc., use the image detail enhancement method to enhance the details of the input image to a certain extent, offset part of the loss of details caused by the pyramid enlargement and reduction, so as to achieve the purpose of the invention. Thereby inventing a kind of multi-exposure image fusion method based on Laplacian pyramid decomposition, the method comprises:

Step 1: Select images obtained by taking different exposures of the same scene. Each image in these images contains the details of the scene, and the size is the same. At the same time, the image size is adjusted so that the image size can meet the conditions of pyramid decomposition;

Step 2: performing detail enhancement processing on each selected image;

Step 3: Perform Laplacian pyramid decomposition on each processed image;

Step 4: Obtain the weight map of each image selected in step 1;

Step 5: Perform the same Laplacian pyramid decomposition as in step 3 for each weight map;

Step 6: Fuse the decomposed image of step 3 by using the weight map decomposition diagram obtained in step 5;

Step 7: Reconstruct the fused image to obtain the final image.

Further, in the step 2, the spatial detail enhancement method is used to perform detail enhancement processing on each selected image; as shown in the following formula:

O(x,y)=I(x,y)+α×[I(x,y)-LP(I(x,y))]

Among them, O(x,y) represents the output image after enhancing the details, I(x,y) represents the original input image, LP(I(x,y)) represents smoothing and filtering the input image, and α represents the weight factor for detail enhancement .

Further, the step 3 uses the Laplacian pyramid decomposition method based on Gaussian pyramid decomposition to decompose each processed image; as shown in the following formula:

Where G _l represents the l-th layer Gaussian pyramid image of image G ₀ , Indicates that the l+1 layer Gaussian pyramid image of G ₀ is interpolated and enlarged by one level, n indicates the total number of layers decomposed by the Gaussian pyramid, and Lap _l indicates the l-th layer image of the Laplacian pyramid.

Further, the step 4 determines the weight map of each image according to the three quality evaluation criteria of image detail, saturation, and exposure.

Further, the specific steps of the step 6 are:

Step 6-1: Carry out corresponding fusion of the images of each layer decomposed by the Laplacian pyramid of the same image and the image of the same layer decomposed by its weight map, and obtain a fused image for each layer;

Step 6-2: Fusing the fused images of the same layer of each image to obtain n fused images, where n represents the number of layers of the Laplacian pyramid decomposition image.

Further, the specific steps of the step 7 are:

Step 7-1: Calculate the image features of each fusion image obtained in step 6, the calculation formula is:

Where w and h represent the width and height of the image, e _r , e _g and e _b represent the overexposure of the pixel at (w, h) in the three channels r, g, and b respectively;

Step 7-2: Construct the enhancement factor of the fusion image of each layer: Among them, l indicates that the image is the lth layer of the Laplacian pyramid after fusion, and C is the image feature;

Step 7-3: Reconstruct the image combined with the enhancement factor:

Where Res represents the reconstructed result image, l represents the level of the Laplacian pyramid, n represents the total number of layers of the pyramid, Lap _{l, l} represents the image of the fused Laplacian pyramid of the l layer after l times Interpolate the resulting enlarged image.

The present invention is a multi-exposure image fusion method based on Laplacian pyramid decomposition, which aims at the loss of image details and other information in the process of multi-scale decomposition and reconstruction in the traditional fusion method based on Laplacian pyramid The drawbacks of the image detail enhancement method are used to enhance the details of the input image to a certain extent, offsetting part of the loss of details caused by the pyramid enlargement and reduction, so that the generated image after the multi-exposure image fusion contains rich details and high The effect of clarity.

Description of drawings

Fig. 1 is a traditional fusion flow chart based on the Laplacian pyramid;

Fig. 2 is a kind of flow chart of multi-exposure image fusion method based on Laplacian pyramid decomposition of the present invention;

detailed description

The main idea of this new multi-exposure image fusion method is: use the image detail enhancement method to enhance the details of the input image to a certain extent, and offset part of the loss of details caused by the pyramid enlargement and reduction. A new reconstruction method is proposed to enhance the detail and sharpness of the image again. The present invention mainly comprises the following steps:

A. Select a group of multi-exposure image sequences of the same scene as the input image, and adjust the image size at the same time so that the image size can meet the conditions of pyramid decomposition;

B. Enhance the details of each input image;

C. Perform Laplacian pyramid decomposition on the enhanced input image;

D. Obtain the weight map of the original image;

E. Use the weight map to fuse the image decomposed in step C;

F. Pyramid reconstruction.

The input images selected in step A are images obtained by taking different exposures of the same scene. Each of these images contains some details of the scene, and the images are of the same size without deviation. Adjust the width and height of the image It is a certain power of 2 and the size loss of the image is the smallest.

The detail enhancement of the image in step B adopts the spatial detail enhancement technology. The details of the image are enhanced to a certain extent. The detail enhancement algorithm is shown in formula (1).

O(x,y)=I(x,y)+α×[I(x,y)-LP(I(x,y))] (1)

Where I(x,y) represents the original input image, O(x,y) represents the output image after enhancing the details, LP(I(x,y)) represents the smoothing and filtering of the input image, and α represents the weight factor for detail enhancement .

Step C decomposes each set of input images with different exposures into a Laplacian pyramid. Laplacian pyramids are commonly used in the fields of image analysis and image processing. When the Laplacian pyramid is used, the image is decomposed into multiple levels with different resolutions. The decomposition of the Laplacian pyramid is based on the decomposition of the Gaussian pyramid. The composition of the Laplacian pyramid of the image G ₀ is shown in formula (2).

Where G _l represents the l-th layer Gaussian pyramid image of image G ₀ , Indicates that the l+1 layer Gaussian pyramid image of G ₀ is interpolated and enlarged by one level, n indicates the total number of layers decomposed by the Gaussian pyramid, and Lap _l indicates the l-th layer image of the Laplacian pyramid, where the Gaussian pyramid The total number of layers n is determined according to the smaller value of the length and width of the adjusted image, which is the power of 2 minus 1. For example, if the width of the image is small and the size is 32, the total number of layers of the Gaussian pyramid n is equal to 4.

Step D obtains the weight map of the original input image. The weight map can reflect the quality of pixels in an image to a certain extent. The weight map is a grayscale image. The area with higher pixel value indicates the higher quality of the image in this area. The quality evaluation standard of the image in the present invention adopts the detail, saturation and appropriate exposure of the image. The acquisition of the weight map further includes:

D1. Calculate the detail of the image. The gradient information of each pixel in the grayscale image of the image is used as a measure of detail. The calculation method of the detail degree D(x, y) of the image at (x, y) is shown in formulas (3) and (4).

Where I(x, y) represents the gray value of the pixel at image (x, y), ΔI _x represents the gradient value in the horizontal direction at (x, y), and ΔI _y represents the gradient in the vertical direction at (x, y). value.

D(x,y)=ΔI _x +ΔI _y (4)

D2. Calculate the saturation of the image. As the exposure time increases, the color saturation of the image will gradually decrease until it disappears completely, so saturation is necessary as an evaluation factor of image quality. The saturation factor S can be obtained from the standard deviations of the R, G, and B channels, as shown in formula (5).

Among them, R, G, and B respectively represent the pixel values of the pixels at the image (x, y) in the three channels of red, green, and blue.

D3. Calculate the moderate exposure of the image. The overexposed area in the image is generally displayed as white, and the pixel value of the corresponding pixel point is 1, and the same underexposed area is displayed as black, and the pixel value of the corresponding pixel point is 0, so the pixel value in the image is neither close to 1 nor Pixels close to 0 should be reserved, and the closer the pixel value is to 0.5, the higher the weight should be given. The calculation of exposure E at image (x, y) is shown in formula (6).

Among them, r, g, and b respectively represent the pixel values of the pixels at the image (x, y) in the three channels of red, green and blue, and the value of the parameter θ is 0.2, which controls the range of the weight function.

D4. Fusing the measurement factors to generate a weight map. The three measurement factors of each pixel are integrated by multiplication to form the final weight. The weight function of the kth sub-image is shown in formulas (7) and (8).

W _k (x, y) = D _k (x, y) × S _k (x, y) × E _k (x, y) (7)

Among them, D _k (x, y), S _k (x, y) and E _k (x, y) represent the detail, saturation and moderate exposure of the image at (x, y), respectively. For consistent results, the weight maps are normalized. After normalization, the sum of the weight values of the same position of all input images is one. Normalization is shown in formula (8).

Where n represents the number of input images, and W _nor,k (x,y) represents the normalized weight map.

Step E uses the weight map to fuse the decomposed images. The weight map of the input image is first decomposed into the same number of decomposition layers as the original input image using a Gaussian pyramid. The image fused with the decomposed weight map will contain rich details, high saturation and moderate exposure. Image fusion is to combine the image of a certain layer of a certain input image of the Laplacian Pyramid with the same input image of the Gaussian Pyramid and the weight map of the same layer, and then further combine all the combined images of the same layer. The method is shown in formula (9).

The fused layer-1 image of the Laplacian pyramid is the weighted average of the images of the layer-1 Laplacian pyramid of all input images. Lap(F) ^l represents the l-th layer image of the fused Laplacian pyramid, Represents the l-th layer image of the Gaussian pyramid of the k-th sub-image weight map, Represents the l-th layer image of the Laplacian pyramid of the k-th input image.

Step F pyramid reconstruction. The reconstruction of the Laplacian Pyramid is to interpolate and enlarge the image of each layer of the fused Laplacian Pyramid. After several times of enlargement, it becomes an image with the same size as the original input image, and then performs pixel-level image reconstruction. The addition operation can be reconstructed. The degree of the image interpolation method is the same as the layer number of the Laplacian pyramid in which it is located. The new reconstruction method proposed in the present invention can make the reconstructed image contain rich details and high definition. Refactoring methods further include:

F1. Calculate the fused image features. Compared with the original input image, the quality of the fused image has been greatly improved, not only in the overall details, but also in terms of saturation and exposure. Layers 0 to n-1 of the Laplacian Pyramid are residual image sequences, and the nth layer of the Laplacian Pyramid, which is the highest layer, is the nth layer of the Gaussian Pyramid, which has the same characteristics as the original input image to a certain extent. Characteristics. The nth layer of the fused Laplacian pyramid can also reflect some features of the final reconstructed image to some extent. In the present invention, the enhancement degree of the image is controlled by calculating the overexposure degree of the fused Laplacian pyramid, so as to avoid over-enhanced and over-exposed regions. The calculation of the fused image features is shown in formula (10).

Where w and h represent the width and height of the image, e _r , e _g and e _b respectively represent the overexposure of the pixel at (w, h) in the red, green and blue channels, such as formulas (11), (12) and (13).

Where r represents the pixel value of the pixel in the r channel.

Where g represents the pixel value of the pixel in the g channel.

Where b represents the pixel value of the pixel in the b channel.

F2. Structural enhancement factor. The enhancement factor should be constructed by combining the features of the fused image. The features of the fused image can control the exposure of the image. In addition, some enhancement factors should be added. Due to the different degrees of filtering of different layers of the fused image, the clarity and detail of each layer will be lost to varying degrees. When reconstructing, the images of each layer still need to be filtered again, and the details and clarity will also be lost. The present invention will assign different weights to each layer according to its sharpness and detail. The loss of detail and sharpness in the high layers is more severe, and will be assigned a lower weight, while the details and sharpness in the lower layers are richer, and will be assigned higher weights. The construction of the enhancement factor of the Laplacian pyramid image after layer l fusion is shown in formula (14).

Among them, L indicates that the image is the lth layer of the fused Laplacian pyramid, and C is the feature of the image.

F3. Combining enhancement factors for image reconstruction. The reconstruction of the image is to interpolate and enlarge the images of each layer of the Laplacian pyramid for different times until the size of the original image is the same. The enhancement factors of each layer are combined with the interpolated and enlarged images of each layer and reconstructed. The combination of the enhancement factor and the corresponding image and the reconstruction are shown in Equation (15).

Where Res represents the reconstructed result image, l represents the level of the Laplacian pyramid, n represents the total number of layers of the pyramid, Lap _{l, l} represents the image of the fused Laplacian pyramid of the l layer after l times Interpolate the resulting enlarged image.

The specific flow chart of the present invention is shown in Fig. 2 .

Aiming at the multi-exposure image fusion method based on the Laplacian pyramid, the invention proposes a new multi-exposure image fusion method. The present invention no longer adopts the traditional pyramid reconstruction method, and proposes a new reconstruction method. The images of each layer of the fused Laplacian pyramid are interpolated and enlarged to become images of the same size as the original image, and then an enhancement factor is added to these images to enhance the details and clarity of the image, and finally the enhanced image is compared add.

When the image is reconstructed, the images at different levels of the Laplacian pyramid will be assigned different weights according to their clarity and detail. The higher the number of layers in the Laplacian pyramid of the image before interpolation and magnification, the lower the detail and clarity of the image, so it should be assigned a lower weight. However, images that are at lower levels in the Laplacian pyramid before interpolation and magnification will contain higher detail and clarity and should be assigned higher weights. This ensures that the fused image contains high detail and clarity. In order to ensure that the enhancement degree of the image is moderate, the features of the fused image will also be fused into the enhancement factor. The highest layer of the Laplacian pyramid after fusion is not the residual layer, but the image layer, which can reflect the characteristics of the fused image to a certain extent. The present invention obtains the image characteristics by calculating the degree of overexposure of the image. When the exposure When it is high, in order to avoid overexposure, the degree of enhancement should be weakened to a certain extent, and when the degree of exposure is low, the degree of enhancement should be moderately increased.

This new reconstruction method based on Laplacian pyramid decomposition adds an enhancement factor during reconstruction, which not only ensures that the reconstructed high dynamic range image contains rich details and high definition, but also can control Overexposure phenomenon caused by excessive enhancement. This new multi-exposure image fusion method makes image enhancement and enhancement degree reach a balance.

Compared with other methods, the method described in the present invention contains richer details, higher definition and has satisfactory operation speed. The experimental results of this patent are measured using traditional image quality evaluation criteria: saturation, contrast, and sharpness. Compared with other three multi-exposure image fusion methods to obtain high dynamic range images, the experimental results are shown in Table 1, Table 2 and Table 3; Table 1 shows the saturation of the high dynamic range images obtained by the four fusion methods degree data. It can be seen from the data that the saturation of the high dynamic range image generated by the present invention is the highest, and the saturation of the image generated by other methods is insufficient. Table 2 shows the contrast data of the images. The contrast of the resulting images of the present invention has the same contrast data as the method proposed by Mertens et al., to the extent that the results of the present invention can be considered to contain the highest contrast. Table 3 shows the detail information of the image. Except for the first set of experimental results, the result image of the present invention contains much higher detail than other results. The experimental results and data analysis fully demonstrate the advantages of the present invention, and the experimental results conform to the proposed idea of the present invention: improving image quality, especially image details. The speed of the present invention is similar to other similar methods, and has high practical value.

Table 1 Saturation of images

Table 2 Contrast of images

Table 3 The degree of detail of the image

Claims (5)

1. A multi-exposure image fusion method based on Laplacian pyramid decomposition, the method comprising: Step 1: Select images obtained by taking different exposures of the same scene. Each image in these images contains the details of the scene, and the size is the same. At the same time, the image size is adjusted so that the image size can meet the conditions of pyramid decomposition; Step 2: performing detail enhancement processing on each selected image; Step 3: Perform Laplacian pyramid decomposition on each processed image; Step 4: Obtain the weight map of each image selected in step 1; Step 5: Perform the same Laplacian pyramid decomposition as in step 3 for each weight map; Step 6: Fuse the decomposed image of step 3 by using the weight map decomposition diagram obtained in step 5; Step 7: Reconstruct the fused image to obtain the final image; It is characterized in that the specific steps of the step 7 are: Step 7-1: Calculate the image features of each fusion image obtained in step 6, the calculation formula is: Where w and h represent the width and height of the image, e _r , e _g and e _b represent the overexposure of the pixel at (w, h) in the three channels r, g, and b respectively; the calculation method is: Where r represents the pixel value of the pixel in the r channel; Where g represents the pixel value of the pixel in the g channel; Where b represents the pixel value of the pixel in the b channel; Step 7-2: Construct the enhancement factor of the fusion image of each layer: Among them, l indicates that the image is the lth layer of the Laplacian pyramid after fusion, and C is the image feature; Step 7-3: Reconstruct the image combined with the enhancement factor: Where Res represents the reconstructed result image, l represents the level of the Laplacian pyramid, n represents the total number of layers of the pyramid, Lap _{l, l} represents the image of the fused Laplacian pyramid of the l layer after l times Interpolate the resulting image after zooming in. 2. a kind of multi-exposure image fusion method based on Laplacian Pyramid decomposition as claimed in claim 1, it is characterized in that in the described step 2, adopt the spatial detail enhancement method to carry out detail enhancement process to each image selected; as follows The formula shows: O(x,y)=I(x,y)+α×[I(x,y)-LP(I(x,y))] Among them, O(x,y) represents the output image after enhancing the details, I(x,y) represents the original input image, LP(I(x,y)) represents smoothing and filtering the input image, and α represents the weight factor for detail enhancement . 3. a kind of multi-exposure image fusion method based on Laplacian pyramid decomposition as claimed in claim 1, it is characterized in that described step 3 adopts the Laplacian pyramid decomposition method based on Gaussian pyramid decomposition to process each The final image is decomposed; as shown in the following formula: Where G _l represents the l-th layer Gaussian pyramid image of image G ₀ , Indicates that the l+1 layer Gaussian pyramid image of G ₀ is interpolated and enlarged by one level, n indicates the total number of layers decomposed by the Gaussian pyramid, and Lap _l indicates the l-th layer image of the Laplacian pyramid. 4. a kind of multi-exposure image fusion method based on Laplacian pyramid decomposition as claimed in claim 1, it is characterized in that described step 4 comes out according to these 3 quality evaluation criteria of detail, saturation, exposure of image Determine the weight map for each image. 5. a kind of multi-exposure image fusion method based on Laplace pyramid decomposition as claimed in claim 1, is characterized in that the concrete steps of described step 6 are: Step 6-1: Carry out corresponding fusion of the images of each layer decomposed by the Laplacian pyramid of the same image and the image of the same layer decomposed by its weight map, and obtain a fused image for each layer; Step 6-2: Fusing the fused images of the same layer of each image to obtain n fused images, where n represents the number of layers of the Laplacian pyramid decomposition image.