CN106251365A - Many exposure video fusion method and device - Google Patents

Abstract

The present invention provides a multi-exposure video fusion method and device, wherein the method includes: determining a reference frame and at least one frame to be processed in the obtained multi-exposure image sequence, and respectively converting each of the frames to be processed according to the consistency sensitive hash algorithm Frame alignment to the reference frame; calculating a weight map of each aligned frame to be processed according to image features of each aligned frame to be processed, and calculating a weight map of the reference frame according to image features of the reference frame Weight map: using the reference frame to optimize the weight map of each aligned frame to be processed, and fusing each optimized weight map with the weight map of the reference frame to obtain a fused image. Through the multi-exposure video fusion method and device of the present invention, in view of the situation that there is a moving target in the captured scene, the multi-exposure image sequence can be fused to obtain a clear, fuzzy, and ghost-free fused image.

Description

Multi-exposure video fusion method and device

technical field

The present invention relates to the field of video processing, in particular to a multi-exposure video fusion method and device.

Background technique

Multi-exposure video fusion refers to the fusion of multi-exposure image sequences in the captured multi-exposure video to obtain a well-exposed fused image. The multi-exposure video is obtained in the following way: using multiple cameras with different The scene is recorded to obtain a multi-exposure video composed of a sequence of multi-exposure images, wherein the multi-exposure images are also called video frames.

In order to obtain a well-exposed fused image, the existing technology provides a variety of multi-exposure video fusion methods, for example, a fusion method based on the Laplacian pyramid for color image fusion, using local contrast and neighborhood color consistency metrics, A probabilistic fusion method proposed under the random walk framework, and a method to infer fused pixels by constructing a probabilistic model using the maximum a posteriori probability framework.

However, the fusion methods in the prior art are all proposed for static scenes, without considering the impact of moving objects on the fusion effect. When there are moving objects in the captured scene, the fused image obtained by the fusion method of the prior art There is blurring and ghosting, and it is not possible to obtain a complete and clear fused image.

Contents of the invention

In view of this, the purpose of the present invention is to provide a multi-exposure video fusion method and device, which can fuse the multi-exposure image sequence to obtain clear, fuzzy and ghost-free fusion in view of the presence of moving objects in the captured scene. image.

In the first aspect, an embodiment of the present invention provides a multi-exposure video fusion method, the method includes: determining a reference frame and at least one frame to be processed in the acquired multi-exposure image sequence, respectively Each of the frames to be processed is aligned to the reference frame; a weight map of each aligned frame to be processed is calculated according to the image features of the aligned frames to be processed, and a weight map is calculated according to the image features of the reference frame The weight map of the reference frame; using the reference frame to optimize the weight map of each aligned frame to be processed, and fusing each optimized weight map with the weight map of the reference frame to obtain a fused image .

With reference to the first aspect, the embodiment of the present invention provides a first possible implementation manner of the first aspect, wherein respectively aligning each of the frames to be processed to the reference frame according to a consistency-sensitive hash algorithm includes: respectively performing scale decomposition on each of the frames to be processed and the reference frame to obtain each subframe set to be processed corresponding to each frame to be processed and a reference subframe set corresponding to the reference frame; Each reference subframe in the reference subframe set and each subframe to be processed in each subframe set to be processed are divided into blocks; each subframe in each subframe set to be processed is divided into blocks according to the consistency sensitive hash algorithm. The divided subframes to be processed are aligned to the respective corresponding divided reference subframes in the set of reference subframes; a grayscale mapping function is used to map each of the subframes to be processed Performing mismatch correction on the aligned subframes to be processed, and optimizing the result of the mismatch correction by using a Poisson video fusion method; The subframes are processed for reconstruction to obtain the aligned frames to be processed.

With reference to the first possible implementation manner of the first aspect, the embodiment of the present invention provides a second possible implementation manner of the first aspect, wherein, according to the consistency-sensitive hash algorithm, the The subframes to be processed after each block are aligned to the reference subframes after each corresponding block in the reference subframe set, including: the current block in the current set of subframes to be processed The image block to be matched is determined in the subsequent subframe to be processed, and a plurality of neighboring image blocks corresponding to the image block to be matched are determined in the reference subframe after the current block in the set of reference subframes ; Calculate the matching distance between the image block to be matched and each of the neighborhood image blocks according to the consistency-sensitive hash algorithm, and align the image block to be matched to the neighborhood image corresponding to the smallest matching distance block; repeating the actions of determining the image block to be matched, determining the neighborhood image block, calculating the matching distance, and aligning the image blocks to be matched, until the sub-frames in each sub-frame set to be processed are divided into blocks. The subframes are aligned to each corresponding block of the reference subframes in the set of reference subframes.

In combination with the first aspect, the embodiment of the present invention provides a third possible implementation manner of the first aspect, wherein the weight map of each aligned frame to be processed is calculated according to the image features of each aligned frame to be processed , and calculating the weight map of the reference frame according to the image features of the reference frame, including: respectively calculating the phase consistency feature, local contrast feature and color saturation feature of each aligned frame to be processed, and calculating the The phase consistency feature, local contrast feature and color saturation feature of the reference frame; according to the phase consistency feature, local contrast feature and color saturation feature of each aligned frame to be processed, calculate each aligned the initial weight map of the frame to be processed, and calculate the initial weight map of the reference frame according to the phase consistency feature, local contrast feature and color saturation feature of the reference frame; The initial weight map is normalized to obtain the weight map of each aligned frame to be processed, and the initial weight map of the reference frame is normalized to obtain the weight map of the reference frame.

In combination with the first aspect, an embodiment of the present invention provides a fourth possible implementation manner of the first aspect, wherein optimizing the weight map of each aligned frame to be processed by using the reference frame includes: using guided filtering The device uses the reference frame to optimize the weight map of each aligned frame to be processed through the following formula;

${\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; ;</span>$

${\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&omega;</span>}<span\; class="notranslate">\; |</span>}\underset{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; :</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\frac{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$

where i and j represent pixel indices, G represents the reference frame, Represents the filter kernel, p _j represents the weight map of the frame to be processed after alignment, q _i represents the optimized weight map, ω _z represents the local window whose center is z, |ω| represents the number of pixels at ω _z , μ _z and respectively represent the mean and variance of the reference frame on ω _z , ε represents the regularization parameter, G _i is the i-th pixel in the reference frame G, and G _j represents the j-th pixel in the reference frame G pixels.

In combination with the first aspect, an embodiment of the present invention provides a fifth possible implementation manner of the first aspect, wherein merging each optimized weight map with the weight map of the reference frame to obtain a fused image includes: The formula fuses each optimized weight map with the weight map of the reference frame to obtain a fused image;

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ;</span>$

Among them, F represents the obtained fused image, k represents the serial number of each frame, N represents the total number of frames, Denotes the optimized weight map of the kth frame, and f _k denotes the matrix of the kth frame.

In the second aspect, an embodiment of the present invention provides a multi-exposure video fusion device, the device includes: an alignment module, used to determine a reference frame and at least one frame to be processed in the acquired multi-exposure image sequence, according to the consistency sensitive The hash algorithm respectively aligns each of the frames to be processed to the reference frame; the weight map calculation module is used to calculate the weight of each aligned frame to be processed according to the image features of each aligned frame to be processed , and calculate the weight map of the reference frame according to the image features of the reference frame; the fusion module is used to optimize the weight map of each aligned frame to be processed by using the reference frame, and optimize each optimized The weight map of the reference frame is fused with the weight map of the reference frame to obtain a fused image.

In combination with the second aspect, the embodiment of the present invention provides a first possible implementation manner of the second aspect, wherein the alignment module includes: a scale decomposition unit configured to separately perform Scale decomposition to obtain each subframe set to be processed corresponding to each frame to be processed, and a reference subframe set corresponding to the reference frame; Each reference subframe and each subframe to be processed in each subframe set to be processed are divided into blocks; an alignment unit is used to divide each subframe in each subframe set to be processed according to a consistency sensitive hash algorithm. The block-to-be-processed subframes are aligned to each corresponding block-blocked reference subframe in the reference subframe set; an optimization unit is configured to use a grayscale mapping function for each of the to-be-processed subframes Each aligned sub-frame to be processed in the set is subjected to mis-match correction, and a Poisson video fusion method is used to optimize the result of the mis-match correction; Each optimized subframe to be processed in the set is reconstructed to obtain each aligned frame to be processed.

With reference to the first possible implementation manner of the second aspect, the embodiment of the present invention provides a second possible implementation manner of the second aspect, wherein the alignment unit includes: a neighborhood image block determination subunit, configured to Determine the image block to be matched in the subframe to be processed after the current block in the set of subframes to be processed, and determine the image block to be matched in the reference subframe after the current block in the set of reference subframes matching a plurality of neighborhood image blocks corresponding to the image block; the image block alignment subunit is used to calculate the matching distance between the image block to be matched and each of the neighborhood image blocks according to the consistency sensitive hash algorithm, and The image block to be matched is aligned to the adjacent image block corresponding to the minimum matching distance; the repeating subunit is used to repeat the determination of the image block to be matched, the determination of the adjacent image block, the calculation of the matching distance, and the image block to be matched The alignment operation is until the block-to-be-processed subframes in each to-be-processed subframe set are aligned to each corresponding block-blocked reference subframe in the reference subframe set.

In combination with the second aspect, an embodiment of the present invention provides a third possible implementation manner of the second aspect, wherein the weight map calculation module includes: a feature calculation unit, configured to calculate the weight of each aligned frame to be processed phase consistency features, local contrast features, and color saturation features, and calculating the phase consistency features, local contrast features, and color saturation features of the reference frame; the initial weight map calculation unit is used to calculate according to each aligned The phase consistency feature, local contrast feature and color saturation feature of the frame to be processed respectively calculate the initial weight map of each aligned frame to be processed, and according to the phase consistency feature, local contrast feature and color saturation feature of the reference frame The saturation feature calculates the initial weight map of the reference frame; the weight map normalization unit is used to normalize the initial weight maps of the frames to be processed after each alignment, and obtain the frame to be processed after each alignment. The weight map of the frame is processed, and the initial weight map of the reference frame is normalized to obtain the weight map of the reference frame.

The method and device in the embodiments of the present invention first determine the reference frame and at least one frame to be processed in the acquired multi-exposure image sequence, respectively align each frame to be processed to the reference frame according to the consistency sensitive hash algorithm, and then according to each Calculate the weight map of each aligned frame to be processed according to the image features of the aligned frames to be processed, and calculate the weight map of the reference frame according to the image features of the reference frame, and finally use the reference frame to calculate the weight map of each aligned frame to be processed Optimization is performed, and each optimized weight map is fused with the weight map of the reference frame to obtain a fused image. Since the method and device in the embodiment of the present invention align each frame to be processed to the reference frame according to the consistency-sensitive hash algorithm, the multi-exposure image sequence can be fused for the case where there is a moving target in the captured scene, A fused image that is clear, blur-free and ghost-free is obtained.

In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments are specifically cited below, together with the accompanying drawings, to be described in detail as follows.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, so it is not It should be regarded as a limitation on the scope, and those skilled in the art can also obtain other related drawings based on these drawings without creative work.

FIG. 1 shows a schematic flow diagram of a multi-exposure video fusion method provided by an embodiment of the present invention;

FIG. 2 shows a schematic structural diagram of a multi-exposure video fusion device provided by an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is a part of embodiments of the present invention, but not all embodiments. The components of the embodiments of the invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of the present invention.

Considering that the fusion methods in the prior art are all proposed for static scenes, without considering the impact of moving objects on the fusion effect, when there are moving objects in the captured scene, the fused image obtained by the fusion method of the prior art There are blurring and ghosting, and a complete and clear fusion image cannot be obtained. The present invention provides a multi-exposure video fusion method and device, which will be described in detail below with reference to the embodiments. It should be noted that, in this embodiment, in order to simplify the description language, the video frame is referred to as a frame for short, that is, a reference frame is equivalent to a reference video frame, and a frame to be processed is equivalent to a video frame to be processed.

Fig. 1 shows a schematic flow chart of a multi-exposure video fusion method provided by an embodiment of the present invention. As shown in Fig. 1, the method includes the following steps:

In step S102, a reference frame and at least one frame to be processed are determined in the acquired multi-exposure image sequence, and each frame to be processed is respectively aligned to the reference frame according to a consistency-sensitive hash algorithm.

In this embodiment, considering that when there are moving objects in the acquired multi-exposure image sequence, the final fused image may have blur and ghosting, so it is necessary to select the reference frame and the frame to be processed, and align each frame to be processed to the reference frame , to avoid possible blurring and ghosting of the fused image.

In this embodiment, the video frame with the least oversaturated pixels or the least undersaturated pixels is selected as the reference frame in the multi-exposure image sequence, and in one case, the video frames other than the reference frame in the multi-exposure image sequence are used as the to-be-processed frame. In another case, considering the large number of video frames in the multi-exposure image sequence and the large amount of calculation, or there are video frames that do not need to be processed in the multi-exposure image sequence, select some video frames in the multi-exposure image sequence as a pending frame.

Aligning each frame to be processed to the reference frame according to the consistency-sensitive hash algorithm specifically includes the following four steps (1) to (4):

(1) Scale decomposition is performed on each frame to be processed and the reference frame to obtain each subframe set to be processed corresponding to each frame to be processed and a reference subframe set corresponding to the reference frame.

Define the reference frame as R, the frame to be processed as S, and decompose the scale of each frame to be processed by the following formula (1) to obtain each subframe set to be processed corresponding to each frame to be processed, the subframe set to be processed includes For the multiple subframes to be processed obtained after the scale decomposition of the frame to be processed, the reference frame is decomposed by the following formula (2), and the reference subframe set corresponding to the reference frame is obtained. The reference subframe set includes the scale decomposition of the reference frame The obtained multiple reference subframes.

P(x,y,R,σ-1)=P _2↓ (x,y,R,σ); (1)

P(x,y,S,σ-1)=P _2↓ (x,y,S,σ); (2)

In formulas (1) and (2), {P(x,y,Δ,σ-1)Δ=R,S} represents the scale function, (x,y) represents the pixel position, σ represents the pyramid level, and the value of σ The range is [3, 5], and 2↓ represents the downsampling factor.

(2) Divide each reference subframe in the reference subframe set and each subframe to be processed in each subframe set to be processed into blocks respectively, and the blocking manner may be a sliding window block.

(3) Align each divided subframe to be processed in each subframe set to be processed to each corresponding divided reference subframe in the reference subframe set according to the consistency sensitive hash algorithm.

For example, a set of subframes to be processed A includes five subframes A1, A2, A3, A4, and A5 to be processed in order of image size from small to large. Five reference subframes B1, B2, B3, B4, and B5 arranged in order from small to large, after block A1, A2, A3, A4, A5, B1, B2, B3, B4, and B5, align A1 Align to B1, A2 is aligned to B2 ... A5 is aligned to B5, that is, the sub-frame to be processed after being divided is aligned to the reference sub-frame after being divided into blocks with the same image size.

In this action (3), according to the consistency-sensitive hash algorithm, each divided subframe to be processed in each subframe set to be processed is aligned to each corresponding divided reference subframe in the reference subframe set. Frames can be decomposed into the following actions:

(31) Determine the image block to be matched in the subframe to be processed after the current block in the current subframe set to be processed, and determine the corresponding image block to be matched in the reference subframe after the current block in the reference subframe set Multiple neighborhood image blocks of .

Neighboring image blocks can be determined in the following manner: after the first image block to be matched is determined in the subframe to be processed after the current block, an image matching the first image block to be matched is determined in the reference subframe Block, called the first matching image block, determine the second image block to be matched in the neighborhood of the first image block to be matched, and use multiple image blocks in the neighborhood of the first matching image block as the first A plurality of neighboring image blocks corresponding to the two to-be-matched image blocks are searched for an image block matching the second to-be-matched image block in the plurality of neighboring image blocks. That is, the image blocks in the neighborhood of the last matched image block in the reference subframe are used as the neighborhood image blocks of the current to-be-matched image block.

(32) Calculate the matching distance between the image block to be matched and each neighborhood image block according to the consistency-sensitive hash algorithm, and align the image block to be matched to the neighborhood image block corresponding to the smallest matching distance.

Calculate the matching distance between the image block to be matched and each neighborhood image block by the following formula (3):

d _y =||xy||=min{||x'-y||;x'∈S}; (3)

Among them, x' represents each neighborhood image block, y represents the image block to be matched, x represents the neighborhood image block with the smallest matching distance with y, d _y represents the calculated minimum matching distance, and S represents the distance of each neighborhood image block gather.

(33) Repeat the above-mentioned actions of determining the image block to be matched, determining the adjacent image block, calculating the matching distance, and aligning the image blocks to be matched until the subframes to be processed after each block in each subframe set to be processed are aligned to Each corresponding divided reference subframe in the reference subframe set.

In this step, the above-mentioned actions of determining the image block to be matched, determining the neighborhood image block, calculating the matching distance, and aligning the image blocks to be matched can be repeated, and the subframes to be processed after each block in the current set of subframes to be processed are Align to each corresponding block reference subframe in the reference subframe set, and then repeat the above actions in other subframe sets to be processed until each block to be processed in each subframe set to be processed The subframes are aligned to each corresponding segmented reference subframe in the set of reference subframes.

Through actions (31) to (33), each divided subframe to be processed in each subframe set to be processed can be aligned to each corresponding divided reference subframe in the reference subframe set.

(4) The grayscale mapping function is used to perform mismatch correction on each aligned subframe to be processed in each subframe set to be processed, and the Poisson video fusion method is used to optimize the result of the mismatch correction.

Taking the currently aligned subframes to be processed as an example, each image block in the currently aligned subframes to be processed corresponds to a matching distance, and each matching distance is detected. When it is detected that there is a matching distance that does not meet the preset requirements, Perform mismatch correction on the currently aligned subframes to be processed, and optimize the result of the mismatch correction by using the Poisson video fusion method. A distance threshold is set, and when the minimum matching distance obtained by the above calculation is greater than the distance threshold, it is determined that the minimum matching distance does not meet the preset requirement.

In this step, the grayscale mapping function is used to fill the initial holes, and the mismatch correction is performed on the subframes to be processed after the current alignment to improve the matching accuracy. The grayscale mapping function is defined as follows:

${\mathrm{<span\; class="notranslate">\; \&tau;</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; \&tau;</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&tau;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}^{\mathrm{<span\; class="notranslate">\; w</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Among them, τ _c ′( )≥0, τ _c ( )∈[0,1], c∈{r,g,b}, Represents the value of the color channel of the subframe to be processed after the current alignment, R _c represents the value of the color channel of the reference subframe corresponding to the subframe to be processed after the current alignment, where the value of the color channel refers to the r, g, b color channel The value of , that is, the red channel component, the green channel component and the blue channel component, (x, y) represents the pixel position, and the initial value τ _c of the grayscale mapping function is set as the reference subframe corresponding to the subframe to be processed after the current alignment In order to eliminate the fitting deviation caused by abnormalities, the iterative weighted least squares method is preferably used to achieve optimal solution.

The value of each color channel of the subframe to be processed after the current alignment can be updated through the formula (4), and the subframe to be processed after the value of the color channel is updated is the result of mismatch correction.

Using the Poisson video fusion method to optimize the result of the mis-match correction refers to extracting the gradient information of the reference subframe corresponding to the result of the mis-match correction, and using the Poisson video fusion method to combine the extracted gradient information with the above-mentioned mis-match correction. The results are synthesized to optimize the results of the mismatch correction. Wherein, the gradient information of the reference subframe corresponding to the mismatch correction result may also be replaced with the gradient information of the current subframe to be processed (that is, the subframe to be processed without alignment).

(5) Reconstruct each optimized subframe to be processed in each subframe set to be processed to obtain each aligned frame to be processed.

Taking frame A` to be processed as an example, frame A` to be processed corresponds to subframe set A to be processed, and subframe set A to be processed includes five subframes to be processed A1, A1, A2, A3, A4, and A5, A1, A2, A3, A4, and A5 are multiple sub-images obtained after size decomposition of the frame A' to be processed. After performing the optimization operation in action (4) on A1, A2, A3, A4, and A5, reconstruct the aligned A1, A2, A3, A4, and A5, that is, the aligned A1, A2, A3, and A4 Synthesize with A5 to obtain the aligned frame A0 to be processed.

Through the above actions (1) to (5), each frame to be processed can be aligned to the reference frame according to the consistency sensitive hash algorithm, and each frame to be processed after alignment can be obtained, so as to avoid moving objects in the multi-exposure image sequence When , there are blurring and ghosting problems in the fused image.

Step S104, calculating the weight map of each aligned frame to be processed according to the image features of each aligned frame to be processed, and calculating the weight map of the reference frame according to the image feature of the reference frame.

This step can be specifically decomposed into the following three steps:

(1) Calculate the phase consistency feature, local contrast feature and color saturation feature of each aligned frame to be processed, and calculate the phase consistency feature, local contrast feature and color saturation feature of the reference frame.

In order to measure the importance of each pixel in the video frame and obtain the weight map of the video frame, this embodiment proposes a feature-based weight estimation method. When the frames to be processed are aligned, three image features are integrated to estimate the weight of the pixel, the three image features include: phase consistency feature, local contrast feature and color saturation. By integrating three image feature metrics, the weights of corresponding pixels at different exposure scales can be accurately measured.

Calculating the phase consistency features, local contrast features, and color saturation features of each aligned frame to be processed is the same as calculating the phase consistency features, local contrast features, and color saturation features of the reference frame. The following is to calculate the current alignment. The phase consistency feature, local contrast feature and color saturation feature of the frame to be processed are taken as an example to illustrate.

Convert the currently aligned frames to be processed from the RGB color space to the YIQ color space, where Y represents brightness information, and I and Q represent chrominance information. The conversion from RGB space to YIQ space can be realized by formula (5):

$\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; Y</span>}\\ \mathrm{<span\; class="notranslate">\; I</span>}\\ \mathrm{<span\; class="notranslate">\; Q</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\left[\begin{array}{ccc}\mathrm{<span\; class="notranslate">\; 0.299</span>}& \mathrm{<span\; class="notranslate">\; 0.587</span>}& \mathrm{<span\; class="notranslate">\; 0.114</span>}\\ \mathrm{<span\; class="notranslate">\; 0.596</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 0.274</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 0.322</span>}\\ \mathrm{<span\; class="notranslate">\; 0.211</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 0.523</span>}& \mathrm{<span\; class="notranslate">\; 0.312</span>}\end{array}\right]\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; R</span>}\\ \mathrm{<span\; class="notranslate">\; G</span>}\\ \mathrm{<span\; class="notranslate">\; B</span>}\end{array}\right]<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

In formula (5), RGB represents the red channel component value, the green channel component value and the blue channel component value, respectively.

In order to calculate the phase consistency feature of the currently aligned frame to be processed, the brightness component Y of the currently aligned frame to be processed is convolved with a two-dimensional log-Gabor filter, and the transformation function of the two-dimensional log-Gabor filter is calculated as follows:

$\mathrm{<span\; class="notranslate">\; L</span>}\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&omega;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; log</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&omega;</span>}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

Where, ω ₀ is the center frequency of the filter, θ _j = jπ/J, j = {0,1,...,J-1} is the direction angle of the filter, J is the direction number, and σ _r controls the bandwidth of the filter , σ _θ determines the angular bandwidth of the filter, θ represents the angle at which the two filters are tangent, J=4 is set according to experience, and the ω ₀ of the four scales are 1/6, 1/12, 1/24, 1/ 48, σ _r =0.5978, σ _θ =0.6545. The convolution result generates a set of orthogonal vectors [e _n,o (x,y),o _n,o (x,y)] (scale n, direction o, pixel position (x,y)), and obtains the local amplitude calculation as follows:

${\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; o</span>}}<span\; class="notranslate">\; =</span>\sqrt{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; o</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; o</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

Then, the phase consistency feature at the position (x, y) of the frame to be processed after the current alignment is calculated as follows:

${\mathrm{<span\; class="notranslate">\; PC</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; o</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\sqrt{{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; o</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; o</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; o</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}{\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; o</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\underset{\mathrm{<span\; class="notranslate">\; no</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; o</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 8</span>}<span\; class="notranslate">\; )</span>$

Among them, ε is a small normal quantity, and the value of the phase consistency characteristic PC _k (x, y) is between 0 and 1. The closer PC _k (x, y) is to 1, the more significant the feature is.

Because the phase consistency feature is contrast-invariant, and the contrast information affects the perception of video quality by the human visual system, local contrast is used as the second measure to complement the phase consistency feature.

In this embodiment, the local contrast feature is measured using the gradient energy of the video frame, and the gradient metric assigns higher weights to important detail features, such as edges and textures. Video frame gradients can be computed using convolution masks. In this embodiment, the Sobel operator is used as the gradient operator, and the Sobel operator is calculated as follows:

$\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 4</span>}}\left[\begin{array}{ccc}\mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 2</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 2</span>}\\ \mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]<span\; class="notranslate">\; ,</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 4</span>}}\left[\begin{array}{ccc}\mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; 2</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 2</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 9</span>}<span\; class="notranslate">\; )</span>$

Along the horizontal and vertical directions of the video frame, the luminance channel Y of the currently aligned frame to be processed is convolved with the Sobel operator to obtain two directional gradients G _X and G _Y , and the local area of the currently aligned frame to be processed The contrast feature G _K is calculated as follows:

${\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; =</span>\sqrt{{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span>$

Both the phase consistency feature and the local contrast feature are only measured on the luminance channel. For color video frames, if only phase consistency and local contrast are used as metrics, an accurate weight map cannot be obtained, so color saturation is used as the second measure. three features. The calculation of the color saturation feature S _k is defined as the standard deviation of each pixel in the R, G, and B channels, and is calculated as follows:

${\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; =</span>\sqrt{\frac{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 3</span>}}}<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 11</span>}<span\; class="notranslate">\; )</span>$

$\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; )</span>$

in, Represents the average of the three RGB color channels.

In this step, the phase consistency features, local contrast features, and color saturation features of each aligned frame to be processed are calculated through the process shown in the above formula (5) to formula (12), and the phase consistency features of the reference frame , local contrast feature and color saturation feature, the following step (2) is performed.

(2) Calculate the initial weight map of each aligned frame to be processed according to the phase consistency feature, local contrast feature and color saturation feature of each aligned frame to be processed, and according to the phase consistency feature of the reference frame, local Contrast features and color saturation features calculate the initial weight map of the reference frame.

The process of calculating the initial weight map of the frame to be processed after alignment is the same as the process of calculating the initial weight map of the reference frame, so the calculation of the initial weight map of the frame to be processed after alignment is taken as an example for illustration.

Considering that the three metrics of phase consistency feature, local contrast feature and color saturation feature are complementary, the weight of the video frame is estimated by combining the three image features by direct multiplication, as shown in formula (13) Show,

W _k = PC _k × G _k × S _k ; (13)

Among them, W _k represents the initial weight map of the frame to be processed after the alignment, PC _k represents the phase consistency feature of the frame to be processed after the alignment, G _k represents the local comparison feature of the frame to be processed after the alignment, and S _k represents the alignment The color saturation feature of the frame to be processed later. In this way, the resulting fused image preserves all important details of the original video sequence.

(3) Normalize the initial weight map of each aligned frame to be processed to obtain the weight map of each aligned frame to be processed, and normalize the initial weight map of the reference frame to obtain the weight map of the reference frame weight map.

In this step, the normalization process of the initial weight map of each aligned frame to be processed is the same as the normalization process of the initial weight map of the reference frame. The normalization process is described as an example.

Assuming that the number of video frames in the multi-exposure sequence is N, in order to maintain the consistency of the fusion results, the N weight maps are normalized to ensure that the weight sum at each pixel (x, y) is 1, normalized Normalized initial weight map defined as:

${\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; W</span>}}_{{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 14</span>}<span\; class="notranslate">\; )</span>$

Among them, (x, y) represents the pixel position, Indicates the normalized initial weight map, W _k' (x, y) represents the initial weight map, k represents the serial number of the current initial weight map, k' represents the serial number of each initial weight map, and the value range of k and k' is [1, N].

Through step S104, the feature-based weight estimation method integrates the three image features of phase consistency, local contrast and color saturation, accurately measures the quality of pixels, and obtains a weight map.

Step S106 , using the reference frame to optimize the weight map of each aligned frame to be processed, and fusing each optimized weight map with the weight map of the reference frame to obtain a fused image.

In this step, the weight map of each aligned frame to be processed is optimized by using the reference frame, including:

Using the guidance filter to optimize the weight map of each aligned frame to be processed by using the reference frame through the following formulas (15) and (16);

${\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 15</span>}<span\; class="notranslate">\; )</span>$

${\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&omega;</span>}<span\; class="notranslate">\; |</span>}\underset{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; :</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\frac{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 16</span>}<span\; class="notranslate">\; )</span>$

Among them, i and j represent the pixel index, G represents the reference frame, Represents the filter kernel, p _j represents the weight map of the frame to be processed after alignment, q _i represents the optimized weight map, ω _z represents the local window centered at z, |ω| represents the number of pixels in ω _z , μ _z and Denote the mean and variance of the reference frame on ω _z , respectively, ε represents the regularization parameter, providing a criterion for "planar block" or "high variance/edge", the impact of ε is similar to the range variance in bilateral filters G _i is the i-th pixel in the reference frame G, and G _j is the j-th pixel in the reference frame G. The above filter kernel satisfy

In the above optimization process, the weight map p _j of the reference frame G and the aligned frames to be processed are all set as the weight map calculated in step S104.

The optimized weight map obtained by the above formula (15) and formula (16) is smoother, does not contain discontinuous regions, and can more accurately represent the importance of pixels.

In this step, each optimized weight map is fused with the weight map of the reference frame to obtain a fused image, including:

Each optimized weight map and the weight map of the reference frame are fused by the following formula to obtain a fused image;

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ;</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 17</span>}<span\; class="notranslate">\; )</span>$

Among them, F represents the obtained fused image, k represents the serial number of each frame, N represents the total number of frames, Denotes the optimized weight map of the kth frame, and f _k denotes the matrix of the kth frame.

The optimized weight map indicates which pixels in the video frame are well-exposed, so that the fusion result includes all well-exposed pixels, resulting in a fused image with vivid visual effects.

The method in the embodiment of the present invention first determines the reference frame and at least one frame to be processed in the acquired multi-exposure image sequence, aligns each frame to be processed to the reference frame according to the consistency sensitive hash algorithm, and then according to each aligned Calculate the weight map of each aligned frame to be processed according to the image features of the frame to be processed, and calculate the weight map of the reference frame according to the image feature of the reference frame, and finally use the reference frame to optimize the weight map of each aligned frame to be processed , each optimized weight map is fused with the weight map of the reference frame to obtain a fused image. Since the method in the embodiment of the present invention aligns each frame to be processed to the reference frame according to the consistency-sensitive hash algorithm, the multi-exposure image sequence can be fused to obtain a clear Fused images with no blur and no ghosting.

In summary, the method in the embodiment of the present invention has the following technical effects:

(1) The block matching video frame alignment method based on the consistency-sensitive hash algorithm can solve the problem of fusion image blur and ghosting caused by target motion in the captured scene, and finally obtain a high dynamic range, light Good full and clear scene video.

(2) The feature-based weight estimation method integrates three image features of phase consistency, local contrast and color saturation, which can accurately measure the quality of pixels to obtain the weight map.

(3) The weight map optimization method based on guided filtering improves the accuracy of the weight map and ensures that high-quality fusion images can be obtained.

Corresponding to the above-mentioned multi-exposure video fusion method, an embodiment of the present invention also provides a multi-exposure video fusion device, as shown in FIG. 2 , the device includes: an alignment module 21 for determining a reference frame and at least one frame to be processed, each frame to be processed is aligned to a reference frame according to the consistency sensitive hash algorithm; the weight map calculation module 22 is used to calculate each aligned frame according to the image features of each aligned frame to be processed The weight map of the frame to be processed, and calculate the weight map of the reference frame according to the image features of the reference frame; the fusion module 23 is used to optimize the weight map of each aligned frame to be processed by using the reference frame, and combine each optimized weight The weight map of the image and the reference frame are fused to obtain a fused image.

Wherein, the alignment module 21 includes: a scale decomposition unit, which is used to perform scale decomposition on each frame to be processed and the reference frame, respectively, to obtain each subframe set to be processed corresponding to each frame to be processed, and the reference subframe corresponding to the reference frame The frame set; the block unit, which is used to block each reference subframe in the reference subframe set and each pending subframe in each pending subframe set; the alignment unit, which is used according to the consistency sensitive hash The algorithm respectively aligns each divided subframe to be processed in each subframe set to be processed to each corresponding divided reference subframe in the reference subframe set; the optimization unit is used to use a grayscale mapping function to Each aligned sub-frame to be processed in each sub-frame set to be processed is subjected to mis-match correction, and a Poisson video fusion method is used to optimize the result of mis-match correction; the reconstruction unit is used to combine each sub-frame set to be processed Each optimized sub-frame to be processed is reconstructed to obtain each aligned frame to be processed.

The above-mentioned alignment unit includes: a neighboring image block determination subunit, which is used to determine the image block to be matched in the subframe to be processed after the current block in the current subframe set to be processed, and the current block in the reference subframe set In the subsequent reference subframe, a plurality of neighboring image blocks corresponding to the image block to be matched are determined; the image block alignment subunit is used to calculate the distance between the image block to be matched and each neighboring image block according to the consistency sensitive hash algorithm. Matching distance, aligning the image block to be matched to the adjacent image block corresponding to the minimum matching distance; repeating subunits, used to repeatedly determine the image block to be matched, determine the adjacent image block, calculate the matching distance, and align the image blocks to be matched Actions until each subframe to be processed after being divided into blocks in each set of subframes to be processed is aligned to each corresponding reference subframe after being divided into blocks in the set of reference subframes.

The above-mentioned weight map calculation module 22 includes: a feature calculation unit, which is used to calculate the phase consistency feature, local contrast feature and color saturation feature of each aligned frame to be processed, and calculate the phase consistency feature and local contrast feature of the reference frame. Features and color saturation features; an initial weight map calculation unit, used to calculate the initial weight map of each aligned frame to be processed according to the phase consistency feature, local contrast feature and color saturation feature of each aligned frame to be processed , and calculate the initial weight map of the reference frame according to the phase consistency feature, local contrast feature and color saturation feature of the reference frame; the weight map normalization unit is used to perform the initial weight map of each aligned frame to be processed respectively Normalize to obtain the weight map of each aligned frame to be processed, and normalize the initial weight map of the reference frame to obtain the weight map of the reference frame.

The above fusion module 23 includes: a weight map optimization unit, which is used to optimize the weight map of each aligned frame to be processed by using the reference frame through the following formula by using the guidance filter;

${\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; ;</span>$

${\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&omega;</span>}<span\; class="notranslate">\; |</span>}\underset{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; :</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\frac{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$

Among them, i and j represent the pixel index, G represents the reference frame, Represents the filter kernel, p _j represents the weight map of the frame to be processed after alignment, q _i represents the optimized weight map, ω _z represents the local window centered at z, |ω| represents the number of pixels in ω _z , μ _z and denote the mean and variance of the reference frame on ω _z , respectively, ε denotes the regularization parameter, G _i is the i-th pixel in the reference frame G, and G _j is the j-th pixel in the reference frame G.

The above-mentioned fusion module 23 includes: an image fusion unit, which is used to fuse each optimized weight map and the weight map of the reference frame by the following formula to obtain a fusion image;

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ;</span>$

Among them, F represents the obtained fused image, k represents the serial number of each frame, N represents the total number of frames, Denotes the optimized weight map of the kth frame, and f _k denotes the matrix of the kth frame.

The device in the embodiment of the present invention firstly determines the reference frame and at least one frame to be processed in the acquired multi-exposure image sequence, respectively aligns each frame to be processed to the reference frame according to the consistency sensitive hash algorithm, and then according to each aligned Calculate the weight map of each aligned frame to be processed according to the image features of the frame to be processed, and calculate the weight map of the reference frame according to the image feature of the reference frame, and finally use the reference frame to optimize the weight map of each aligned frame to be processed , each optimized weight map is fused with the weight map of the reference frame to obtain a fused image. Since the device in the embodiment of the present invention aligns each frame to be processed to the reference frame according to the consistency-sensitive hash algorithm, the multi-exposure image sequence can be fused to obtain a clear Fused images with no blur and no ghosting.

The multi-exposure image fusion apparatus provided by the embodiment of the present invention may be specific hardware on the device or software or firmware installed on the device. The implementation principles and technical effects of the device provided by the embodiment of the present invention are the same as those of the foregoing method embodiment. For brief description, for the parts not mentioned in the device embodiment, reference may be made to the corresponding content in the foregoing method embodiment. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the above-mentioned method embodiments, and will not be repeated here.

In the embodiments provided in the present invention, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments provided by the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes. .

It should be noted that like numerals and letters denote similar items in the following drawings, therefore, once an item is defined in one drawing, it does not require further definition and explanation in subsequent drawings, In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions, and should not be construed as indicating or implying relative importance.

Finally, it should be noted that: the above-described embodiments are only specific implementations of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting them, and the scope of protection of the present invention is not limited thereto, although referring to the foregoing The embodiment has described the present invention in detail, and those skilled in the art should understand that any person familiar with the technical field can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention Changes can be easily imagined, or equivalent replacements can be made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (10)

1. A multi-exposure video fusion method, characterized in that the method comprises: Determining a reference frame and at least one frame to be processed in the acquired multi-exposure image sequence, respectively aligning each frame to be processed to the reference frame according to a consistency-sensitive hash algorithm; calculating a weight map of each aligned frame to be processed according to image features of each aligned frame to be processed, and calculating a weight map of the reference frame according to image features of the reference frame; Optimizing each aligned weight map of the frame to be processed by using the reference frame, and merging each optimized weight map with the weight map of the reference frame to obtain a fused image. 2. The method according to claim 1, wherein aligning each of the frames to be processed to the reference frame according to a consistency-sensitive hash algorithm comprises: respectively performing scale decomposition on each of the frames to be processed and the reference frame to obtain each subframe set to be processed corresponding to each frame to be processed and a reference subframe set corresponding to the reference frame; Block each reference subframe in the set of reference subframes and each subframe to be processed in each set of subframes to be processed; According to the consistency-sensitive hash algorithm, respectively align each subframe to be processed in each subframe set to be processed after being divided into blocks to the reference subframe after each corresponding block in the reference subframe set subframe; Using a grayscale mapping function to perform mismatch correction on each of the aligned subframes in each subframe set to be processed, and using a Poisson video fusion method to optimize the result of the mismatch correction; Reconstruct each optimized subframe to be processed in each set of subframes to be processed to obtain each aligned frame to be processed. 3. The method according to claim 2, wherein, according to a consistency-sensitive hash algorithm, the subframes to be processed after each block in each subframe set to be processed are respectively aligned to the reference The reference subframe after each corresponding block in the subframe set includes: Determine the image block to be matched in the subframe to be processed after the current block in the current subframe set to be processed, and determine the image block to be matched in the reference subframe after the current block in the reference subframe set a plurality of neighboring image blocks corresponding to the image block to be matched; Calculate the matching distance between the image block to be matched and each of the neighborhood image blocks according to the consistency sensitive hash algorithm, and align the image block to be matched to the neighborhood image block corresponding to the smallest matching distance ; Repeating the actions of determining image blocks to be matched, determining neighboring image blocks, calculating matching distances, and aligning image blocks to be matched, until each of the subframes to be processed is divided into blocks in each set of subframes to be processed. Aligning to each of the corresponding divided reference subframes in the set of reference subframes. 4. The method according to claim 1, characterized in that, calculating the weight map of each aligned frame to be processed according to the image features of each aligned frame to be processed, and according to the image of the reference frame A feature computes a weight map for the reference frame, comprising: respectively calculating phase consistency features, local contrast features, and color saturation features of each aligned frame to be processed, and calculating phase consistency features, local contrast features, and color saturation features of the reference frame; Calculate the initial weight map of each aligned frame to be processed according to the phase consistency feature, local contrast feature and color saturation feature of each aligned frame to be processed, and according to the phase consistency of the reference frame feature, local contrast feature and color saturation feature calculate the initial weight map of the reference frame; Respectively normalize the initial weight map of each aligned frame to be processed to obtain the weight map of each aligned frame to be processed, and normalize the initial weight map of the reference frame to obtain A weight map for the reference frame. 5. The method according to claim 1, wherein the weight map of each aligned frame to be processed is optimized using the reference frame, comprising: Using the guide filter to optimize the weight map of each aligned frame to be processed by using the reference frame through the following formula; ${\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; ;</span>$ ${\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&omega;</span>}<span\; class="notranslate">\; |</span>}\underset{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; :</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\frac{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$ where i and j represent pixel indices, G represents the reference frame, Represents the filter kernel, p _j represents the weight map of the frame to be processed after alignment, q _i represents the optimized weight map, ω _z represents the local window whose center is z, |ω| represents the number of pixels at ω _z , μ _z and respectively represent the mean and variance of the reference frame on ω _z , ε represents the regularization parameter, G _i is the i-th pixel in the reference frame G, and G _j represents the j-th pixel in the reference frame G pixels. 6. The method according to claim 1, characterized in that merging each optimized weight map with the weight map of the reference frame to obtain a fusion image, comprising: Each optimized weight map is fused with the weight map of the reference frame by the following formula to obtain a fused image; $\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ;</span>$ Among them, F represents the obtained fusion image, k represents the serial number of each frame, N represents the total number of frames, Denotes the optimized weight map of the kth frame, and f _k denotes the matrix of the kth frame. 7. A multi-exposure video fusion device, characterized in that the device comprises: An alignment module, configured to determine a reference frame and at least one frame to be processed in the acquired multi-exposure image sequence, and respectively align each frame to be processed to the reference frame according to a consistency-sensitive hash algorithm; A weight map calculation module, configured to calculate a weight map of each aligned frame to be processed according to image features of each aligned frame to be processed, and calculate a weight of the reference frame according to image features of the reference frame picture; The fusion module is configured to use the reference frame to optimize the weight map of each aligned frame to be processed, and fuse each optimized weight map with the weight map of the reference frame to obtain a fusion image. 8. The device according to claim 7, wherein the alignment module comprises: A scale decomposition unit, configured to perform scale decomposition on each of the frames to be processed and the reference frame to obtain each subframe set to be processed corresponding to each frame to be processed, and a reference frame corresponding to the reference frame set of subframes; a blocking unit, configured to block each reference subframe in the set of reference subframes and each subframe to be processed in each set of subframes to be processed; An alignment unit, configured to respectively align the to-be-processed subframes after each block in each of the to-be-processed subframe sets to each corresponding block in the reference subframe set according to a consistency-sensitive hash algorithm the subsequent reference subframe; An optimization unit, configured to use a grayscale mapping function to perform mismatch correction on each of the aligned subframes in each subframe set to be processed, and use a Poisson video fusion method to correct the mismatch results are optimized; The reconstruction unit is configured to reconstruct each optimized subframe to be processed in each set of subframes to be processed to obtain each aligned frame to be processed. 9. The device according to claim 8, wherein the alignment unit comprises: The neighborhood image block determination subunit is used to determine the image block to be matched in the subframe to be processed after the current block in the current subframe set to be processed, and the current block in the reference subframe set determining a plurality of neighboring image blocks corresponding to the image block to be matched in the reference subframe after the block; The image block alignment subunit is used to calculate the matching distance between the image block to be matched and each of the neighboring image blocks according to the consistency sensitive hash algorithm, and align the image block to be matched to the smallest Match the neighborhood image blocks corresponding to the distance; The repeating subunit is used to repeat the actions of determining the image block to be matched, determining the adjacent image block, calculating the matching distance, and aligning the image blocks to be matched until each sub-frame set to be processed is divided into blocks. The to-be-processed subframes are aligned to corresponding block-divided reference subframes in the set of reference subframes. 10. The device according to claim 7, wherein the weight map calculation module comprises: A feature calculation unit, configured to calculate phase consistency features, local contrast features, and color saturation features of each aligned frame to be processed, and calculate phase consistency features, local contrast features, and color saturation features of the reference frame Degree feature; An initial weight map calculation unit, configured to calculate an initial weight map of each aligned frame to be processed according to the phase consistency feature, local contrast feature, and color saturation feature of each aligned frame to be processed, and according to calculating an initial weight map of the reference frame based on the phase consistency feature, local contrast feature, and color saturation feature of the reference frame; A weight map normalization unit, configured to normalize the initial weight maps of each aligned frame to be processed, to obtain the weight map of each aligned frame to be processed, and the weight map of the reference frame The initial weight map is normalized to obtain the weight map of the reference frame.