CN115841523A - Double-branch HDR video reconstruction algorithm based on Raw domain - Google Patents

Abstract

The invention discloses a double-branch HDR video reconstruction algorithm based on a Raw domain, belonging to the technical field of video signal processing; a double-branch HDR video reconstruction algorithm based on a Raw domain specifically comprises the following steps: s1, establishing a synthesized Raw domain video HDR data set; s2, designing a double-branch Raw video HDR reconstruction algorithm based on the S1; s3, training a model; s4, inputting the Raw video sequence of the Low Dynamic Range (LDR) in the test set into the model to obtain a corresponding output result of the high dynamic range; the invention synthesizes a first video HDR data set simulating real noise distribution in a Raw domain, and provides a reference data set for training and evaluating an HDR reconstruction method at night or in an extreme scene; meanwhile, the dynamic range expansion of the LDR video with noise under the difficult scene is realized by utilizing the proposed content enhancement module.

Description

A Dual-branch HDR Video Reconstruction Algorithm Based on Raw Domain

Technical Field

The present invention relates to the technical field of video signal processing, and in particular to a dual-branch HDR video reconstruction algorithm based on Raw domain.

Background Art

High dynamic range (HDR) technology uses multiple low dynamic range (LDR) images with different exposures to expand the dynamic range of an image, enrich image details and improve image contrast. The scene irradiance in natural scenes may vary from 10 ^-5 to 10 ⁹ , while the photos recorded by ordinary cameras may only have a bit depth of 8 or 10 bits, which cannot fully capture the brightness range of the scene. When the dynamic range captured by the camera is smaller than the dynamic range of the scene, the captured image may have overexposed or underexposed areas, affecting the visual quality.

With the development of HDR technology, the traditional statistical fusion method has gradually shifted to deep learning-based methods to reconstruct HDR images. These deep learning methods often go through two steps: one is to align LDR images at different times and exposures, and the other is to fuse the aligned LDR images into HDR images. Compared with image HDR reconstruction, video HDR reconstruction requires reconstruction of the HDR results of each frame in the original LDR frame sequence. Existing methods often use a sliding window-based method to input three or five adjacent LDR frames of different exposures for alternating exposure video sequences (such as -2EV, +2EV, -2EV, ...), and reconstruct the HDR results of the intermediate frames after alignment and fusion.

Previous video HDR reconstruction methods often directly process sRGB images. However, sRGB images have gone through a complex image processing pipeline (ISP) inside the camera, such as black level correction, de-mosaicing, white balance, gamma correction, and color gamut conversion. This not only causes the sRGB image to lose some of its original information, but also some nonlinear mapping operations make HDR reconstruction difficult. The above problems can be solved well by using the Raw domain data output by the camera sensor. Raw domain data contains a wider number of bits and contains richer scene information. At the same time, because it is not affected by subsequent ISP processing, Raw domain data has better linear properties and is more conducive to HDR reconstruction.

On the other hand, for HDR reconstruction of overly dark scenes, it is often necessary to consider the serious influence of noise. Raw domain data can model noise more accurately, so that the model can better learn to denoise real scenes and is widely used in image and video denoising tasks. At the same time, existing methods such as Deep HDR video from sequences with alternating exposures proposed by Kalantari et al. Computer Graphics Forum 38, 193–205 (2019) lack special designs for noise in overly dark areas, which makes the network perform poorly for HDR reconstruction of noisy images. HDR video reconstruction: A coarse-to-fine network and a real-world benchmark dataset. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2502–2511 (2021) proposed by Chen et al. uses two-stage processing of alignment and fusion, so that the details are over-smoothed when the short exposure image is used as a reference, while the noise is retained when the long exposure image is used as a reference.

Based on the above content, it can be seen that the reconstruction methods of HDR images and videos based on deep learning are often limited by noise. The existing methods mainly train images in the sRGB domain and cannot model the actual noise situation. At the same time, the network structure is mainly designed for alignment and fusion, and lacks a module for processing noise, resulting in low HDR reconstruction quality in difficult scenarios. In order to solve the above problems, the present invention proposes a dual-branch HDR video reconstruction algorithm based on the Raw domain.

Summary of the invention

The purpose of the present invention is to establish a Raw video HDR dataset, and on this basis propose a dual-branch HDR video reconstruction algorithm based on the Raw domain to solve the problems mentioned in the background technology.

In order to achieve the above object, the present invention adopts the following technical solutions:

A dual-branch HDR video reconstruction algorithm based on the Raw domain specifically includes the following steps:

S1. Establish a synthetic Raw domain video HDR dataset: Establish a Raw dataset, specifically including the following contents:

S101, source sRGB video data selection: 21 sRGB domain HDR videos shot by Froehlich and Kronander et al. and the high-quality video dataset Vimeo-90K are selected as source sRGB video data. Each HDR video is re-exposed with the selected exposure parameters to simulate the alternating exposure LDR video sequence;

S102, Raw data set establishment: by simulating the camera imaging pipeline process, the sRGB video is converted into a Raw video data set. The specific steps are: inverse camera response (CRF) curve, simulated bayer format, data enhancement, re-exposure, and adding noise to obtain HDR and LDR images in the Raw domain, which are used as training data pairs in the Raw domain;

和HDR的Raw帧

作为训练对来设计双支路Raw视频HDR重建算法；S2. Design reconstruction algorithm: Based on the data pairs obtained in S1, use the Raw frame of LDR

and Raw frames of HDR

Use them as training pairs to design a dual-branch Raw video HDR reconstruction algorithm;

S3, training model: build a model based on the reconstruction algorithm designed in S2, and use the deep learning framework Pytorch platform to train the model. Iterate 15 epcohs on the entire dataset, then reduce the learning rate to 0.00001, and continue to iterate until the loss converges to obtain the final model;

S4, output results: Input the low dynamic range Raw video sequence in the test set into the final model obtained in S3 to obtain the corresponding high dynamic range output results.

Preferably, the analog camera imaging process mentioned in S102 converts the sRGB video into a Raw video, which specifically includes the following steps:

S1021. For the Vimeo-90K dataset, the video frames are converted from the nonlinear domain to the linear domain by estimating the CRF curve.

S1022, down-sample the 3-channel sRGB frame into four channels of one-quarter resolution: red, green, green, and blue, and combine them into a mosaic image according to the GRBG bayer format;

S1023, converting the mosaic image into a 4-channel image of G, R, B, G, and then randomly scaling, translating, and rotating;

S1024, converting the Raw domain HDR frame into a Raw domain LDR frame according to specific exposure parameters;

S1025. Add simulated Gaussian and Poisson noise to the Raw domain LDR frame.

Preferably, the design of the dual-branch Raw video HDR reconstruction algorithm mentioned in S2 specifically includes the following steps:

利用噪声估计网络估计噪声水平图：S201, noise estimation: input three consecutive raw domain LDR frames each time

Estimate the noise level map using a noise estimation network:

和其对应的曝光系数t_i-1、t_i、t_i+1，利用曝光系数对输入LDR图像进行曝光校正，校正公式为：S202, data processing and feature extraction: input three consecutive raw domain LDR frames

And its corresponding exposure coefficients ti _-1 , _ti , ti ₊₁ , use the exposure coefficients to perform exposure correction on the input LDR image, the correction formula is:

映射到同一曝光水平；Using the above formula, input

Mapped to the same exposure level;

Then, through the feature extraction module, convolution is used to extract features:

Where _Fi represents the features extracted from the i-th frame, the input LDR image is used to help detect overexposed and underexposed areas, the input HDR image is used to help with subsequent alignment, and the noise level image helps detect noise areas;

S203, feature alignment: cascaded pyramid deformable convolution structure, first downsample the input features twice to obtain features of multiple scales:

表示下采样；in,

represents downsampling;

与中间帧特征

级联估计偏移量

At the sth scale, using

Intermediate frame features

Cascade Estimation Offset

作为可变形卷积的偏移量，用可变形卷积对上一帧特征处理后得到当前尺度下的对齐结果：The calculated offset

As the offset of the deformable convolution, the deformable convolution is used to process the features of the previous frame to obtain the alignment result at the current scale:

Among them, ↑2 represents 2 times bilinear interpolation upsampling, and the alignment result of each scale is further fused with the alignment result of the previous scale through convolution; the coarse-to-fine joint prediction in the feature domain can more accurately estimate the displacement at a large scale;

S204, Temporal Fusion: The spatial attention structure obtains the attention correlation between adjacent frames through convolution, helping the network to reconstruct HDR images without ghosting and with accurate exposure:

表示时间融合后的特征；Among them, _Ai represents the predicted spatial attention, ⊙ represents element-by-element multiplication,

Represents the features after time fusion;

经过残差估计分支REB提取输入特征的高频信息，帮助恢复上一支路中缺少的内容：S205, content enhancement branch: the aligned features

The residual estimation branch REB extracts high-frequency information of the input features to help restore the missing content in the previous branch:

Among them, H _res represents the estimated residual information, REB is the residual estimation branch, which contains multiple dense residual blocks;

经过一系列残差块、跳跃连接后和H_res相加，再经过sigmoid层得到最终的raw域HDR结果

S206, Reconstruct HDR:

After a series of residual blocks and jump connections, it is added to H _res and then passed through a sigmoid layer to obtain the final raw domain HDR result.

和真值

之间的差异化网络：S207, loss function: output after tone mapping

and truth value

Differentiation network between:

表示经过色调映射后的raw域真值图像和预测结果。Among them, μ is 5000, T _i ^raw and

Represents the raw domain true value image and prediction result after tone mapping.

Compared with the prior art, the present invention provides a dual-branch HDR video reconstruction algorithm based on the Raw domain, which has the following beneficial effects:

(1) This paper synthesizes the first video HDR dataset that simulates real noise distribution in the Raw domain, providing a benchmark dataset for the training and evaluation of HDR reconstruction methods at night or in extreme scenes;

(2) Based on the proposed Raw video HDR dataset, this paper proposes a dual-branch HDR video reconstruction method, and uses the proposed deformable convolution alignment module and content enhancement module to achieve dynamic range expansion of noisy LDR videos in difficult scenarios;

(3) The reconstruction algorithm proposed in the present invention is compared with the mainstream reconstruction method on the market. The results show that the reconstruction algorithm proposed in the present invention is superior to the current mainstream sRGB-based HDR reconstruction method, and is superior to or equivalent to the result of directly converting the mainstream method to the Raw domain. Through the research and exploration of the present invention, it is hoped that more research on Raw-domain-based video HDR reconstruction methods can be inspired.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is an algorithm flow chart of a dual-branch HDR video reconstruction algorithm based on the Raw domain proposed by the present invention;

FIG2 is a visual comparison diagram of the results of a dual-branch HDR video reconstruction algorithm based on the Raw domain proposed in the present invention and other video/image HDR reconstruction algorithms on a test set.

DETAILED DESCRIPTION

The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

Embodiment 1:

A dual-branch HDR video reconstruction algorithm based on Raw domain specifically includes the following steps:

S1. Establish a synthetic Raw domain video HDR dataset: Establish a Raw dataset, specifically including the following contents:

S101, source sRGB video data selection: 21 sRGB domain HDR videos shot by Froehlich and Kronander et al. and the high-quality video dataset Vimeo-90K are selected as source sRGB video data. Each HDR video is re-exposed with the selected exposure parameters to simulate the alternating exposure LDR video sequence;

S102, Raw data set establishment: by simulating the camera imaging pipeline process, the sRGB video is converted into a Raw video data set. The specific steps are: inverse camera response (CRF) curve, simulated bayer format, data enhancement, re-exposure, and adding noise to obtain HDR and LDR images in the Raw domain, which are used as training data pairs in the Raw domain;

S102 mentions that the analog camera imaging process converts sRGB video into Raw video, which specifically includes the following steps:

S1021. For the Vimeo-90K dataset, the video frames are converted from the nonlinear domain to the linear domain by estimating the CRF curve.

S1022, down-sample the 3-channel sRGB frame into four channels of one-quarter resolution: red, green, green, and blue, and combine them into a mosaic image according to the GRBG bayer format;

S1023, converting the mosaic image into a 4-channel image of G, R, B, G, and then randomly scaling, translating, and rotating;

S1024, converting the Raw domain HDR frame into a Raw domain LDR frame according to specific exposure parameters;

S1025. Add simulated Gaussian and Poisson noise to the Raw domain LDR frame.

Raw domain data has a different data format from sRGB, so it is necessary to first resample the original RGB three channels to RGGB four channels, and then rearrange them according to the Bayer format to simulate the Raw data format; secondly, add Gaussian and Poisson noise to the image to simulate the noise distribution in real situations;

和HDR的Raw帧

作为训练对来设计双支路Raw视频HDR重建算法；S2. Design reconstruction algorithm: Based on the data pairs obtained in S1, use the Raw frame of LDR

and Raw frames of HDR

Use them as training pairs to design a dual-branch Raw video HDR reconstruction algorithm;

S3, training model: build the model based on the reconstruction algorithm designed in S2, the model input frame number is 3 frames, the input video is cropped into 256 x 256 blocks, and each batch is 16 groups of sample data. The Adam optimizer is selected, and the initial learning rate is set to 0.0001. Use the deep learning framework Pytorch platform to train the model, iterate 15 epochs on the entire data set, then reduce the learning rate to 0.00001, and continue to iterate until the loss curve converges to obtain the final model;

S4, output results: Input the low dynamic range Raw video sequence in the test set into the final model obtained in S3 to obtain the corresponding high dynamic range output results.

The first video HDR dataset simulating real noise distribution is synthesized in the Raw domain, providing a benchmark dataset for training and evaluation of HDR reconstruction methods in night or extreme scenes.

Embodiment 2:

Please refer to Figure 1, which is based on Example 1 but differs in that:

The design of the dual-branch Raw video HDR reconstruction algorithm mentioned in S2 specifically includes the following steps:

利用噪声估计网络估计噪声水平图：S201, noise estimation: input three consecutive raw domain LDR frames each time

Estimate the noise level map using a noise estimation network:

和其对应的曝光系数t_i-1、t_i、t_i+1，利用曝光系数对输入LDR图像进行曝光校正，校正公式为：S202, data processing and feature extraction: input three consecutive raw domain LDR frames

And its corresponding exposure coefficients ti _-1 , _ti , ti ₊₁ , use the exposure coefficients to perform exposure correction on the input LDR image, the correction formula is:

映射到同一曝光水平；Using the above formula, input

Mapped to the same exposure level;

Then, through the feature extraction module, convolution is used to extract features:

Where _Fi represents the features extracted from the i-th frame, the input LDR image is used to help detect overexposed and underexposed areas, the input HDR image is used to help with subsequent alignment, and the noise level image helps detect noise areas;

S203, feature alignment: cascaded pyramid deformable convolution structure, first downsample the input features twice to obtain features of multiple scales:

表示下采样；in,

represents downsampling;

与中间帧特征

级联估计偏移量

At the sth scale, using

Intermediate frame features

Cascade Estimation Offset

作为可变形卷积的偏移量，用可变形卷积对上一帧特征处理后得到当前尺度下的对齐结果：The calculated offset

As the offset of the deformable convolution, the deformable convolution is used to process the features of the previous frame to obtain the alignment result at the current scale:

Among them, ↑2 represents 2 times bilinear interpolation upsampling, and the alignment result of each scale is further fused with the alignment result of the previous scale through convolution; the coarse-to-fine joint prediction in the feature domain can more accurately estimate the displacement at a large scale;

S204, Temporal Fusion: The spatial attention structure obtains the attention correlation between adjacent frames through convolution, helping the network to reconstruct HDR images without ghosting and with accurate exposure:

表示时间融合后的特征。Among them, _Ai represents the predicted spatial attention, ⊙ represents element-by-element multiplication,

Represents the features after time fusion.

经过残差估计分支REB提取输入特征的高频信息，帮助恢复上一支路中缺少的内容：S205, content enhancement branch: the aligned features

The residual estimation branch REB extracts high-frequency information of the input features to help restore the missing content in the previous branch:

Where H _res represents the estimated residual information.

经过一系列残差块、跳跃连接后和H_res相加，再经过sigmoid层得到最终的raw域HDR结果

S206, Reconstruct HDR:

After a series of residual blocks and jump connections, it is added to H _res and then passed through a sigmoid layer to obtain the final raw domain HDR result.

和真值

之间的差异化网络：S207, loss function: output after tone mapping

and truth value

Differentiation network between:

表示经过色调映射后的raw域真值图像和预测结果。Among them, μ is 5000, T _i ^raw and

Represents the raw domain true value image and prediction result after tone mapping.

Based on the Raw video HDR dataset, the present invention proposes a dual-branch HDR reconstruction method, and utilizes the proposed deformable convolution alignment module and content enhancement module to achieve the dynamic range extension of noisy LDR videos in difficult scenes.

Embodiment 3:

Please refer to Figure 2, which is based on Example 1-2 but is different in that:

The dual-branch HDR video reconstruction algorithm based on the Raw domain proposed in the present invention is compared with the mainstream methods on the market in the Raw domain, and the comparison results on the test set are shown in Figure 2 and Table 1.

Table 1 Index comparison table

It can be seen from Figure 2 and Table 1 that the dual-branch HDR video reconstruction algorithm based on the Raw domain proposed in the present invention can better reduce the influence of noise while retaining the original image details through the joint learning and information complementarity of the two branches; combined with the actual image effect and the data in the table, it can be obviously seen that the reconstruction algorithm proposed in the present invention has achieved better visual effects and data indicators.

The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims (3)

1. A dual-branch HDR video reconstruction algorithm based on Raw domain, characterized by comprising the following steps: S1. Establish a synthetic Raw domain video HDR dataset: Establish a Raw dataset, specifically including the following contents: S101, source sRGB video data selection: select a number of existing sRGB domain HDR videos and high-quality video datasets as source sRGB video data, and re-expose each HDR video by using selected exposure parameters to simulate an alternately exposed LDR video sequence; S102, Raw data set establishment: by simulating the camera imaging pipeline process, the sRGB video is converted into a Raw video data set. The specific steps are: obtain the inverse camera response curve → simulate the bayer format → data enhancement → re-exposure → add noise to obtain the HDR and LDR images in the Raw domain, and use the obtained HDR and LDR images in the Raw domain as the training data pairs in the Raw domain; S2. Design a reconstruction algorithm: Based on the data pair obtained in S1, use the LDR Raw frame I _i ^raw and the HDR Raw frame H _i ^raw as training pairs to design a dual-branch Raw video HDR reconstruction algorithm; S3, training model: build a model based on the reconstruction algorithm designed in S2, and use the deep learning framework Pytorch platform to train the model. Iterate 15 epcohs on the entire dataset, then reduce the learning rate to 0.00001, and continue to iterate until the loss converges to obtain the final model; S4, output results: Input the low dynamic range Raw video sequence in the test set into the final model obtained in S3 to obtain the corresponding high dynamic range output results. 2. According to the Raw domain-based dual-branch HDR video reconstruction algorithm of claim 1, it is characterized in that the analog camera imaging process mentioned in S102 converts the sRGB video into a Raw video data set, specifically comprising the following steps: S1021. For a high-quality video dataset, convert the video frames from a nonlinear domain to a linear domain by estimating a CRF curve; S1022, down-sample the 3-channel sRGB frame into four channels of one-quarter resolution: red, green, green, and blue, and combine them into a mosaic image according to the GRBG bayer format; S1023, converting the mosaic image into a 4-channel image of G, R, B, G, and then randomly scaling, translating, and rotating; S1024, converting the Raw domain HDR frame into a Raw domain LDR frame according to specific exposure parameters; S1025. Add simulated Gaussian and Poisson noise to the Raw domain LDR frame. 3. According to the Raw domain-based dual-branch HDR video reconstruction algorithm of claim 1, it is characterized in that the design of the dual-branch Raw video HDR reconstruction algorithm mentioned in S2 specifically comprises the following steps: S201, noise estimation: input three consecutive raw domain LDR frames each time

Estimate the noise level map using a noise estimation network:

S202, data processing and feature extraction: input three consecutive raw domain LDR frames

And the corresponding exposure coefficients ti _-1 , _ti , ti ₊₁ , the exposure coefficients are used to perform exposure correction on the input LDR image, and the correction formula is:

Using the above formula, input

Mapped to the same exposure level; Then, through the feature extraction module, convolution is used to extract features:

Where _Fi represents the features extracted from the i-th frame, the input LDR image is used to help detect overexposed and underexposed areas, the input HDR image is used to help with subsequent alignment, and the noise level image helps detect noise areas; S203, feature alignment: The cascaded pyramid deformable convolution structure first downsamples the input features twice to obtain features of multiple scales:

in,

represents downsampling; At the sth scale, using

Intermediate frame features

Cascade Estimation Offset

The calculated offset

As the offset of the deformable convolution, the deformable convolution is used to process the features of the previous frame to obtain the alignment result at the current scale:

Among them, ^↑2 represents 2 times bilinear interpolation upsampling, and the alignment result of each scale is further fused with the alignment result of the previous scale through convolution; the coarse-to-fine joint prediction in the feature domain can more accurately estimate the displacement at a large scale; S204, Temporal Fusion: The spatial attention structure obtains the attention correlation between adjacent frames through convolution, helping the network to reconstruct HDR images without ghosting and with accurate exposure:

Among them, _Ai represents the predicted spatial attention, ⊙ represents element-by-element multiplication,

Represents the features after time fusion; S205, content enhancement branch: The aligned features are passed through the residual estimation branch REB to extract the high-frequency information of the input features to help restore the missing content in the previous branch:

Among them, H _res represents the estimated residual information;

represents the aligned features; S206, Reconstruct HDR:

After a series of residual blocks and jump connections, it is added to H _res and then passed through a sigmoid layer to obtain the final raw domain HDR result.

S207, loss function: output after tone mapping

and truth value

Differentiation network between:

Among them, μ is 5000,

and

Represents the raw domain true value image and prediction result after tone mapping.

Images