CN111292264B - A high dynamic range image reconstruction method based on deep learning - Google Patents

Abstract

The invention discloses an image high dynamic range reconstruction method based on deep learning, and belongs to the field of computational photography and digital image processing. The invention establishes a mapping network from a single LDR image to an HDR image by adopting a method based on deep learning. The method first sequentially generates LDR training data, HDR sample labels with aligned brightness units and mask images of high brightness areas from the collected HDR data set. The neural network is then constructed and trained to obtain a network model with an LDR to HDR mapping relationship. And finally, directly inputting the LDR image into the network model by utilizing the generated network model obtained by training, and outputting the reconstructed HDR image. The method can effectively reconstruct the dynamic range of the real scene from a single common digital image, and can be used for HDR simulation effect display of the common digital image or providing more realistic rendering effect for the image illumination technology.

Description

A method for image high dynamic range reconstruction based on deep learning

Technical Field

The present invention belongs to the fields of computational photography and digital image processing, and relates to a method for reconstructing a high dynamic range of an image, in particular to a method for reconstructing a high dynamic range of an image based on deep learning.

Background Art

High Dynamic Range Imaging (HDRI) technology is an image representation method used to achieve a larger exposure range than ordinary digital images. High Dynamic Range (HDR) images can provide a larger brightness range and more light and dark details than ordinary digital images, which enables HDR images to present brightness change information closer to the real scene. In recent years, with the continuous evolution of display devices and the increasing demand for physical rendering, high dynamic range imaging technology has become increasingly important in practical applications. However, the current method of directly obtaining HDR images requires high professional skills, is costly and time-consuming. For methods of reconstructing HDR from a single ordinary digital image, traditional methods can only reduce the non-adaptability of the problem as much as possible by adding constraints, which makes them only effective for certain specific application scenarios. Some scholars have also done some fruitful work based on deep learning, but they have failed to consider factors such as the invariance of brightness levels between HDR images, which leads to limitations in the reconstruction effect. The invention can effectively reconstruct the dynamic range of a real scene from a single ordinary digital image, which can be used for HDR simulation effect display of ordinary digital images or provide more realistic rendering effects based on image lighting technology.

Summary of the invention

The purpose of the present invention is to restore the high dynamic range image of the original scene from a single ordinary digital image as much as possible. Here, the ordinary digital image refers to a low dynamic range (LDR) image saved with 8-bit color depth and 256 color levels, and the high dynamic range image refers to a high dynamic range image saved in the format of ".EXR" or ".HDR" that is close to the light and dark changes of the real scene.

In order to achieve the above-mentioned purpose, the present invention adopts a deep learning-based method to establish a mapping network from LDR images to HDR images, and trains the network through training data to establish an end-to-end mapping relationship from LDR images to HDR images. The overall framework diagram is shown in Figure 1. The algorithm is divided into two parts: data preprocessing and deep neural network training. The data preprocessing part includes three parts: the generation of training sample pairs, the alignment of HDR image brightness units, and the generation of image highlight area masks. The neural network structure is divided into a basic HDR reconstruction network and a training optimization network, as shown in Figure 2. Its loss function includes three items, namely, the scale-invariant loss of the HDR reconstructed image, the cross entropy loss of the highlight area classification, and the generative adversarial loss.

The method specifically includes the following contents and steps:

1. Data Preprocessing

1) Generate LDR training sample input

Before using a deep neural network for supervised training, it is necessary to obtain a training dataset corresponding to the network input and output. The training dataset contains several LDR-HDR image pairs, where the HDR image data can use existing available HDR images, which are used as labels for training samples to supervise the training of the network; the LDR image data, as the sample input corresponding to the HDR image, needs to be generated from the original HDR image. There are two ways to generate it: one is to use a tone mapping algorithm to complete the generation from HDR images to LDR images, and the other is to construct a virtual camera to align the HDR image as a simulated scene for simulated shooting to obtain an LDR image.

Generate LDR image using tone mapping algorithm: Select an appropriate tone mapping algorithm and directly use the HDR image as the algorithm input to get the corresponding LDR image output.

The LDR image is obtained by constructing a virtual camera: first, the value range of the virtual camera's dynamic range is determined based on the commonly used digital SLR camera, and each time an LDR image is obtained, a value within the range is randomly selected as the dynamic range of the camera for the simulated shooting; then the virtual camera automatically exposes according to the input HDR image, takes the boundary value of the pixels whose brightness exceeds the dynamic range of the virtual camera, and then linearly maps it to the low dynamic range of the LDR image; finally, the obtained image is mapped from the linear space to an ordinary digital image through a randomly selected approximate camera response function, that is, the required LDR image is obtained.

2) Align HDR sample labels

For HDR images stored in the relative brightness domain, their brightness units need to be aligned before they are used as training sample labels. Let the original HDR image be H, the LDR image be converted to linear space and normalized to [0,1], let L, H _l,c , L _l,c be the pixel values of the image at position l and channel c respectively, and the alignment method is:

为对齐后的HDR图像，m_l,c定义为in

is the aligned HDR image, m _l,c is defined as

Where τ is a constant in [0,1]. The aligned HDR image and its corresponding LDR image form a training sample pair for neural network training.

3) Generate highlight mask image

After obtaining the aligned HDR image, the mask image of the high brightness area in the image can be obtained by binarization. The area with a value of 1 in the mask image represents objects or surfaces with higher brightness in the scene, such as light sources, strong light reflective surfaces, etc. These highlight areas are often clipped in brightness due to overexposure in LDR images. The highlight mask image generated from the HDR image can be used as a sample label for the network optimization training part to optimize the training process of the neural network.

2. Training of Neural Networks

1) Neural network structure

The network structure is mainly divided into two parts: the generating network and the discriminating network. The structural diagram is shown in Figure 3. The generating network is a U-Net structure. The network accepts an LDR image as input. After the encoding network composed of the ResNet50 model and the decoding network composed of the 6-layer "upsampling + convolution layer" module, an HDR image and a highlight mask image are output respectively. The HDR image output by the network is the HDR reconstructed image obtained according to the LDR input image, and the highlight mask image is the network's prediction of the highlight area in the LDR image, which can be used as data for optimizing network training. The discriminator network is a fully convolutional network composed of 4 convolutional layers. The discriminator accepts an HDR image and a highlight mask image as input, and outputs a feature map representing the probability that the input HDR image is a real HDR image or a false HDR image generated by the network. The feature map can be used as data for training the neural network.

2) Neural network training method

The invention uses supervised learning to train the above network. The training uses the Adam optimizer to perform back propagation optimization on the generative network and the discriminative network one by one, where the generative network contains three sets of optimization functions, and the loss function is defined as follows:

L _G ＝α ₁ L _hdr +α ₂ L _mask +α ₃ L _gan

The loss is controlled by three loss functions, including the scale-invariant loss of HDR reconstructed images, the cross-entropy loss for highlight area classification, and the generative adversarial loss.

Scale invariance loss of HDR reconstructed images: This loss function is proposed based on the scale invariance of HDR images in the relative brightness domain. This loss controls the HDR image output by the network to make it as close to the HDR label value as possible. Its definition is as follows:

为目标图像，下标l，c分别表示像素位置和颜色通道，

表示在对数域中网络输出与样本标签的差值，∈为一个防止对数计算为零的微小值。该损失第一项即为普通的L₂损失，引入第二项后，该项损失在计算时仅由预测值与样本标签的差值所影响，与预测值或样本标签的实际大小无关。Where y represents the HDR image output by the network,

is the target image, subscripts l and c represent pixel position and color channel respectively.

It represents the difference between the network output and the sample label in the logarithmic domain, and ∈ is a small value that prevents the logarithm from being calculated to zero. The first term of this loss is the ordinary _L2 loss. After the introduction of the second term, this loss is only affected by the difference between the predicted value and the sample label during calculation, and has nothing to do with the actual size of the predicted value or the sample label.

Cross entropy loss for highlight area classification: This loss controls the network's detection of high brightness areas in the image. The highlight mask image output by the network is the classification result of the highlight area or non-highlight area in the input LDR image. The result should be as close as possible to the highlight mask image generated in the preprocessing step. The loss function is defined as follows:

分别为网络预测值和标签值。Where m,

are the network prediction value and label value respectively.

Generate adversarial loss: This loss can make the HDR image predicted by the network as close to the real HDR image as possible in distribution, preventing the predicted result from being reduced in numerical difference from the label result while ignoring the difference in overall distribution. The loss function is defined as follows:

Where D(y) is the calculation result of the discriminant network with the output of the generative network as input.

The loss function of the discriminant network is the standard WGAN-GP loss, which controls the discriminant network to make it judge as accurately as possible whether the image input to the discriminant network is a real HDR image. It is defined as follows:

Where ∈ is a random number in [0,1].

The network is trained according to the above training method. When the loss function converges, the generative network establishes a mapping relationship from a single LDR image to an HDR image. Using this trained generative network, the goal of reconstructing a realistic high dynamic range image as much as possible from a single ordinary digital image can be achieved.

Compared with the prior art, the present invention has the following advantages:

1. The present invention constructs an end-to-end neural network that can reconstruct a sufficiently realistic HDR image with only one picture, without the need for manual interaction;

2. The present invention aligns HDR data and trains the network based on scale invariance, which can achieve better reconstruction effect;

3. The present invention optimizes network training by generating highlight mask images, which can achieve better reconstruction effects in high-brightness areas.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a general framework diagram of the present invention;

FIG2 is a diagram of a deep neural network structure of the present invention;

FIG. 3 is a schematic diagram showing the effect of the present invention.

DETAILED DESCRIPTION

The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

As shown in FIG1 , the present invention is a method for reconstructing an image with high dynamic range based on deep learning, comprising the following steps:

Step 1: Preprocess the HDR dataset to construct training data for the neural network. First, generate LDR data from the collected HDR dataset, then use the LDR-HDR data to align the HDR data, and then use the aligned HDR image to generate a highlight mask image. Finally, integrate the three as training data for the neural network. The LDR data is the data input, and the aligned HDR data and highlight mask image are the label data.

Step 2: Build and train a neural network to obtain a network model with a mapping relationship from LDR to HDR. According to the training strategy, the generator network and the discriminator network are back-propagated and trained alternately until the loss function converges. At this point, the generator network is the network model that is ultimately used to reconstruct high dynamic range images.

Step 3: According to the generated network model trained in step 2, the LDR image to be reconstructed is directly input into the network model to output the reconstructed HDR image.

The method is described in detail below with reference to examples.

1. Data Preprocessing

The HDR data set is composed of an existing public HDR data set, and a set of data sets of the same size and type are formed by cropping, scaling and other operations on the collected data. According to this integrated data set, the display adaptive tone mapping algorithm and the aforementioned virtual camera shooting method are respectively applied to obtain LDR data. Regarding tone mapping, it is selected according to the type of picture in actual application. For example, if most of the pictures in the application are pictures taken without post-processing, the method of this embodiment is selected. If it is a picture for a certain processing, a method similar to the post-processing effect is selected. Specifically, for each HDR image, a display adaptive tone mapping algorithm is applied to obtain an LDR image, and then a virtual camera with random parameters is used to shoot once to obtain an LDR image, that is, each HDR image generates two LDR images obtained by different methods. The generated LDR image is converted to a linear space and normalized from an integer value in the range [0,255] to a decimal value [0,1].

Then, based on each pair of LDR-HDR images, the HDR images are brightness aligned. The alignment method is based on the formula in the description:

为对齐后的HDR图像，m_l,c定义为in

is the aligned HDR image, m _l,c is defined as

Where τ can be a constant in [0,1], and here τ = 0.08 is taken. After applying this formula to each pair of LDR-HDR images, the aligned HDR image data is obtained, which will be used as the sample label when training the network, and the LDR image data is used as the sample input.

Finally, based on the aligned HDR data, the highlight mask image is calculated by the following formula:

为对齐后的HDR图像的通道均值图像，t为常数，这里取t＝e^0.1。该掩码图像将作为训练网络时的另一项样本标签，同HDR图像一起监督网络的学习过程。in

is the channel mean image of the aligned HDR image, t is a constant, here we take t = e ^0.1 . The mask image will be used as another sample label when training the network, and supervise the learning process of the network together with the HDR image.

2. Neural Network Training

The structure of the neural network is constructed as shown in Figure 2. Specifically, for the ResNet50 model, the trained model in the existing classification task based on the ImageNet dataset is used as the initialization value of the weight of this part of the network; each of the remaining network blocks represents a convolutional block consisting of a convolutional layer, an instance normalization operation, and a relu activation function; the input of the decoder part of the generated network is composed of the output of the previous convolutional block and the output of the encoder at a symmetrical position; for the discriminator network part, its input is composed of the output of the generated network, and after passing through four layers of convolutional blocks, a probability feature map is output to determine whether the input HDR image is a real image.

Among them, the output of the generator network and the corresponding input sample label data, as well as the probability feature map calculated by the discriminant network based on the output data, are used together to calculate the generator loss according to the following formula:

L _G ＝α ₁ L _hdr +α ₂ L _mask +α ₃ L _gan

The loss is controlled by three loss functions, including the scale-invariant loss of HDR reconstructed images, the cross-entropy loss for highlight area classification, and the generative adversarial loss.

Scale invariance loss of HDR reconstructed images: This loss function is proposed based on the scale invariance of HDR images in the relative brightness domain. This loss controls the HDR image output by the network to make it as close to the HDR label value as possible. Its definition is as follows:

为目标图像，下标l，c分别表示像素位置和颜色通道，

表示在对数域中网络输出与样本标签的差值，∈为一个防止对数计算为零的微小值。该损失第一项即为普通的L₂损失，引入第二项后，该项损失在计算时仅由预测值与样本标签的差值所影响，与预测值或样本标签的实际大小无关。Where y represents the HDR image output by the network,

is the target image, subscripts l and c represent pixel position and color channel respectively.

It represents the difference between the network output and the sample label in the logarithmic domain, and ∈ is a small value that prevents the logarithm from being calculated to zero. The first term of this loss is the ordinary _L2 loss. After the introduction of the second term, this loss is only affected by the difference between the predicted value and the sample label during calculation, and has nothing to do with the actual size of the predicted value or the sample label.

Cross entropy loss for highlight area classification: This loss controls the network's detection of high brightness areas in the image. The highlight mask image output by the network is the classification result of the highlight area or non-highlight area in the input LDR image. The result should be as close as possible to the highlight mask image generated in the preprocessing step. The loss function is defined as follows:

分别为网络预测值和标签值。Where m,

are the network prediction value and label value respectively.

Generate adversarial loss: This loss can make the HDR image predicted by the network as close to the real HDR image as possible in distribution, preventing the predicted result from being reduced in numerical difference from the label result while ignoring the difference in overall distribution. The loss function is defined as follows:

Where D(y) is the calculation result of the discriminant network with the output of the generative network as input.

The loss function of the discriminant network is the standard WGAN-GP loss, which controls the discriminant network to make it judge as accurately as possible whether the image input to the discriminant network is a real HDR image. It is defined as follows:

Where ∈ is a random number in [0,1].

Wherein α ₁ , α ₂ , α ₃ are 1, 0.1, and 0.02 respectively, and the specific calculation formula of each loss is described in the content of the invention. According to the calculated loss value, the network is back-propagated and the weight is updated using the Adam optimization algorithm. In addition, each time the weight of the generating network is updated, it is necessary to continue to calculate the loss of the discriminant network and update the weight of the discriminant network. The update algorithm also uses the Adam algorithm, and the specific formula is described in the content of the invention.

According to the above training method, one or more pairs of training data are delivered to the network for training each time, and the entire training data set is iterated in sequence until the loss function converges, and the network training is completed.

(III) Network model application

After the network training is completed, the network and its weight parameters of the generative network part are extracted to obtain the final reconstruction model of the high dynamic range image. Using this model, only one LDR picture is needed as input each time to obtain an approximately realistic HDR picture. Figure 3 shows an example of an application, in which the generative network model is the generative network part of the neural network trained by the above method. The model can accept LDR pictures of any size as input and directly output HDR reconstructed images of the same size.

The present invention proposes a method for reconstructing high dynamic range from a single ordinary digital image based on deep learning, which can effectively and realistically reconstruct high dynamic range for images of general scenes. The present invention has a wide range of applications. It can adapt to scenes with different requirements by training different data sets, and the same training set only needs to be trained once and can be applied multiple times.

Claims (4)

1. The image high dynamic range reconstruction method based on deep learning is characterized by comprising the following steps of:

step 1, a neural network is established based on a deep learning method, and comprises a generation network from a low dynamic range image to a high dynamic range image and a discrimination network for discriminating whether the high dynamic network image is real;

step 2, preprocessing an HDR data set to form training data, wherein the data preprocessing is divided into three parts of generation of LDR data, alignment of HDR image brightness units and generation of image highlight region masks, the LDR data obtained after preprocessing is used as training input data for generating a network and is output as an HDR image and a highlight mask image, the aligned HDR data and the highlight mask image after the data preprocessing are used as sample label data for training, the network is judged to accept one HDR image and the highlight mask image as input, and a feature map representing the probability that the input HDR image is a real HDR image or a false HDR image generated by the network is output;

step 3, training the neural network based on three loss functions, training the neural network in a supervised learning mode, performing back propagation optimization on a generation network and a discrimination network successively by adopting an Adam optimizer, wherein the three loss functions are respectively a scale invariant loss function of an HDR reconstructed image, a cross entropy loss function of highlight region classification and a generation antagonism loss function, and the loss functions are defined as follows:

L _G ＝α ₁ L _hdr +α ₂ L _mask +α ₃ L _gan

the scale invariant loss function of the HDR reconstructed image is defined as follows:

where y represents the HDR image output by the network,

for an aligned HDR image, +.>

Representing the difference between the network output and the sample label in the logarithmic domain, e being a small value preventing logarithmic calculation to zero, subscript l, c representing pixel position and color channel, respectively;

a cross entropy loss function for highlight region classification, defined as follows:

wherein the method comprises the steps of

Respectively a network predicted value and a label value;

the challenge loss function is generated as defined below:

wherein D (y) is a calculation result for discriminating the network to generate network output as input;

training the network according to the loss function, and extracting and generating a network model as a final algorithm model after the loss function converges.

2. The method according to claim 1, characterized in that: the low dynamic range image refers to a low dynamic range image stored at 8-bit color depth, 256-tone scale, and the high dynamic range image refers to a high dynamic range image stored in ". EXR" or ". HDR" format that approximates the change in the brightness of a real scene.

3. The method according to claim 1, characterized in that: the neural network described in the step 1 comprises a generating network and a judging network, wherein the generating network is of a U-Net structure, the network receives an LDR image as input, and after a decoding network is formed by a coding network formed by a ResNet50 model and a 6-layer 'up-sampling+convolution layer' module, an HDR image and a highlight mask image are respectively output; the judging network is a full convolution network formed by 4 layers of convolution layers, receives an HDR image and a high light mask image as inputs, and outputs a feature map representing the probability that the input HDR image is a real HDR image or a false HDR image generated by the network.

4. The method according to claim 1, characterized in that: the data preprocessing described in the step 2 comprises the following specific processes:

step 2.1, generating LDR data, generating LDR training sample input, respectively shooting each HDR image by using a tone mapping algorithm and a virtual camera to obtain an LDR image, selecting a proper tone mapping algorithm, and directly inputting the HDR image as the algorithm to obtain a corresponding LDR image output; meanwhile, obtaining an LDR image by constructing a virtual camera, firstly determining a value range of a dynamic range of the virtual camera based on a common digital single-phase inverter, randomly selecting a value in the range as the dynamic range of the camera for simulating shooting every time the LDR image is obtained, then automatically exposing the virtual camera according to an input HDR image, taking a boundary value for a pixel with brightness exceeding the dynamic range of the virtual camera, linearly mapping the pixel to a low dynamic range of the LDR image, and finally mapping the obtained image from a linear space to a common digital image through a randomly selected approximate camera response function to obtain the required LDR image;

step 2.2, alignment of HDR image luminance units for preservingHDR images with relative luminance domains exist, aligning their luminance units before they are labeled as training samples; let the original HDR image be H, the LDR image is converted to linear space and normalized to [0,1]]Let L, H _l，c ，L _l，c The pixel values of the image at the position l and the channel c are aligned by the following methods:

wherein the method comprises the steps of

For an aligned HDR image, m _l，c Is defined as

Wherein τ is a constant of [0,1], and the aligned HDR image and the corresponding LDR image form a training sample pair for training of the neural network;

step 2.3, generating an image highlight region mask, obtaining an aligned HDR image brightness unit, and obtaining a mask image of a highlight region in the image in a binarization mode, wherein the formula is as follows:

wherein the method comprises the steps of

For the channel mean image of the aligned HDR image, t is a constant, and the region with the median value of 1 in the mask image represents an object or surface with higher brightness in the scene, including a light source and a strong light reflecting surface. />

Images