CN109712083B - Single image defogging method based on convolutional neural network - Google Patents

Abstract

The present invention proposes a single image dehazing method based on convolutional neural network. The method first constructs a training set as the input of a deep convolutional neural network model. The network model includes a shallow neural network model and a deep neural network model. The network model is used to extract and fuse the features of the RGB color space of the foggy image, and output the scene depth map of the foggy image; the deep network model performs multi-scale mapping, pooling, Convolution and other operations, the output is the transmittance map of the hazy image. Finally, the haze-free image can be recovered by the transmittance, atmospheric light values, and atmospheric scattering model. The present invention constructs a shallow convolutional neural network model by extracting and fusing the features of the RGB color space of the foggy image, and connects with the multi-scale deep neural network model to establish an end-to-end neural network model, which can realize the removal of The fog is sharpened, especially in dark environments, to avoid color distortion in the image.

Description

A single image dehazing method based on convolutional neural network

technical field

The invention relates to a single image dehazing method, in particular to a single image dehazing method based on a convolutional neural network.

Background technique

Due to waste incineration, construction dust, automobile exhaust emissions and other reasons, many cities in China have cast the shadow of smog. Smog has become an environmental problem that has received continuous attention in recent years. The images taken in haze weather are unclear due to the decrease in contrast and color saturation, which affects the use of images. For example, the foggy traffic surveillance video is blurred, resulting in deviations in the image recognition and processing process, which is not conducive to accurately recording traffic information. Therefore, there are urgent theoretical and practical demands to improve image quality in foggy weather and reduce the impact of foggy weather on outdoor imaging.

With the development of computer technology, video and image dehazing algorithms have received extensive attention and are widely used in civil and military fields, such as remote sensing, target detection, and traffic monitoring.

At present, image dehazing algorithms can be mainly divided into three types: the first type is the dehazing method of image enhancement. This method does not consider the cause of image degradation, and turns the problem of image dehazing into a problem of contrast enhancement. The enhanced image has higher contrast, making the restored image more in line with human aesthetic concepts, but the processed image There is a problem of information loss in the image, and distortion will occur. The second category is the dehazing method for image restoration. This method analyzes from the perspective of image degradation, establishes a foggy imaging model, deduces the process of image degradation, and restores the image after dehazing. This method makes the processed image clearer and more natural, with loss of details. less. However, the effect of dehazing is related to the selection of model parameters, and inaccurate parameters will directly affect the effect of the restored image. In recent years, with the continuous development of deep learning, more and more are used in the field of image processing, such as image classification, object recognition, face recognition, etc., and have achieved good results. Therefore, the dehazing algorithm based on deep learning can be considered as the third type of dehazing algorithm. In the existing deep learning-based image dehazing algorithms, most of the foggy images are artificially synthesized by using the haze-free image through the atmospheric scattering model and randomly setting parameters; The fog-free image is calculated by inverse calculation. Convolutional Neural Network (CNN) is a deep learning model, which reduces the number of parameters and connections through weight sharing and local receptive fields, reduces the complexity of neural networks, and has strong adaptability . Therefore, CNN is widely used in image processing research, and its application in the field of image recognition is a current research hotspot.

Although the dehazing algorithm based on image restoration has a relatively good effect, it is not universal because the simplified physical model is based on the condition that the atmosphere is single scattering and the medium is homogeneous, such as uneven fog or sky area, and in the dark It is easy to cause color distortion in the environment.

SUMMARY OF THE INVENTION

Purpose of the invention: In order to solve the above technical problems, the present invention proposes a new image dehazing method based on convolutional neural network and multi-channel color information fusion, so as to achieve the purpose of image dehazing.

Technical scheme: In order to realize the above-mentioned technical effect, the technical scheme proposed by the present invention is:

A single image dehazing method based on convolutional neural network, comprising steps (1) to (9) performed in sequence:

(1) Construct training samples and test samples: obtain several fog-free images, add fog of different concentrations to the fog-free images to obtain foggy images, convert the foggy and fog-free images into image blocks in HDF5 format, and follow The preset ratio divides the image block of the foggy image and the image block of the non-fog image into two parts, one part is used as a training sample, and the other part is used as a test sample;

(2) building a multi-scale deep convolutional network model, the multi-scale deep convolutional network model includes a shallow convolutional neural network and a deep convolutional neural network;

The shallow convolutional neural network includes: an input layer, a convolutional layer, a fully connected layer, a pooling layer, and an output layer that are cascaded in sequence; the input layer maps the input image block i to the R, G, and B color spaces, and the volume The product layer uses a Gaussian filter to convolve the values of the R, G, and B color channels of the input image block respectively, and the result after convolution is

Wherein, I ^c represents the pixel value matrix of a certain color channel of the input image block R, G, B color space, W ₁ and B ₁ respectively represent the weight coefficient matrix and the deviation matrix of the corresponding convolutional network;

The fully connected layer combines the results of the above convolutional layers, and the combined result is:

After the pooling layer downsamples the results of the fully connected layer, the high-dimensional feature vector F ₂ of the input image block can be obtained:

Among them, F ₂ (x) represents the feature value at the pixel point x in the input image block, Ω(x) is a certain area centered on the pixel point x in the input image block; the result of the pooling layer F ₂ is output through the output _The layer is output to the deep convolutional neural network, and F2 is the scene depth matrix of the input image block;

The deep convolutional neural network includes: input layer, multi-scale mapping unit, multi-scale connection layer, pooling layer, convolution layer, BReLU excitation layer and output layer;

Among them, the input layer receives the high-dimensional feature vector F ₂ output by the shallow convolutional neural network; the multi-scale mapping unit maps F ₂ as

Among them, W ₃ is the weight coefficient matrix corresponding to three groups of convolutional networks of different scales in the multi-scale mapping unit, B ₃ is the deviation matrix, and * represents the convolution operation;

The multi-scale connection layer combines the output results of the above multi-scale mapping units to obtain F ₃ :

分别为所述3组不同尺度的卷积网络的输出结果；in,

are the output results of the three groups of convolutional networks of different scales;

Then, the pooling layer downsamples the output of the multi-scale connection layer, and the output of the pooling layer is _F4 :

The convolutional layer maps the result of the pooling layer _F4 to _F5 :

F ₅ =W ₅ *F ₄ +B ₅

Among them, W ₅ is the weight coefficient matrix of the convolution network in the multi-scale mapping unit, B ₅ is the deviation matrix, and * represents the convolution operation;

The BReLU activation layer uses the bilateral modified linear unit BReLU activation function to perform nonlinear regression on the output result F ₅ of the convolution layer, and obtains F ₆ :

F ₆ (x)=min(a _max ,F ₅ (x))

Among them, a _max is the upper amplitude value of the bilateral modified linear unit BReLU activation function;

Finally, let t=F ₆ , the output layer outputs t, and t is the transmittance matrix of the input image block;

(3) Construct the loss function:

When there is only a single training sample i, the loss function is:

When there are multiple training samples, the loss function is:

表示训练样本i的实际透射率矩阵；Among them, t _i represents the transmission map of the training sample i, n is the number of training samples, λ represents the attenuation parameter, W _ji represents the weight coefficient matrix W _j of the training sample i,

represents the actual transmittance matrix of training sample i;

(4) For each W _ji , use a Gaussian distribution with a mean value of 0 and a standard deviation of 0.001 to randomly initialize the components in W _ji ; initialize B _ji to 0, and B _ji represents the deviation matrix B _j of the training sample i; Initialize ΔW _ji =0, ΔB _ji =0;

和

(5) For each sample i, use the backpropagation algorithm to find the partial derivatives of W _ji and B _ji :

and

Find the amount of change in W _ji and B _ji :

(6) Update:

(7) Substitute the updated W _ji and B _ji into the loss function, and repeat steps (5) to (7) until the value of the loss function is the smallest. step (8);

(8) Input the new foggy image into the trained multi-scale deep convolutional network model, and the obtained output is used as the initial transmittance of the new foggy image;

(9) Estimate the atmospheric light intensity A when the new foggy image is taken; restore the corresponding fog-free image according to the atmospheric scattering model.

根据大气散射模型计算得到，大气散射模型为：Further, the said

According to the calculation of the atmospheric scattering model, the atmospheric scattering model is:

Among them, I represents the light intensity matrix of the training sample i, J represents the light intensity matrix of the image block corresponding to the training sample i in the original haze-free image, and A is the atmospheric light intensity when the foggy image corresponding to the training sample i was taken. A.

Further, the method for estimating the atmospheric light intensity A when the foggy image is photographed is:

Estimate the dark channel map for a hazy image:

Among them, I ^dark represents the dark channel map estimation result of the pixel matrix I of the foggy image, I ^c represents the value of a certain color channel of I, and y represents the pixel point;

Sort the pixel values of the obtained dark channel map in descending order, select the top one thousandth pixel point as the candidate point, correspond the candidate point to the corresponding position in the foggy image, and calculate the corresponding position of the candidate point in the foggy image. The brightness value of the location, and the calculated maximum brightness value is used as the estimated value of atmospheric light intensity A.

Beneficial effect: Compared with the prior art, the present invention has the following advantages:

1. Compared with the image restoration algorithm based on the physical model, the present invention, especially the classical dark channel prior, utilizes the diversity of the sample set and the universality of the network structure, and can effectively solve the problem of uneven medium and fog images. Flat areas have better dehazing effect;

2. The present invention designs an end-to-end deep convolutional neural network by fusing the information of the three channels of the RGB color space of the foggy image, so as to avoid color distortion caused by fog removal in a dark environment;

3. The present invention combines the convolutional neural network with the image prior information, which can more accurately estimate the medium transmittance of the foggy image, obtain a better dehazing effect, and the image after dehazing is more real and natural.

Description of drawings

Fig. 1 is a flow chart of a method for dehazing a single image based on a convolutional neural network according to the invention;

2 is a structural diagram of a shallow convolutional neural network in the present invention;

FIG. 3 is a structural diagram of a deep convolutional neural network in the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

FIG. 1 is a flowchart of a method for dehazing a single image based on a convolutional neural network according to the present invention, and the method includes:

Step 1. Obtain the PASCAL VOC dataset and the haze-free images downloaded on the Internet as the haze-free image set in the training sample;

Step 2. Use Perlin Noise to add fog of different concentrations to the fog-free image set in step 1 to obtain a foggy image set; crop the images in the foggy image set and the fog-free image set into 64*64 images Then convert the image block of the foggy image and the image block of the non-fog image into two parts in proportion, one part is used as a training sample, and the other part is used as a test sample, which is convenient for training; in order to adapt to Fog density under different weather conditions, the characteristics of images with different fog density are learned, and the fog-free images are assembled with fog concentrations of 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, respectively, and get Foggy image set; select 2506 pairs of foggy images and non-fog images as training samples, and the remaining 502 pairs as test samples;

Step 3. Design an end-to-end multi-scale deep convolutional network model with the training samples in HDF5 format in step 2 as input; the multi-scale deep convolutional network model includes a shallow convolutional neural network and a deep convolutional neural network;

The structure of the shallow convolutional neural network is shown in Figure 2. The shallow convolutional neural network includes: 1 input layer, 1 convolutional layer, 1 fully connected layer, 1 pooling layer and 1 output layer.

Among them, the convolution layer includes 32 Gaussian filters, and the three channels of the RGB color space of the input image block are convolved with 32 parallel Gaussian filters, so that each input image block is represented by a high-dimensional feature vector. :

Wherein, I ^c represents the pixel value matrix of a certain color channel of the input image block R, G, B color space, W ₁ and B ₁ respectively represent the weight coefficient matrix and the deviation matrix of the corresponding convolutional network;

The fully connected layer combines the results of the above convolutional layers, and the combined result is:

Among them, ∩ represents the connection operation;

The pooling layer downsamples the high-dimensional feature vector output by the fully-connected layer, adopts the Max pooling method, the filter size is 3*3, and the high-dimensional feature vector F ₂ of the input image block can be obtained:

Among them, F ₂ (x) represents the feature value at the pixel point x in the input image block, and Ω(x) is a certain area centered on the pixel point x in the input image block;

The result of the pooling layer F ₂ is output to the deep convolutional neural network through the output layer, and F ₂ is the scene depth matrix of the input image block;

The structure of the deep convolutional neural network is shown in Figure 3, including: input layer, multi-scale mapping unit, multi-scale connection layer, pooling layer, convolution layer, BReLU excitation layer and output layer;

Among them, the input layer receives the high-dimensional feature vector F ₂ output by the shallow convolutional neural network; the multi-scale mapping unit maps F ₂ as

Among them, W ₃ is the weight coefficient matrix corresponding to three groups of convolutional networks of different scales in the multi-scale mapping unit, B ₃ is the deviation matrix, and * represents the convolution operation;

The multi-scale connection layer combines the output results of the above multi-scale mapping units to obtain F ₃ :

分别为所述3组不同尺度的卷积网络的输出结果；in,

are the output results of the three groups of convolutional networks of different scales;

Then, the pooling layer downsamples the output of the multi-scale connection layer, and the output of the pooling layer is _F4 :

The convolutional layer maps the result of the pooling layer _F4 to _F5 :

F ₅ =W ₅ *F ₄ +B ₅

Among them, W ₅ is the weight coefficient matrix of the convolution network in the multi-scale mapping unit, B ₅ is the deviation matrix, and * represents the convolution operation;

The BReLU activation layer uses the bilateral modified linear unit BReLU activation function to perform nonlinear regression on the output result F ₅ of the convolution layer, and obtains F ₆ :

F ₆ (x)=min(a _max ,F ₅ (x))

Among them, a _max is the upper amplitude value of the activation function of the bilateral modified linear unit BReLU; the gradient function of the activation function of the bilateral modified linear unit BReLU is:

Finally, let t=F ₆ , the output layer outputs t, and t is the transmittance matrix of the input image block;

Step 4: Build the loss function:

When there is only a single training sample i, the loss function is:

When there are multiple training samples, the loss function is:

表示训练样本i的实际透射率矩阵；所述

根据大气散射模型计算得到，大气散射模型为：Among them, t _i represents the transmission map of the training sample i, n is the number of training samples, λ represents the attenuation parameter, W _ji represents the weight coefficient matrix W _j of the training sample i,

represents the actual transmittance matrix of training sample i; the

According to the calculation of the atmospheric scattering model, the atmospheric scattering model is:

中，等式右侧第一项

是均方差项，第二项

是规则项。可以看出，规则项和偏置B_ji无关，仅能控制权重的大小，因此也称为权重衰减项。权重衰减项中权重的衰减参数λ可以用来决定两项在损失函数中的比重。训练的关键就是通过不断调整权重W_ji和偏置B_ji，获得最小的损失函数。Among them, I represents the light intensity matrix of the training sample i, J represents the light intensity matrix of the image block corresponding to the training sample i in the original haze-free image, and A is the atmospheric light intensity when the foggy image corresponding to the training sample i was taken. A. in the loss function

, the first term on the right-hand side of the equation

is the mean square error term, the second term

is the rule item. It can be seen that the rule term has nothing to do with the bias B _ji , and can only control the size of the weight, so it is also called the weight decay term. The decay parameter λ of the weight in the weight decay term can be used to determine the weight of the two terms in the loss function. The key to training is to obtain the smallest loss function by continuously adjusting the weight W _ji and the bias B _ji .

During training, all weights W _ji and bias B _ji parameters are initialized first. The weights of each layer of the network model are randomly initialized with a Gaussian distribution with a mean of 0 and a standard deviation of 0.001, and the initial bias is set to 0.

After initialization, the stochastic gradient descent algorithm is used to update the weights W _ji and biases B _ji . The update rules respectively obey the following formulas:

where α represents the learning rate. The partial derivatives in the above two formulas can be obtained by the back-propagation algorithm, that is, the partial derivatives of the weight W _ji and the bias B _ji are obtained respectively for the loss function formula:

Among them, the main steps of the back-propagation algorithm are: firstly forward the given sample to obtain the output values of all neural nodes in the network, then calculate the total error, and use the total error to take a partial derivative of a node to obtain the The effect of the node on the final output.

Therefore, the complete network model training steps are as follows:

Initialize the parameters of each layer of the network;

For each sample i:

和

a: Use backpropagation to find

and

b: Find the variation of parameters W _ji and B _ji :

c: complete parameter update:

d: Substitute the updated weight W _ji and bias B _ji into the loss function, repeat step a to step d, until the loss function is the smallest, the update is over, and go to step 5. Use an Nvidia Ge Force GTX 1050 8G GPU for acceleration during training.

Step 5: Input the new foggy image into the trained multi-scale deep convolutional network model, and the obtained output is used as the initial transmittance of the new foggy image;

Step 6: Estimate the atmospheric light intensity A when the new foggy image is taken; restore the corresponding fog-free image according to the atmospheric scattering model. Among them, the method of estimating the atmospheric light intensity A when the foggy image is taken is:

Estimate the dark channel map for a hazy image:

Among them, I ^dark represents the dark channel image estimation result of the pixel matrix I of the foggy image, I ^c represents the value of a certain color channel of I, and y represents the pixel point.

Sort the pixel values of the obtained dark channel map in descending order, select the top one thousandth pixel point as the candidate point, correspond the candidate point to the corresponding position in the foggy image, and calculate the corresponding position of the candidate point in the foggy image. The brightness value of the location, and the calculated maximum brightness value is used as the estimated value of atmospheric light intensity A.

The above is only the preferred embodiment of the present invention, it should be pointed out that: for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims (3)

1. a single image dehazing method based on convolutional neural network, is characterized in that, comprises the steps (1) to (9) that carry out successively: (1) Construct training samples and test samples: obtain several fog-free images, add fog of different concentrations to the fog-free images to obtain foggy images, convert the foggy and fog-free images into image blocks in HDF5 format, and follow The preset ratio divides the image block of the foggy image and the image block of the non-fog image into two parts, one part is used as a training sample, and the other part is used as a test sample; (2) building a multi-scale deep convolutional network model, the multi-scale deep convolutional network model includes a shallow convolutional neural network and a deep convolutional neural network; The shallow convolutional neural network includes: an input layer, a convolutional layer, a fully connected layer, a pooling layer, and an output layer that are cascaded in sequence; the input layer maps the input image block i to the R, G, and B color spaces, and the volume The product layer uses a Gaussian filter to convolve the values of the R, G, and B color channels of the input image block respectively, and the result after convolution is

Wherein, I ^c represents the pixel value matrix of a certain color channel of the input image block R, G, B color space, W ₁ and B ₁ respectively represent the weight coefficient matrix and the deviation matrix of the corresponding convolutional network; The fully connected layer combines the results of the above convolutional layers, and the combined result is:

After the pooling layer downsamples the results of the fully connected layer, the high-dimensional feature vector F ₂ of the input image block can be obtained:

Among them, F ₂ (x) represents the feature value at the pixel point x in the input image block, Ω(x) is a certain area centered on the pixel point x in the input image block; the result of the pooling layer F ₂ is output through the output _The layer is output to the deep convolutional neural network, and F2 is the scene depth matrix of the input image block; The deep convolutional neural network includes: input layer, multi-scale mapping unit, multi-scale connection layer, pooling layer, convolution layer, BReLU excitation layer and output layer; Among them, the input layer receives the high-dimensional feature vector F ₂ output by the shallow convolutional neural network; the multi-scale mapping unit maps F ₂ as

Among them, W ₃ is the weight coefficient matrix corresponding to three groups of convolutional networks of different scales in the multi-scale mapping unit, B ₃ is the deviation matrix, and * represents the convolution operation; The multi-scale connection layer combines the output results of the above multi-scale mapping units to obtain F ₃ :

in,

are the output results of the three groups of convolutional networks of different scales; Then, the pooling layer downsamples the output of the multi-scale connection layer, and the output of the pooling layer is _F4 :

The convolutional layer maps the result of the pooling layer, _F4 , to _F5 : F ₅ =W ₅ *F ₄ +B ₅ Among them, W ₅ is the weight coefficient matrix of the convolution network in the multi-scale mapping unit, B ₅ is the deviation matrix, and * represents the convolution operation; The BReLU activation layer uses the bilateral modified linear unit BReLU activation function to perform nonlinear regression on the output result F ₅ of the convolution layer, and obtains F ₆ : F ₆ (x)=min(a _max ,F ₅ (x)) Among them, a _max is the upper amplitude value of the bilateral modified linear unit BReLU activation function; Finally, let t=F ₆ , the output layer outputs t, and t is the transmittance matrix of the input image block; (3) Construct the loss function: When there is only a single training sample, the loss function is:

When there are multiple training samples, the loss function is:

Among them, t represents the transmission map of the training sample, n is the number of training samples, λ represents the attenuation parameter, W _ji represents the weight coefficient matrix of the training sample i, and t ^* represents the actual transmission matrix of the training sample; (4) For each W _ji , use a Gaussian distribution with a mean value of 0 and a standard deviation of 0.001 to randomly initialize the components in W _ji ; initialize B _ji to 0, and B _ji represents the deviation matrix of training sample i; initialize ΔW _ji = 0, ΔB _ji = 0; (5) For each sample i, use the backpropagation algorithm to find the partial derivatives of W _ji and B _ji :

and

Find the amount of change in W _ji and B _ji :

(6) Update:

(7) Substitute the updated W _ji and B _ji into the loss function, and repeat steps (5) to (7) until the value of the loss function is the smallest. step (8); (8) Input the new foggy image into the trained multi-scale deep convolutional network model, and the obtained output is used as the initial transmittance of the new foggy image; (9) Estimate the atmospheric light intensity A when the new foggy image is taken; restore the corresponding fog-free image according to the atmospheric scattering model. 2. a kind of single image dehazing method based on convolutional neural network according to claim 1, is characterized in that, described t ^* is calculated according to atmospheric scattering model, and atmospheric scattering model is: I=J×t ^* +A(1-t ^* ) Among them, I represents the light intensity matrix of the training sample, J represents the light intensity matrix of the image block corresponding to the training sample in the original haze-free image, and A is the atmospheric light intensity A of the foggy image corresponding to the training sample. 3. a kind of single image dehazing method based on convolutional neural network according to claim 2, is characterized in that, the method for the atmospheric light intensity A when described estimation hazy image is photographed is: Estimate the dark channel map for a hazy image:

Among them, I ^dark represents the dark channel map estimation result of the pixel matrix I of the foggy image, I ^c represents the value of a certain color channel of I, and y represents the pixel point; Sort the pixel values of the obtained dark channel map in descending order, select the top one thousandth pixel point as the candidate point, correspond the candidate point to the corresponding position in the foggy image, and calculate the corresponding position of the candidate point in the foggy image. The brightness value of the location, and the calculated maximum brightness value is used as the estimated value of atmospheric light intensity A.

Images