CN110728643A - Low-illumination band noise image optimization method based on convolutional neural network - Google Patents

Abstract

The present invention discloses a low-illuminance and noise-based image optimization method based on a convolutional neural network. Use skip connections to connect from the convolutional layer to the corresponding mirror deconvolutional layer; S2, collect normal illumination images, and artificially synthesize the corresponding low-illumination and noisy images, and use the data to enhance and expand the images as training data. The training data trains the convolutional neural network model, dynamically adjusts the learning rate of the network model for training, and obtains the trained LLED-Net convolutional neural network; S3, through the trained LLED-Net convolutional neural network model, the collected real Reconstruction and optimization of low-illumination and noisy images to achieve high-quality image reconstruction. The present invention can use the model to automatically learn the characteristics of the image under the condition that the noise intensity and brightness of the low-illumination noise image are unknown, improve the image brightness, remove the image noise, and reconstruct a high-quality picture.

Description

A low-illumination and noisy image optimization method based on convolutional neural network

technical field

The invention relates to the field of image processing, in particular to a low-illuminance and noisy image optimization method based on a convolutional neural network.

Background technique

Taking high-quality pictures and videos with camera and other imaging equipment can be useful in many situations, but not all images captured are of good quality. Due to insufficient lighting at the time of shooting and various electrical noise, mechanical noise, channel noise and other noise interference during the transmission process, the captured image is usually dark as a whole, accompanied by image noise, resulting in image degradation. It is difficult to clearly identify objects or textures, so it is necessary to improve the quality of low-light images.

At present, the mainstream algorithms for improving low luminance noise images at home and abroad are histogram equalization (HE), contrast limited adaptive histogram equalization (CLAHE), and gamma correction (GC). , although these methods improve the image quality of low-luminance noise images, the effect is still not ideal, and they have their own shortcomings, which are easy to cause color distortion in the image, and a large number of white blocks in the image.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a low-illumination noise image optimization method based on a convolutional neural network, which can automatically learn the characteristics of the image by using the model under the condition that the noise intensity and brightness of the low-illumination noise image are unknown, and improve the image brightness. , remove image noise and reconstruct high-quality images.

In order to achieve the above object, the present invention realizes through the following technical solutions:

A low-light and noisy image optimization method based on convolutional neural network, including:

Step S1, constructing an LLED-Net convolutional neural network; the LLED-Net convolutional neural network is a convolution-deconvolution structure, including N convolutional layers and N-layer corresponding mirrored deconvolutional layers, using skipping The connection is connected from the convolution layer to the corresponding mirror deconvolution layer, and the skip connection means that the output of a single convolution layer is passed to the next layer of convolution layer, and the output is passed to the corresponding mirror deconvolution layer; The LLED-Net convolutional neural network uses structural similarity loss as the loss function of the convolutional neural network;

Step S2: Collect normal illumination images, and artificially synthesize corresponding low-illuminance noisy images, use the low-illuminance noisy images for data enhancement and expansion as training data, and use the training data to train the LLED-Net convolution. Neural network model, dynamically adjust the learning rate of the network model for training, and obtain a trained LLED-Net convolutional neural network;

Step S3: Reconstructing and optimizing the collected real low-illuminance image with noise by using the trained LLED-Net convolutional neural network model to achieve high-quality image reconstruction.

Preferably, the LLED-Net convolutional neural network includes ten convolutional layers for processing images and ten corresponding mirrored deconvolutional layers; the convolutional layers are used as feature extractors, and the convolutional layers pass through the convolutional layers. After forwarding, 10 deconvolution layers are followed to restore the details in the image, and the number of deconvolution layer feature maps is the same as the number of corresponding convolution layer feature maps; the LLED-Net convolutional neural network model uses skip connections Connect from the convolutional layer to the corresponding mirror deconvolutional layer, sum up the convolutional layer feature maps transmitted layer by layer and use it as the input of the deconvolutional layer, and correct it through skip connections before passing it to the next layer deconvolution layer.

Preferably, the structure of the LLED-Net convolutional neural network further includes:

The ten-layer convolutional layer is recorded as the first convolutional layer, the second convolutional layer... until the tenth convolutional layer from the input end to the output end;

The ten deconvolution layers are sequentially recorded as the first deconvolution layer, the second deconvolution layer... until the tenth deconvolution layer from the input end to the output end;

The i-th convolutional layer is correspondingly skip-connected with the 11-i-th deconvolutional layer, and 1≤i≤10.

Preferably, the structure of the LLED-Net convolutional neural network further includes:

The number of convolution kernels from the first convolutional layer to the fourth convolutional layer is 128, the number of convolutional kernels from the fifth convolutional layer to the seventh convolutional layer is 256, and the eighth convolutional layer The number of convolution kernels up to the tenth convolution layer is 512;

The number of convolution kernels from the first deconvolution layer to the second deconvolution layer is 512, the number of convolution kernels from the third deconvolution layer to the fifth deconvolution layer is 256, and the number of convolution kernels of the sixth deconvolution layer is 256. The number of convolution kernels from the deconvolution layer to the ninth deconvolution layer is 128, and the number of convolution kernels of the tenth deconvolution layer is 3.

Preferably, the convolutional neural network-based low-illumination image optimization method with noise further includes the following process: after each convolution or deconvolution operation, padding is used to perform a zero-filling operation; each convolution layer or inverse The convolutional layers are activated with a non-linear rectification function after operation; the kernel size of all convolutional and deconvolutional layers is set to 3 × 3.

Preferably, in the step S1, it further comprises:

Taking the structural similarity SSIM loss as the loss function of the LLED-Net convolutional neural network, the formula for structural similarity is as follows:

是x的方差；

是y的方差；σ_xy是x与y的协方差；c₁＝(k₁L)²，c₂＝(k₂L)²，是用来维持稳定的常数；L是像素值的动态范围；k₁＝0.01，k₂＝0.03；where μ _x is the mean value of x; μ _y is the mean value of y;

is the variance of x;

is the variance of y; σ _xy is the covariance of x and y; c ₁ =(k ₁ L) ² , c ₂ =(k ₂ L) ² , are constants used to maintain stability; L is the dynamic range of pixel values ; k ₁ =0.01, k ₂ =0.03;

The loss function formula is as follows:

Among them, N represents the number of training data sets; x represents the artificially synthesized low-illuminance image with noise; X represents the artificially synthesized low-illumination image dataset with noise; y represents the normal illumination image; Y represents the normal illumination image dataset.

Preferably, in the step S2, it further comprises:

Step S21: Select multiple noise-free images taken under normal lighting, rotate all noise-free images clockwise by 90°, 180°, and 270° in turn, and then perform horizontal flipping, and then perform data enhancement and data expansion on all images. Then it becomes the training data to be processed;

Step S22, adding Gaussian noise to each image processed in step S21, and then performing non-linear adjustment to become a low-brightness image, then the processed image becomes a low-quality image with random brightness and random Gaussian noise;

Step S23, randomly cutting a picture block with a pixel of 41×41 for each low-quality image processed in the step S22, and using the obtained picture set as a training data set;

Step S24, input the training data set obtained in step S23 into the LLED-Net convolutional neural network to realize forward propagation; wherein, each traversed 4000 samples is one generation, and the learning rate is reduced to the original 0.1 every two generations, The number of iterations is 10 generations, and finally the trained LLED-Net convolutional neural network model is obtained.

Preferably, in the step S22, it further comprises:

The noise intensity σ of Gaussian noise is randomly selected in the range of (0, 25); the image brightness value is randomly selected between γ (2, 5).

Compared with the prior art, the beneficial effects of the present invention are: the present invention constructs an end-to-end convolutional neural network, artificially synthesizes training data, without human intervention, autonomously learns the main features of the image from the training data, constructs and removes noise and the optimal model for improving image brightness; the present invention does not need to know the brightness and noise intensity of the image to be processed, can use the trained model to quickly process unknown low-illuminance noise images, reconstruct high-quality images, and has good generalization effect and high robustness. Robustness; the processing effect of the present invention is excellent, and the brightness of the reconstructed image, the degree of color restoration, the image texture, and the degree of noise removal have been greatly improved compared with the previous technology.

Description of drawings

1 is a schematic flowchart of a method for optimizing a low-illuminance image with noise based on a convolutional neural network of the present invention;

2 is a structural diagram of an overall model in the method for optimizing a low-illuminance image with noise based on a convolutional neural network of the present invention;

Fig. 3 is the low illumination noise image of the present invention;

FIG. 4 is an image processed by the LLED-Net convolutional neural network model of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

As shown in FIG. 1-FIG. 4, the present invention discloses a method for optimizing low-illuminance images with noise based on convolutional neural network, which includes the following steps:

Step S1, constructing an LLED-Net convolutional neural network; wherein, the LLED-Net convolutional neural network includes N (for example, 10) convolutional layers and N layers (10 layers) corresponding mirror deconvolution layers, using Skip connections connect from each convolutional layer to the corresponding mirror deconvolutional layer, as shown in Figure 2; and use the Structural Similarity (SSIM) loss as the loss function of the convolutional neural network, as shown in Figure 1.

Step S2, collecting normal illumination images, and artificially synthesizing corresponding low-illuminance and noisy images, performing data enhancement and expansion on the low-illumination and noisy images as training data (the training data is a set of low-illumination and noisy images), and using the training data The LLED-Net convolutional neural network model is trained with data, and the learning rate of the network model is dynamically adjusted for training to obtain a trained LLED-Net convolutional neural network.

Step S3: Reconstructing and optimizing the collected real low-illuminance image with noise by using the trained LLED-Net convolutional neural network model to achieve high-quality image reconstruction.

In this embodiment, the LLED-Net convolutional neural network model is a convolution-deconvolution structure. First, 10 convolutional layers are used to process the image, and the number of feature maps of the convolutional layer is gradually increased; the convolutional layer is used as feature extraction. It extracts the main features of the objects in the image, improves the image brightness and removes image noise; after forwarding through the convolution layer, 10 deconvolution layers are followed to restore the details in the image. The number of feature maps in the deconvolution layer is the same as The corresponding convolutional layer feature maps have the same number of mirrors, which are used to reconstruct high-quality images; skip connections refer to not only passing the output of a single convolutional layer to the next convolutional layer, but also passing the output to the corresponding mirror The deconvolution layer, that is, the present invention uses skip connections to connect from the convolution layer to the corresponding mirror deconvolution layer. The convolutional feature maps passed layer by layer are summed item by item and used as the input of deconvolution, corrected by skip connection and then passed to the next deconvolution layer; avoid the occurrence of convolutional neural network in the training process Gradient disappearance and gradient explosion make the network more stable in the training process, increase the network convergence speed and reduce the training time.

Further, the LLED-Net convolutional neural network network structure is:

(1) As shown in Figure 2, the 10-layer convolutional layers are sequentially recorded as convolutional layer 1, convolutional layer 2...convolutional layer 9, and convolutional layer 10 along the input end to the output end; The input end to the output end are denoted as the deconvolution layer 11 , the deconvolution layer 12 , the deconvolution layer 19 , and the deconvolution layer 20 in sequence. Correspondingly, convolutional layer 1 is connected to deconvolution layer 20 using skip connections, convolutional layer 2 is connected to deconvolution layer 19 using skip connections, and convolutional layer 3 is connected to deconvolution layer 18 using skip connections. ...convolutional layer 10 is connected to deconvolutional layer 11 using skip connections.

(2) The convolution layer uses convolution kernels to perform convolution operations on the image. The number of convolution kernels in convolution layers 1 to 4 is 128, the number of convolution kernels in convolution layers 5 to 7 is 256, and the number of convolution layers 8 to 10 The number of layer convolution kernels is 512; the deconvolution layer uses deconvolution kernels to perform deconvolution operations on the image. The number of deconvolution kernels in deconvolution layers 11 to 12 is 512, and the number of kernels in convolution layers 8 to 10 layers is 512. The same, the number of deconvolution kernels in the deconvolution layers 13 to 15 is 256, which is the same as the number of kernels in the convolution layers 5 to 7, and the number of deconvolution kernels in the deconvolution layers 16 to 19 is 128, which is the same as the convolution layer. The number of kernels in layers 1 to 4 is the same, and the number of deconvolution kernels in the 20-layer deconvolution layer is 3, which is used for image reconstruction.

In this embodiment, padding is used to perform a zero-filling operation after each convolution operation or after a deconvolution operation. After each convolution layer or deconvolution layer, use the nonlinear rectification function (Relu) for activation (that is, use the nonlinear rectification function for activation after each convolution operation, and use it again after each deconvolution operation. The nonlinear rectification function is activated). The kernel size of all convolutional and deconvolutional layers is set to 3 × 3.

In the step S1, the following process is further included:

Taking structural similarity (SSIM) loss as the loss function of the LLED-Net convolutional neural network, the formula for structural similarity is:

是x的方差；

是y的方差；σ_xy是x与y的协方差；c₁＝(k₁L)²，c₂＝(k₂L)²，是用来维持稳定的常数；L是像素值的动态范围；k₁＝0.01，k₂＝0.03。where μ _x is the mean value of x; μ _y is the mean value of y;

is the variance of x;

is the variance of y; σ _xy is the covariance of x and y; c ₁ =(k ₁ L) ² , c ₂ =(k ₂ L) ² , are constants used to maintain stability; L is the dynamic range of pixel values ; k ₁ =0.01, k ₂ =0.03.

In addition, the loss function formula is as follows:

Among them, N represents the number of training data sets; x represents a single artificially synthesized low-illuminance image with noise; X represents a synthetic low-illumination image with noise dataset; y represents a single normal illumination image; Y represents a normal illumination image dataset.

In the step S2, the following process is further included:

Step S21, select multiple (for example, 300) noise-free images taken under normal lighting, rotate all images clockwise by 90°, 180°, and 270° in turn, then flip all images horizontally, and perform data analysis on all images. The augmented data becomes the training data to be processed.

In step S22, Gaussian noise is added to each image processed in step S21, and the noise intensity σ is randomly selected within the range of (0, 25); The brightness value γ is randomly selected between (2, 5), and the processed image becomes a low-quality image with random brightness and random Gaussian noise. The above image processing is realized by matlab software.

Step S23: Randomly cut a picture block with a pixel size of 41×41 for each picture processed in the step S22, and use the obtained picture set as a training data set.

Step S24, input the training data set obtained in step S23 into the LLED-Net convolutional neural network to realize forward propagation; each traversal of 4000 samples is one generation, the learning rate is reduced to the original 0.1 every two generations, and the number of iterations is 10 Generation; finally get the trained LLED-Net convolutional neural network model.

In the step S3, the following process is further included:

The collected low-illumination noise pictures are input into the trained LLED-Net convolutional neural network model to obtain an optimized reconstructed image. Compared with the original image, the brightness, color fidelity, texture details and image quality of the reconstructed image are all better. There is a great improvement, as shown in Figure 3.

To sum up, the present invention constructs an end-to-end convolutional neural network, artificially synthesizes training data, without human intervention, autonomously learns the main features of the image from the training data, and constructs an optimal model for removing noise and improving image brightness. ; No need to know the brightness and noise intensity of the image to be processed, can use the trained model to quickly process unknown low-illumination noise images, reconstruct high-quality images, and have good generalization effect and high robustness; The processing effect is excellent, whether it is The brightness, color reproduction, image texture, and noise removal of the reconstructed image are all improved compared to the previous techniques.

While the content of the present invention has been described in detail by way of the above preferred embodiments, it should be appreciated that the above description should not be construed as limiting the present invention. Various modifications and alternatives to the present invention will be apparent to those skilled in the art upon reading the foregoing. Accordingly, the scope of protection of the present invention should be defined by the appended claims.

Claims (8)

1. A low-illumination noisy image optimization method based on a convolutional neural network is characterized by comprising the following steps:

step S1, constructing an LLED-Net convolution neural network; the LLED-Net convolutional neural network is of a convolutional-deconvolution structure and comprises N convolutional layers and N corresponding mirror image deconvolution layers, and the convolutional layers are connected to the corresponding mirror image deconvolution layers by using jump connection, wherein the jump connection means that the output of a single convolutional layer is transmitted to the next convolutional layer and the output is transmitted to the corresponding mirror image deconvolution layer; the LLED-Net convolutional neural network uses the loss of structural similarity as a loss function of the convolutional neural network;

step S2, collecting normal illumination images, artificially synthesizing corresponding low-illumination-intensity-band noise images, performing data enhancement and quantity expansion on the low-illumination-intensity-band noise images to obtain training data, training the LLED-Net convolutional neural network model by using the training data, and dynamically adjusting the learning rate of the network model to train to obtain a trained LLED-Net convolutional neural network;

and step S3, reconstructing and optimizing the acquired real low-illumination band noise image through the trained LLED-Net convolutional neural network model, and realizing the reconstruction of a high-quality image.

2. The convolutional neural network-based low-illumination noisy image optimization method of claim 1,

the LLED-Net convolutional neural network comprises ten convolutional layers for processing images and ten corresponding mirror image deconvolution layers;

the convolutional layer is used as a feature extractor, after being forwarded by the convolutional layer, details in the images are restored by connecting 10 deconvolution layers, and the number of deconvolution layer feature images is the same as the number of corresponding convolutional layer feature images in a mirror image manner;

the LLED-Net convolutional neural network model is connected to a corresponding mirror image deconvolution layer from a convolution layer by using jump connection, the convolution layer characteristic graphs transmitted layer by layer are summed item by item and are used as the input of the deconvolution layer, and the convolution layer characteristic graphs are corrected by the jump connection and then transmitted to the next deconvolution layer.

3. The convolutional neural network-based low-illumination noisy image optimization method of claim 2,

the structure of the LLED-Net convolutional neural network further comprises:

the ten convolutional layers are sequentially recorded as a first convolutional layer, a second convolutional layer … to a tenth convolutional layer from the input end to the output end;

the ten deconvolution layers are sequentially recorded as a first layer deconvolution layer, a second layer deconvolution layer … to a tenth layer deconvolution layer from the input end to the output end;

the ith convolution layer is correspondingly connected with the 11 th-i th deconvolution layer in a jump way, and i is more than or equal to 1 and less than or equal to 10.

4. The convolutional neural network-based low-illumination noisy image optimization method of claim 3,

the structure of the LLED-Net convolutional neural network further comprises:

the number of convolution kernels from the first layer of convolution layer to the fourth layer of convolution layer is 128, the number of convolution kernels from the fifth layer of convolution layer to the seventh layer of convolution layer is 256, and the number of convolution kernels from the eighth layer of convolution layer to the tenth layer of convolution layer is 512;

the number of convolution kernels from the first layer of deconvolution layer to the second layer of deconvolution layer is 512, the number of convolution kernels from the third layer of deconvolution layer to the fifth layer of deconvolution layer is 256, the number of convolution kernels from the sixth layer of deconvolution layer to the ninth layer of deconvolution layer is 128, and the number of convolution kernels from the tenth layer of deconvolution layer is 3.

5. The convolutional neural network-based low-illumination noisy image optimization method of claim 1,

further comprising the following processes:

after each convolution or deconvolution operation, padding is adopted for zero padding operation;

activating each convolution layer or each deconvolution layer by using a nonlinear rectification function after operation;

the convolution kernel size of all convolutional and deconvolution layers is set to 3 × 3.

6. The convolutional neural network-based low-illumination noisy image optimization method of claim 1,

the step S1 further includes:

taking the structural similarity SSIM loss as a loss function of the LLED-Net convolutional neural network, wherein the structural similarity formula is as follows:

wherein, mu_xIs the average value of x; mu.s_yIs the average value of y;is the variance of x;is the variance of y; sigma_xyIs the covariance of x and y; c. C₁＝(k₁L)²，c₂＝(k₂L)²Is a constant used to maintain stability; l is the dynamic range of the pixel value; k is a radical of₁＝0.01，k₂＝0.03；

The loss function is formulated as follows:

wherein N represents the number of training data sets; x represents a artificially synthesized low-illumination noisy image; x represents a synthetic low-illumination noisy image dataset; y represents a normal illumination image; y represents a normal-light image dataset.

7. The convolutional neural network-based low-illumination noisy image optimization method of claim 1,

the step S2 further includes:

s21, selecting a plurality of noise-free images shot under normal illumination, clockwise rotating all the noise-free images by 90 degrees, 180 degrees and 270 degrees, horizontally turning, and performing data enhancement and data expansion on all the images to obtain training data to be processed;

step S22, Gaussian noise is added to each image processed in the step S21, then nonlinear adjustment is carried out to change the image into a low-brightness image, and the processed image becomes a low-quality image with random brightness and random Gaussian noise;

step S23, randomly cutting picture blocks with 41 × 41 pixels for each low-quality image processed in the step S22, and taking the obtained picture set as a training data set;

step S24, inputting the training data set obtained in the step S23 into the LLED-Net convolution neural network to realize forward propagation; wherein, every 4000 samples are traversed to be one generation, every two generations reduce the learning rate to be 0.1 of the original, the iteration times is 10 generations, and finally the trained LLED-Net convolution neural network model is obtained.

8. The convolutional neural network-based low-illumination noisy image optimization method of claim 7,

the step S22 further includes:

the noise intensity σ of the gaussian noise is randomly selected in the range of (0, 25);

the picture brightness values are randomly chosen between gamma (2, 5).

Images