CN110852972B - Single image rain removing method based on convolutional neural network - Google Patents

Abstract

The invention discloses a method for removing rain from a single image based on a convolutional neural network. The present invention First, we do not use guided filtering or other filtering to separate the image in order to preserve the original information of the image as much as possible. Second, we propose our RK block to replace the residual block to extract features more efficiently. Finally, we propose a feature transformation concatenation operation to handle multi-scale rainlines. Furthermore, the batch regularization operation assumes that the features all have the same distribution, however different rainlines have different directions, colors and shapes, so we remove all batch regularization operations from the network. Beneficial effects of the present invention: based on the convolutional neural network, a relatively simple and efficient single-step single-stream rain removal network model is designed, so as to better repair images with rain, while maintaining the difference between repair quality and model size. balance.

Description

A single image rain removal method based on convolutional neural network

technical field

The invention relates to the field of image processing, in particular to a method for removing rain from a single image based on a convolutional neural network.

Background technique

In images collected on rainy days, rain lines can seriously degrade human perception of the background and the performance of other machine vision applications, such as object detection methods. For a long time, researchers have hoped to design a single image rain removal network that can remove rain lines while repairing the background. The single image rain removal network method is mainly divided into three categories: multi-stream rain removal network, multi-step rain removal network and single-step single-stream rain removal network. Multi-stream rain removal networks such as Joint Rain Detection and Removal (JORDER) and Multi-stream Dense Network (DID-MDN) will input rain images into different branches of the network, and then fuse the image features obtained from different branches. The DID-MDN proposed by Zhang et al. uses a rainline density prediction sub-network to predict the density label of the overall rainline in the image, and then concatenates it with the feature maps extracted by three other multi-scale image feature extraction sub-networks for Subsequent fixes. Multi-step rain removal networks such as Recurrent SE Contect AggregationNet (RESCAN), Progressive ResNet (PRN) and Progressive Recurrent Network (PReNet) iteratively remove rain multiple times by taking the output of the previous step of the network as the input of the next step. However, since the network needs to perform multiple transformations between images and features, the deraining effect is reduced and the error of each iteration may be accumulated. The single-step single-stream rain removal network aims to repair images with rain in one go with a single network. However, since people have always assumed that a single network in a single step is not enough to capture all rainline information in different directions and different scales, except for the Deep Detail Network (DDN) proposed by Fu et al., almost all rain removal networks are multi-stream or Multi-step design pattern.

The traditional technology has the following technical problems:

The multi-stream rain removal network requires multiple sub-networks to cooperate to remove rain. As the network design becomes more and more complex, the training of the network becomes more and more tricky, so it is difficult to verify the effectiveness of each sub-network. sex. The multi-step rain removal network depends on the previous state every time the rain is removed, and the error may be gradually accumulated. In addition, both multi-stream deraining networks and multi-step deraining networks have a large number of parameters that make them difficult to apply to real applications. Although the single-step single-stream network has a simple overall design and is easy to train, the effect is far inferior to the multi-step rain removal network and the multi-stream rain removal network due to the inefficient network structure.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a single image rain removal method based on a convolutional neural network. Based on the convolutional neural network, a relatively simple and efficient single-step single-stream rain removal network model is designed, so as to be more Good restoration of rainy images while maintaining a balance between restoration quality and model size.

In order to solve the above technical problems, the present invention provides a single image deraining method based on convolutional neural network, including: firstly, without using guided filtering or other filtering to separate the images to preserve the network as much as possible; secondly, using RK block to replace residual block to extract features more efficiently; use feature transformation concatenation operation to process multi-scale rainlines;

Among them, the network architecture of the convolutional neural network consists of four parts:

(1) An input layer consisting of a convolutional layer activated by the ReLU function to extract shallower features:

X ⁰ =σ((Conv ⁰ (X))); (7)

(2) A series of RK blocks to extract higher-level features:

where the RK block is composed of four different nonlinear mapping modules G and a channel attention weighting operation Squeeze-and-Excitation (SE):

Each nonlinear mapping module consists of two convolutional layers stacked together:

After the extraction of four different nonlinear mapping modules, the SE module is introduced to weight the information of the useful channels one by one and suppress the information of the useless channels; (3) The deeper convolutional layer in the convolutional neural network The larger the receptive field is, it is unfavorable for processing small-scale rainlines; therefore, after each RK block extracts features, a Feature Transmission joint will be used to convert the pre-processing features and the post-processing features. Fusion to prevent loss of critical information:

Among them, FT represents the feature transformation connection, [ , ] is the feature stitching operation; (4) Finally, a convolutional layer without activation function is used to decode the extracted depth features into the repaired rain-free image:

A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any one of the methods when the processor executes the program.

A computer-readable storage medium having a computer program stored thereon, the program implementing the steps of any one of the methods when executed by a processor.

A processor for running a program, wherein the program executes any one of the methods when the program is running.

Beneficial effects of the present invention:

Based on the convolutional neural network, a relatively simple and efficient single-step single-stream rain removal network model is designed to better repair images with rain, while maintaining the balance between repair quality and model size.

Description of drawings

FIG. 1 is a schematic diagram of comparison 1 of image restoration results with rain in the method for removing rain from a single image based on a convolutional neural network according to the present invention.

FIG. 2 is a schematic diagram of comparison 2 of the restoration results of images with rain in the method for removing rain from a single image based on a convolutional neural network according to the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

In order to extract the features of images with rain more accurately and reduce the size of the model to make the rain removal model more widely used, the present invention designs a new single-step single-stream rain removal framework, and proposes a new feature extraction, fusion module and network Parameter sharing method to repair rainy images more efficiently.

(1) Traditional single-step single-stream rain removal network based on convolutional neural network

The traditional single-step single-stream rain removal network structure based on convolutional neural network can be represented by the following four parts: (1) Use guided filtering to separate the detail components and basic components of the input rain image:

是作为网络输入的图像细节成分；(2)使用一个的卷积层抽取网络输入的较浅层特征，然后使用批正则化操作(BatchNormalization)对特征归一化处理，并以ReLU函数作为激活函数：where X is the input rain image, X _base is the basic component of the network,

is the image detail component as the network input; (2) use a convolutional layer to extract the shallower features of the network input, and then use the batch normalization operation (BatchNormalization) to normalize the features, and use the ReLU function as the activation function :

where Conv ¹ represents the first convolutional layer, BN is the batch regularization operation, and σ is the ReLU function; (3) Use a series of residual blocks to extract the deep features of the image:

N是网络层的总数,Res表示堆叠的残差块，他们每个由两层卷积层、批正则化操作，ReLu函数以及一条直连(shortcut)组合而来，具体地，可用公式表示如下in

N is the total number of network layers, and Res represents the stacked residual blocks, each of which is composed of two convolution layers, batch regularization operations, ReLu function and a shortcut (shortcut). Specifically, the formula can be expressed as follows

(4) Then use a convolutional layer without activation function and batch regularization operation to decode the extracted depth features into the inpainted rain-free image:

Usually N is set to 26.

The traditional single-step single-stream rain-removing network based on convolutional neural network separates the detail components of the rain-bearing image with a guided filter and then inputs it. However, under the condition of heavy rainlines, there are still a lot of rainlines that are not separated, which reduces the quality of the repaired image. Also, the network is inefficient due to the use of traditional residual blocks to extract features. At present, with the deepening of neural network research, many more efficient network modules and design methods have been proposed. In addition, due to the different sizes of rainlines in different regions and different distances from the camera, the rainlines have multi-scale characteristics. However, the traditional single-step single-stream rain removal network based on convolutional neural network does not consider the network structure of multi-scale rain line in structure.

(2) The algorithm of this paper

First, we do not use guided filtering or other filtering to separate the image to preserve the original information of the image as much as possible. Second, we propose our RK block to replace the residual block to extract features more efficiently. Finally, we propose a feature transformation concatenation operation to handle multi-scale rainlines. Furthermore, the batch regularization operation assumes that the features all have the same distribution, however different rainlines have different directions, colors and shapes, so we remove all batch regularization operations from the network. Our network architecture consists of four parts: (1) an input layer consisting of a convolutional layer activated by a ReLU function to extract shallower features:

X ⁰ =σ((Conv ⁰ (X))); (7)

(2) A series of RK blocks to extract higher-level features:

where the RK block is composed of four different nonlinear mapping modules G and a channel attention weighting operation Squeeze-and-Excitation (SE):

Each nonlinear mapping module consists of two convolutional layers stacked together:

After the extraction of four different nonlinear mapping modules, we introduce the SE module to weight the information of the useful channels channel by channel and suppress the information of the useless channels; (3) the deeper convolutional layers in the convolutional neural network The larger the receptive field of , which is disadvantageous for dealing with small-scale rainlines. Therefore, after each RK block extracts features, we will use a Feature Transmission joint to fuse the pre-processing features and the post-processing features to prevent the loss of key information:

where FT stands for feature transformation connection, [ , ] is the feature stitching operation; (4) Finally, we use a convolutional layer without activation function to decode the extracted depth features into the inpainted rain-free image:

The effect of the present invention in performing rain image inpainting significantly exceeds other state-of-the-art rain removal networks. See the following analysis for details.

Rain100H is a widely used dataset for rain removal tasks, which contains 100 background images and corresponding 100 synthetic images with rain. Each image with rain contains five rain lines with different directions and scale shapes. In order to test the actual effect of the algorithm in this paper, we compare the repair effect of the traditional rain removal network and the algorithm in this paper. We adopt the signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the inpainted image to the background image as quantitative measures of the inpainting effect. Table 1 gives the statistical values of the quantitative experiments, and Fig. 1 and Fig. 2 give some image samples.

Table 1 Comparison of the average PSNR and SSIM values between the traditional rain removal network and the network repaired in this paper

DDN JORDER RESCAN PRN PReNet This article network PSNR 21.92 26.54 28.88 28.07 29.46 30.26 SSIM 0.764 0.835 0.866 0.884 0.899 0.909

The data in Table 1 show that our network has achieved the best performance in both metrics. The previous single-stream single-step rain removal network DDN only achieved the final score. Our proposed improvements improve the network's efficiency in extracting image features and the ability to handle multi-scale rainlines while ensuring that the key features of the image are preserved to the greatest extent possible.

Referring to Figure 1, in this example, the images repaired by other methods have obvious white smears that seriously degrade the image repair quality. As can be seen from the zoomed-in details, the lawn repaired by our method is smoother than other methods and retains more details such as the edges of buildings.

Referring to Figure 2, in this case, our method still outperforms other methods. As can be seen from the enlarged picture, the granite pillars repaired by other methods are always blurred, while the method in this paper can retain clear details.

The above-mentioned embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention is subject to the claims.

Claims (4)

1. a single image rain removal method based on convolutional neural network, is characterized in that, comprises: Use RK block instead of residual block to extract features more efficiently; use feature transformation concatenation operation to deal with multi-scale rainlines; Among them, the network architecture of the convolutional neural network consists of four parts: (1) An input layer consisting of a convolutional layer activated by the ReLU function to extract shallower features:

(7) (2) A series of RK blocks to extract higher-level features:

(8) where the RK block is composed of four different nonlinear mapping modules G and a channel attention weighting operation Squeeze-and-Excitation (SE):

(9) Each nonlinear mapping module consists of two convolutional layers stacked together:

(10)

(11)

(12)

(13) After four different nonlinear mapping modules are extracted, SE module is introduced to weight the information of useful channels one by one and suppress the information of useless channels; (3) The deeper the convolutional layer in the convolutional neural network, the larger the receptive field, which is unfavorable for processing small-scale rain lines; therefore, after each RK block extracts features, a feature transfer connection (Feature Transmission joint) to fuse the pre-processing features and the post-processing features to prevent the loss of key information:

(14) Among them, FT represents the feature transformation connection, [ , ] is the feature splicing operation; (4) Finally, use a convolutional layer without activation function to decode the extracted depth features into the inpainted rain-free image:

(15). 2. A computer equipment, comprising memory, processor and a computer program stored on the memory and running on the processor, wherein the processor implements the steps of the method of claim 1 when executing the program . 3. A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the steps of the method of claim 1 are implemented. 4. A processor, wherein the processor is used for running a program, wherein the method of claim 1 is executed when the program is running.

Images