CN112767280B - Single image raindrop removing method based on loop iteration mechanism - Google Patents

Abstract

The invention relates to a single image raindrop removing method based on a loop iteration mechanism. The method comprises the following steps: preprocessing the training image pair of the original raindrop degradation image and the clean image to obtain an image block data set consisting of the training image pair of the original raindrop degradation image and the clean image; designing a convolution neural network for removing raindrops of a single image by using a motivation for continuously iterating to remove rain; designing a target loss function loss for optimizing the network, taking the image block data set as training data, calculating the gradient of each parameter in the designed image raindrop removal convolutional neural network by using a back propagation method according to the designed target loss function loss, updating the parameter by using a random gradient descent method, and finally learning the optimal parameter of the model; and inputting the image to be detected, and predicting and generating a clean image after raindrops are removed by using the trained model. The method can obviously improve the raindrop removal performance of the image, and greatly reduce the network parameter size of the image.

Description

A single image raindrop removal method based on loop iteration mechanism

technical field

The invention relates to the fields of image and video processing and computer vision, in particular to a method for removing raindrops from a single image based on a loop iteration mechanism.

Background technique

With the rapid development of the Internet and multimedia technology, images have become an indispensable part of human communication and information transmission, and are of great significance to the development of all aspects of modern society. However, the acquisition of images will inevitably be carried out in an outdoor environment, and the outdoor environment will inevitably be affected by some bad weather, such as rainy days, foggy days, snowy days, etc. These bad weather can bring a series of visual quality degradations to the captured images and videos, such as contrast, saturation, visibility, etc. Rainy days are a common weather phenomenon in daily life. When raindrops adhere to glass windows, windshields or camera lenses, they obstruct the visibility of background scenes, reduce the quality of images, and cause severe image degradation. This makes the detailed information on the image unrecognizable, greatly reduces the use value of the image, and brings certain difficulties to advanced visual understanding tasks such as object detection, pedestrian re-identification, and image segmentation. In rainy weather, raindrops inevitably adhere to the lens of the camera. These attached raindrops may blur parts of the background scene in the image or occlude parts of the foreground scene in the image, so it is of great significance to remove raindrops from a single image to restore a clean background.

Although the research on single image rain streak removal has been well explored, there are few studies on single image rain drop removal, and these well-explored single image rain streak removal methods cannot be directly applied to a single image Raindrop removal. Although raindrops are not as dense as rain streaks, the size of raindrops is generally larger than that of rain streaks and will completely block the background. And the appearance of raindrops is affected by many parameters, its shape is usually not the same as thin and vertical rain streaks. At the same time, the physical modeling of raindrops is completely different from that of rain streaks, which makes the removal of raindrops more difficult.

The current methods for raindrop removal from a single image are generally divided into two categories, namely model-based methods and existing deep learning-based methods. Model-based methods usually first use a filter to decompose it into high-frequency and low-frequency parts, and then learn to distinguish between rain and non-rain components in the high-frequency part through dictionary learning. Most of the model-based methods rely on manually preset parameters for image feature extraction and performance optimization, and cannot extract the features of raindrops well, resulting in poor performance in raindrop removal.

The deep learning-based method is a data-driven method that uses a large amount of data to train a convolutional neural network. The powerful feature learning and representation ability exhibited by the convolutional neural network can better extract image features and achieve a better raindrop removal effect. However, these deep learning-based methods also have a problem, that is, they cannot balance the relationship between the performance of raindrop removal and the amount of network parameters. These methods either achieve good performance, but at the cost of a large number of parameters, which greatly limits their potential value in practical applications with limited computing resources, or have a small number of parameters, but at the cost of poor performance. Therefore, how to design an efficient and practical method for removing raindrops from a single image will become one of the hotspots that researchers will pay attention to in the future.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a single image raindrop removal method based on a loop iteration mechanism, which can significantly improve the performance of image raindrop removal and greatly reduce the size of its network parameters.

In order to achieve the above object, the technical scheme of the present invention is: a method for removing raindrops from a single image based on a loop iteration mechanism, comprising the following steps:

Step A, preprocessing the training image pair of the original attached raindrop degraded image and the clean image to obtain an image block dataset composed of the original attached raindrop degraded image and the training image pair of the clean image;

Step B. Based on the idea of "divide and conquer", design a convolutional neural network for raindrop removal from a single image by using the motivation of continuous iterative rain removal;

Step C. Design a target loss function loss for optimizing the network, using the image block data set as training data, according to the designed target loss function loss, use the back propagation method to calculate the designed single image raindrop to remove the convolutional neural network. The gradient of each parameter in the network, and use the stochastic gradient descent method to update the parameters, and finally learn the optimal parameters of the single image raindrop removal convolutional neural network;

Step D: Input the image to be tested into the designed single image raindrop removal convolutional neural network, and use the trained single image raindrop removal convolutional neural network to predict and generate a clean image after raindrop removal.

In an embodiment of the present invention, the specific implementation method of step A is as follows: the original attached raindrop degraded image and its corresponding clean image are divided into blocks in a consistent manner to obtain image blocks of W×W size, and at the same time, in order to avoid overlapping Dicing is performed every m pixels; after dicing, the W×W attached raindrop image blocks and the W×W clean image blocks correspond one-to-one.

In an embodiment of the present invention, the specific implementation steps of step B are as follows:

Step B1, designing a multi-stage raindrop removal network, which is a convolutional neural network based on a loop iteration mechanism, and is specifically composed of multiple stage sub-networks with the same network structure and shared network parameters;

Step B2, a sub-network in the design stage, which is used to extract the relevant features of raindrops to better remove raindrops;

Step B3, the context aggregation module and the attention context aggregation module in the sub-network in the design stage are used to aggregate the spatial context information lacking in its network.

In an embodiment of the present invention, the specific implementation steps of step B2 are as follows:

Step B21: The result of splicing the image block after the raindrop removal obtained by the sub-network of the previous stage and the corresponding original attached raindrop image block on the channel is used as the input of the sub-network in each stage; for the sub-network of the first stage, its The input is the result of splicing two original attached raindrop image patches on the channel;

Step B22, input the spliced result obtained in step B21 into a convolutional layer whose activation function is ReLU, convert the image to the feature map, and output the feature according to the following formula:

代表按照通道拼接特征操作，F₀代表所提取的特征图；Among them, Conv1 represents the convolutional layer whose activation function is ReLU, I _o is the original attached raindrop image block, I _t-1 represents the image block obtained in the previous stage after raindrop removal, for the first stage, I _t-1 is I _o ,

Represents the operation according to the channel splicing feature, and F ₀ represents the extracted feature map;

Step B23: Input the feature map F ₀ into a convolutional long short-term memory network module, which is composed of a forgetting gate f, an input gate i and an output gate o, and is calculated according to the following formula:

f _t =σ(W _xf *F ₀ +W _hf *H _t-1 +W _cf ⊙C _t-1 +b _f )

i _t =σ(W _xi *F ₀ +W _hi *H _t-1 +W _ci ⊙C _t-1 +b _i )

C _t =f _t ⊙C _t-1 +i _t ⊙tanh(W _xc *F ₀ +W _hc *H _t-1 +b _c )

o _t =σ(W _xo *F ₀ +W _ho *H _t-1 +W _co ⊙C _t +b _o )

F ₁ =H _t =o _t ⊙tanh(C _t )

Among them, the input of the forget gate f _t and the input gate it at time _t are the feature map F ₀ , the output H _t-1 of the convolutional long short-term memory network module at the previous time (ie t-1) and the cell at the previous time. The information state C _t-1 is composed of these three parts. The input of the output gate o _t at time t is composed of the feature map F ₀ , the output H _t-1 of the convolutional long short-term memory network module at the previous time (ie t-1), and The cell information state C _t at time t is composed of three parts; W _* and b _* are the weight parameters and bias parameters of the corresponding convolution kernels, respectively, tanh represents the tangent function, σ represents the Sigmoid function, operator * represents the convolution operation, the operation The symbol ⊙ represents the dot product operation; C _t is the cell information state at the current time t, which will be fed to the convolutional long-term and short-term memory network module at the next time, and H _t is the current convolutional long-term and short-term memory network at time t. The feature map output by the module; for the convenience of description, denote H _t as F ₁ ;

Through a multi-stage method, each moment mentioned above is transformed into each stage; for the first stage of the network, since there is no previous stage, the input of its forget gate and input gate are set to 0;

Step B24, input the output F1 _of the convolutional long short-term memory network module into the designed multiple context aggregation modules and attention context aggregation modules, including in turn a context aggregation module with an expansion rate of 2—> attention with an expansion rate of 2 Force context aggregation module -> context aggregation module with expansion rate 2 -> context aggregation module with expansion rate 4 -> attention context aggregation module with expansion rate 4 -> context aggregation module with expansion rate 4, according to the following formula calculate:

F ₂ = CAU ₄ (SECAU ₄ (CAU ₄ (CAU ₂ (SECAU ₂ (CAU 2 (CAU ₂ (F1)))))))

Among them, CAU _r (*) represents a context aggregation module with an expansion rate of r, and SECAU _r (*) represents an attention context aggregation module with an expansion rate of r;

Step B25, input the output result F2 of step _B24 into a standard residual module, and then send it into a convolutional layer whose activation function is ReLU to complete the conversion from feature map to image, and output the current channel number of 3 according to the following formula. Raindrop removal image for stage sub-network t:

It = _Conv2 (Res(F ₂ ))

Among them, Res(*) represents the standard residual module, Conv2 represents the convolutional layer whose activation function is ReLU, and It is the raindrop removal image of the sub-network _t at the current stage.

In an embodiment of the present invention, the specific implementation steps of step B3 are as follows:

Step B31, in the context aggregation module, first send the input feature F to a smooth hole convolution module, and calculate it according to the following formula:

F ₃ =Dilated _r (Sep(F))

Among them, F ₃ is the output feature of the smooth hole convolution module, F is the input of the context aggregation module, Sep(*) is the separable shared convolution layer, that is, the separable convolution based on channels and all channels share parameters, Dilated _r (*) is a hole (dilated) convolution, the receptive field is increased by the dilation rate r parameter, and the spatial context information is effectively aggregated to better extract features; the dilation rate r indicates how many 0s are between the elements in the convolution kernel. , when r=1, the hole convolution is exactly the same as the ordinary convolution, the elements in the convolution kernel are next to each other, and there is no 0; and when r>1, the elements in the convolution kernel need to be inserted between r- 1 0 to expand its receptive field; the expansion rate r described in step B24 is the expansion rate of the hole convolution here;

The only difference between the attention context aggregation module and the context aggregation module is this step, that is, the attention context aggregation module adds a channel attention module, and the subsequent steps are exactly the same. The attention context aggregation module is calculated according to the following formula:

F ₃ =SE(Dilated _r (Sep(F)))

Among them, SE(*) represents the channel attention module;

In step B32, the attention context aggregation module and the context aggregation module, the feature F3 output in step B31 is sent to the output _of a residual module formed by self-correction convolution, and calculated according to the following formula:

F ₄ =LeakyReLU(F ₃ +SCC(F ₃ ))

F ₄ is the output of the residual module composed of self-correction convolution. This module includes a self-correction convolution, LeakyReLU function and residual connection. The formula of LeakyReLU(*) is as follows:

Among them, x represents the input value of the LeakyReLU function, and a is a fixed linear coefficient;

SCC(*) is the self-correcting convolution, which is defined as follows:

First, input the output feature F3 _of step B31 into a 1×1 convolutional layer without activation function:

X ₁ , X ₂ =Conv1×1(F ₃ )

Among them, Conv _1×1 is the 1×1 convolution layer, X ₁ , X ₂ are the feature maps obtained by the 1×1 convolution layer with the number of channels reduced by half, that is, if the number of channels of F ₃ is C, then The number of channels of X ₁ and X ₂ is C/2;

Then, X ₁ and X ₂ are respectively sent to their corresponding branch operations, wherein X ₁ is sent to the branch of the self-correction operation, and is calculated according to the following formula:

T ₁ =AvgPool _r (X ₁ )

X' ₁ =Up(T ₁ *K ₂ )

Y ₁ =Y′ ₁ *K ₄

为逐元素相乘运算符，+为逐元素相加运算符，σ为sigmoid激活函数；K₂、K₃、K₄为卷积核大小都相同的卷积核；Y₁为自校正操作分支的输出结果；Among them, AvgPool _r (*) is the average pooling with stride r, Up(*) is the upsampling operation, * is the convolution operation,

is the element-wise multiplication operator, + is the element-wise addition operator, σ is the sigmoid activation function; K ₂ , K ₃ , K ₄ are convolution kernels with the same convolution kernel size; Y ₁ is the self-correction operation branch the output result;

At the same time, X ₂ is sent to the corresponding convolution branch, which is calculated according to the following formula:

Y ₂ =X ₂ *K ₁

Finally, the output results of the two branches are spliced on the channel, so that the number of channels is restored to the number of channels C of the original feature map, which is calculated according to the following formula:

为通道拼接操作，Y为自校正卷积模块的输出。

is the channel stitching operation, and Y is the output of the self-correcting convolution module.

In an embodiment of the present invention, the specific implementation steps of step C are as follows:

Step C1, using the loss function as a constraint to optimize the convolutional neural network model for raindrop removal from a single image, the specific formula is as follows:

where SSIM is the structural similarity loss function; suppose a training image pair (X _i , Y _i ) is given, where _i =1, . Image block, Y _i is the image block of its corresponding clean image, Y represents the clean image block after the raindrops predicted and generated by the network when using the training image pair (X _i , Y _i );

Step C2: Randomly divide the image block data set into several batches, each batch contains the same number of image blocks, and train and optimize the designed network until the L value calculated in step C1 converges to the threshold or the number of iterations Reach the threshold, save the trained model, and complete the network training process.

Compared with the prior art, the present invention has the following beneficial effects: the method of the present invention is first based on the idea of divide and conquer, decomposes the raindrop removal task into a cyclic and iterative process, and cyclically removes raindrops by designing a multi-stage convolutional neural network to achieve. Better image restoration results. At the same time, the latest smooth hole convolution module and self-correction convolution module are adopted to better aggregate spatial context information and successfully eliminate the artifact problem generated by the original hole convolution in the raindrop removal process. In this method, an independent convolutional neural network for image raindrop removal is designed for the problem of image raindrop removal, which can ensure the image quality after raindrop removal, and the size of its network parameters is smaller than other methods, which has high use value. .

Description of drawings

Fig. 1 is the realization flow chart of the method of the present invention.

FIG. 2 is a structural diagram of a method model for removing raindrops from a single image based on a loop iteration mechanism in an embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be described in detail below with reference to the accompanying drawings.

The invention provides a method for removing raindrops from a single image based on a loop iteration mechanism, comprising the following steps:

Step A, preprocessing the training image pair of the original attached raindrop degraded image and the clean image to obtain an image block dataset composed of the original attached raindrop degraded image and the training image pair of the clean image;

Step B. Based on the idea of "divide and conquer", design a convolutional neural network for raindrop removal from a single image by using the motivation of continuous iterative rain removal;

Step C. Design a target loss function loss for optimizing the network, take the image block data set as training data, and use the back propagation method to calculate the designed single image raindrop removal convolution neural network according to the designed target loss function loss. The gradient of each parameter in the network, and use the stochastic gradient descent method to update the parameters, and finally learn the optimal parameters of the single image raindrop removal convolutional neural network;

Step D: Input the image to be tested into the designed single image raindrop removal convolutional neural network, and use the trained single image raindrop removal convolutional neural network to predict and generate a clean image after raindrop removal.

The following is the specific implementation process of the present invention.

As shown in Figure 1, a method for removing raindrops from a single image based on a loop iteration mechanism includes the following steps:

Step A, preprocessing the training image pair of the original attached raindrop degraded image and the clean image to obtain an image block dataset composed of the original attached raindrop degraded image and the training image pair of the clean image;

Step B. Based on the idea of "divide and conquer", design a convolutional neural network for raindrop removal from a single image by using the motivation of continuous iterative rain removal;

Step C. Design a target loss function loss for optimizing the network, take the image block data set as training data, and use the back propagation method to calculate the designed single image raindrop removal convolution neural network according to the designed target loss function loss. The gradient of each parameter in the network, and use the stochastic gradient descent method to update the parameters, and finally learn the optimal parameters of the single image raindrop removal convolutional neural network;

Step D: Input the image to be tested into the designed single image raindrop removal convolutional neural network, and use the trained single image raindrop removal convolutional neural network to predict and generate a clean image after raindrop removal.

Further, the step A includes the following steps:

Step A1: Divide the original degraded image with attached raindrops and its corresponding clean image into blocks in a consistent manner to obtain image blocks of W×W size. At the same time, in order to avoid overlapping dicing, dicing is performed every m pixels. After dicing, the W×W attached raindrop image blocks correspond to the W×W clean image blocks one-to-one.

Further, the step B includes the following steps:

Step B1, designing a multi-stage raindrop removal network, which is a convolutional neural network based on a loop iteration mechanism, and is specifically composed of multiple stage sub-networks with the same network structure and shared network parameters;

Step B2, a sub-network in the design stage, which is used to extract the relevant features of raindrops to better remove raindrops;

Step B3, the context aggregation module and the attention context aggregation module in the sub-network in the design stage are used to aggregate the spatial context information lacking in its network;

Further, the step B2 includes the following steps:

Step B21 , the result of splicing the image block after the raindrop removal obtained by the sub-network in the previous stage and the corresponding original attached raindrop image block on the channel is used as the input of the sub-network in each stage. Note that for the sub-network in the first stage, its input is the result of splicing two original attached raindrop image patches on the channel;

Step B22, input the spliced result obtained in step B21 into a convolutional layer whose activation function is ReLU, convert the image to the feature map, and output the feature according to the following formula:

代表按照通道拼接特征操作，F₀代表所提取的特征图。Among them, Conv1 represents the convolutional layer whose activation function is ReLU, I _o is the original attached raindrop image block, I _t-1 represents the image block obtained in the previous stage after raindrop removal, for the first stage, I _t-1 is I _o ,

represents the operation according to the channel stitching feature, and F ₀ represents the extracted feature map.

Step B23: Input the feature map F ₀ into a convolutional long short-term memory network module, which is composed of a forgetting gate f, an input gate i and an output gate o, and is calculated according to the following formula:

f _t =σ(W _xf *F ₀ +W _hf *H _t-1 +W _cf ⊙C _t-1 +b _f )

i _t =σ(W _xi *F ₀ +W _hi *H _t-1 +W _ci ⊙C _t-1 +b _i )

C _t =f _t ⊙C _t-1 +i _t ⊙tanh(W _xc *F ₀ +W _hc *H _t-1 +b _c )

o _t =σ(W _xo *F ₀ +W _ho *H _t-1 +W _co ⊙C _t +b _o )

F ₁ =H _t =o _t ⊙tanh(C _t )

Among them, the input of the forget gate f _t and the input gate it at time _t are the feature map F ₀ , the output H _t-1 of the convolutional long short-term memory network module at the previous time (ie t-1) and the cell at the previous time. The information state C _t-1 is composed of these three parts. The input of the output gate o _t at time t is composed of the feature map F ₀ , the output H _t-1 of the convolutional long short-term memory network module at the previous time (ie t-1), and The cell information state C _t at time t is composed of three parts. W _* and b _* are the weight parameters and bias parameters of their corresponding convolution kernels, respectively, tanh represents the tangent function, σ represents the Sigmoid function, operator * represents the convolution operation, and operator ⊙ represents the dot product operation. C _t is the cell information state at the current time t, which will be fed to the convolutional long-term and short-term memory network module at the next time, and H _t represents the feature map output by the convolutional long-term and short-term memory network module at the current time t. To facilitate the description of the following method, we denote H _t as F ₁ .

Our approach transforms each moment described above into each phase in a multi-stage manner. Note: For the first stage of our network, since it has no previous stage, we set the input of its forget gate and input gate to 0.

Step B24, input the output F1 _of the convolutional long short-term memory network module into the designed multiple context aggregation modules and attention context aggregation modules, including in turn a context aggregation module with an expansion rate of 2—> attention with an expansion rate of 2 Force context aggregation module -> context aggregation module with expansion rate 2 -> context aggregation module with expansion rate 4 -> attention context aggregation module with expansion rate 4 -> context aggregation module with expansion rate 4, according to the following formula calculate:

F ₂ =CAU ₄ (SECAU ₄ (CAU ₄ (CAU ₂ (SECAU ₂ (CAU ₂ (F ₁ )))))))

where CAU _r (*) represents the context aggregation module with dilation rate r, and SECAU _r (*) represents the attention context aggregation module with dilation rate r.

Step B25, input the output result F2 of step _B24 into a standard residual module, and then send it into a convolutional layer whose activation function is ReLU to complete the conversion from feature map to image, and output the current channel number of 3 according to the following formula. Raindrop removal image for stage sub-network t:

It = _Conv2 (Res(F ₂ ))

Among them, Res(*) represents the standard residual module, Conv2 represents the convolutional layer whose activation function is ReLU, and It is the raindrop removal image of the sub-network _t at the current stage.

Further, the step B3 includes the following steps:

Step B31, in the context aggregation module, first send the input feature F to a smooth hole convolution module, and calculate it according to the following formula:

F ₃ =Dilated _r (Sep(F))

Among them, F ₃ is the output feature of the smooth hole convolution module, F is the input of the context aggregation module, Sep(*) is the separable shared convolution layer, that is, the separable convolution based on channels and all channels share parameters, Dilated _r (*) is a hole (dilated) convolution, and the receptive field is increased by the dilation rate r parameter, which effectively aggregates spatial context information to better extract features. The expansion rate r indicates how many 0s are between the elements in the convolution kernel. When r=1, the hole convolution is exactly the same as the ordinary convolution. The elements in the convolution kernel are next to each other without 0; and when r When >1, r-1 0s need to be inserted between elements in the convolution kernel to expand its receptive field. The expansion rate r described in step B24 is the expansion rate of the hole convolution here.

The only difference between the attention context aggregation module and the context aggregation module is this step, that is, the attention context aggregation module adds a channel attention module, and the subsequent steps are exactly the same. The attention context aggregation module is calculated according to the following formula:

F ₃ =SE(Dilated _r (Sep(F)))

where SE(*) represents the channel attention module.

Step B32: In the attention context aggregation module and the context aggregation module, the feature F3 output in step B31 is sent to the output _of a residual module composed of self-correction convolution, and calculated according to the following formula:

F ₄ =LeakyReLU(F ₃ +SCC(F ₃ ))

F ₄ is the output of the residual module composed of self-correction convolution. This module includes a self-correction convolution, LeakyReLU function and residual connection. The formula of LeakyReLU(*) is as follows:

Among them, x represents the input value of the LeakyReLU function, and a is a fixed linear coefficient.

SCC(*) is the self-correcting convolution, which is defined as follows:

First, input the output feature F3 _of step B31 into a 1×1 convolutional layer without activation function:

X ₁ , X ₂ =Conv _1×1 (F ₃ )

Among them, Conv _1×1 is the 1×1 convolution layer, X ₁ , X ₂ are the feature maps obtained by the 1×1 convolution layer with the number of channels reduced by half, that is, if the number of channels of F ₃ is C, then The number of channels of X ₁ and X ₂ is C/2.

Then, X ₁ and X ₂ are respectively sent to their corresponding branch operations, wherein X ₁ is sent to the branch of the self-correction operation, and is calculated according to the following formula:

T ₁ =AvgPool _r (X ₁ )

X' ₁ =Up(T ₁ *K ₂ )

Y ₁ =Y′ ₁ *K ₄

为逐元素相乘运算符，+为逐元素相加运算符，σ为sigmoid激活函数。K₂、K₃、K₄为卷积核大小都相同的卷积核。Y₁为自校正操作分支的输出结果。Among them, AvgPool _r (*) is the average pooling with stride r, Up(*) is the upsampling operation, * is the convolution operation,

is the element-wise multiplication operator, + is the element-wise addition operator, and σ is the sigmoid activation function. K ₂ , K ₃ , and K ₄ are convolution kernels with the same convolution kernel size. Y ₁ is the output result of the self-correction operation branch.

At the same time, X ₂ is sent to the corresponding convolution branch, which is calculated according to the following formula:

Y ₂ =X ₂ *K ₁

Finally, the output results of the two branches are spliced on the channel, so that the number of channels is restored to the number of channels C of the original feature map, which is calculated according to the following formula:

为通道拼接操作，Y为自校正卷积模块的输出。

is the channel stitching operation, and Y is the output of the self-correcting convolution module.

Further, described step C comprises the following steps:

Step C1, we use the common loss function as a constraint to optimize our network model, the specific formula is as follows:

Among them, SSIM is the structural similarity loss function. Suppose given a training image pair (X _i , Y _i ), where i = 1, . . . , N (the total number of samples of training data), X _i is the image patch of the input image corrupted by raindrops, and Y _i is its corresponding clean The image block of the image, Y represents the clean image block after the raindrops are removed and predicted by the network when using the training image pair (X _i , Y _i );

Step C2: Randomly divide the image block data set into several batches, each batch contains the same number of image blocks, and train and optimize the designed network until the L value calculated in step C1 converges to the threshold or the number of iterations Reach the threshold, save the trained model, and complete the network training process.

The above are the preferred embodiments of the present invention, all changes made according to the technical solutions of the present invention, when the resulting functional effects do not exceed the scope of the technical solutions of the present invention, belong to the protection scope of the present invention.

Claims (3)

1. a single image raindrop removal method based on loop iteration mechanism, is characterized in that, comprises the following steps: Step A, preprocessing the training image pair of the original attached raindrop degraded image and the clean image to obtain an image block dataset composed of the original attached raindrop degraded image and the training image pair of the clean image; Step B. Design a single image raindrop removal convolutional neural network with the motivation of continuous iterative rain removal; Step C. Design a target loss function loss for optimizing the network, take the image block data set as training data, and use the back propagation method to calculate the designed single image raindrop removal convolution neural network according to the designed target loss function loss. The gradient of each parameter in the network, and use the stochastic gradient descent method to update the parameters, and finally learn the optimal parameters of the single image raindrop removal convolutional neural network; Step D, input the image to be tested into the designed single image raindrop removal convolutional neural network, and use the trained single image raindrop removal convolutional neural network to predict and generate a clean image after raindrop removal; The specific implementation steps of the step B are as follows: Step B1, designing a multi-stage single image raindrop removal convolutional neural network, the network is a convolutional neural network based on a loop iteration mechanism, and is specifically composed of multiple stage sub-networks with the same network structure and shared network parameters; Step B2, a sub-network in the design stage, which is used to extract the relevant features of raindrops to better remove raindrops; Step B3, the context aggregation module and the attention context aggregation module in the sub-network in the design stage are used to aggregate the spatial context information lacking in its network; The specific implementation steps of the step B2 are as follows: Step B21, using the result of splicing the image block after the raindrop removal obtained by the previous stage sub-network and its corresponding original attached raindrop image block on the channel as the input of each stage sub-network; for the first stage sub-network, its The input is the result of splicing two original attached raindrop image patches on the channel; Step B22, input the spliced result obtained in step B21 into a convolutional layer whose activation function is ReLU, convert the image to the feature map, and output the feature according to the following formula:

Among them, Conv1 represents the convolutional layer whose activation function is ReLU, I _o is the original attached raindrop image block, I _t-1 represents the image block obtained in the previous stage after raindrop removal, for the first stage, I _t-1 is I _o ,

Represents the operation according to the channel splicing feature, and F ₀ represents the extracted feature map; Step B23: Input the feature map F ₀ into a convolutional long short-term memory network module, which is composed of a forgetting gate f, an input gate i and an output gate o, and is calculated according to the following formula: f _t =σ(W _xf *F ₀ +W _hf *H _t-1 +W _cf ⊙C _t-1 +b _f ) i _t =σ(W _xi *F ₀ +W _hi *H _t-1 +W _ci ⊙C _t-1 +b _i ) C _t =f _t ⊙C _t-1 +i _t ⊙tanh(W _xc *F ₀ +W _hc *H _t-1 +b _c ) o _t =σ(W _xo *F ₀ +W _ho *H _t-1 +W _c0 ⊙C _t +b ₀ ) F ₁ =H _t =o _t ⊙tanh(C _t ) Among them, the input of the forgetting gate f _t and the input gate it at time _t are composed of the feature map F ₀ , the output H _t-1 of the convolutional long short-term memory network module at the previous time, and the cell information state C _t-1 at the previous time. The input of the output gate o _t at time t consists of three parts: the feature map F ₀ , the output H _t-1 of the convolutional long short-term memory network module at the previous time, and the cell information state C _{t at time t} ; W _* and b _* are the weight parameters and bias parameters of their corresponding convolution kernels, respectively, tanh represents the tangent function, σ represents the Sigmoid function, operator * represents the convolution operation, operator ⊙ represents the dot product operation; C _t is the current t The cell information state at the time, the cell information state will be fed to the convolutional long-term and short-term memory network module at the next time, H _t represents the feature map output by the convolutional long-term and short-term memory network module at the current time t; for the convenience of description, H _t is denoted as F ₁ ; Through a multi-stage method, each moment mentioned above is transformed into each stage; for the first stage of the network, since there is no previous stage, the input of its forget gate and input gate are set to 0; Step B24: Input the output F ₁ of the convolutional long short-term memory network module into the designed multiple context aggregation modules and attention context aggregation modules, including in turn a context aggregation module with an expansion rate of 2 -> attention with an expansion rate of 2 Force context aggregation module -> context aggregation module with expansion rate 2 -> context aggregation module with expansion rate 4 -> attention context aggregation module with expansion rate 4 -> context aggregation module with expansion rate 4, according to the following formula calculate: F ₂ =CAU ₄ (SECAU ₄ (CAU ₄ (CAU ₂ (SECAU ₂ (CAU ₂ (F ₁ ))))))) Among them, CAU _r (*) represents a context aggregation module with an expansion rate of r, and SECAU _r (*) represents an attention context aggregation module with an expansion rate of r; Step B25, input the output result F2 of step _B24 into a standard residual module, and then send it into a convolutional layer whose activation function is ReLU to complete the conversion from feature map to image, and output the current channel number of 3 according to the following formula. Raindrop removal image for stage sub-network t: It = _Conv2 (Res(F ₂ )) Among them, Res(*) represents the standard residual module, Conv2 represents the convolutional layer whose activation function is ReLU, and I _t is the raindrop removal image of the sub-network t at the current stage; The specific implementation steps of the step B3 are as follows: Step B31, in the context aggregation module, first send the input feature F to a smooth hole convolution module, and calculate it according to the following formula: F ₃ =Dilated _r (Sep(F)) Among them, F3 is the output feature _of the smooth hole convolution module, F is the input of the context aggregation module, Sep(*) is the separable shared convolution layer, that is, the separable convolution based on channels and all channels share parameters, Dilated _r (*) is atrous convolution, the receptive field is increased by the parameter of the expansion rate r, and the spatial context information is effectively aggregated to better extract features; the expansion rate r indicates how many 0s are between the elements in the convolution kernel, when r =1, at this time, the hole convolution is exactly the same as the ordinary convolution. The elements in the convolution kernel are next to each other, and there is no 0; and when r>1, r-1 0s need to be inserted between the elements in the convolution kernel. to expand its receptive field; the expansion rate r described in step B24 is the expansion rate of the hole convolution here; The only difference between the attention context aggregation module and the context aggregation module is this step, that is, the attention context aggregation module adds a channel attention module, and the subsequent steps are exactly the same. The attention context aggregation module is calculated according to the following formula: F ₃ =SE(Dilated _r (Sep(F))) Among them, SE(*) represents the channel attention module; In step B32, the attention context aggregation module and the context aggregation module, the feature F3 output in step B31 is sent to the output _of a residual module formed by self-correction convolution, and calculated according to the following formula: F ₄ =LeakyReLU(F ₃ +SCC(F ₃ )) F ₄ is the output of the residual module composed of self-correction convolution. This module includes a self-correction convolution, LeakyReLU function and residual connection. The formula of LeakyReLU(*) is as follows:

Among them, x represents the input value of the LeakyReLU function, and a is a fixed linear coefficient; SCC(*) is the self-correcting convolution, which is defined as follows: First, input the output feature F3 _of step B31 into a 1×1 convolutional layer without activation function: X ₁ , X ₂ =Conv _1×1 (F ₃ ) Among them, Conv _1×1 is the 1×1 convolution layer, X ₁ , X ₂ are the feature maps obtained by the 1×1 convolution layer with the number of channels reduced by half, that is, if the number of channels of F ₃ is C, then The number of channels of X ₁ and X ₂ is C/2; Next, X ₁ and X ₂ are respectively sent to their corresponding branch operations, wherein X ₁ is sent to the branch of the self-correction operation, and is calculated according to the following formula: T ₁ =AvgPool _r (X ₁ ) X' ₁ =Up(T ₁ *K ₂ )

Y ₁ =Y′ ₁ *K ₄ Among them, AvgPool _r (*) is the average pooling with stride r, Up(*) is the upsampling operation, * is the convolution operation,

is the element-wise multiplication operator, + is the element-wise addition operator, σ is the sigmoid activation function; K ₂ , K ₃ , K ₄ are convolution kernels with the same convolution kernel size; Y ₁ is the self-correction operation branch the output result; At the same time, X ₂ is sent to the corresponding convolution branch, which is calculated according to the following formula: Y ₂ =X ₂ *K ₁ Finally, the output results of the two branches are spliced on the channel, so that the number of channels is restored to the number of channels C of the original feature map, which is calculated according to the following formula:

is the channel stitching operation, and Y is the output of the self-correcting convolution module. 2. A method for removing raindrops from a single image based on a cyclic iteration mechanism according to claim 1, wherein the specific implementation method of the step A is: the original attached raindrop degraded image and its corresponding clean image are consistent with each other. In order to avoid overlapping dicing, dicing is carried out every m pixels; after dicing, the W×W attached raindrop image block and the W×W clean image Blocks correspond one-to-one. 3. a kind of single image raindrop removal method based on loop iteration mechanism according to claim 1, is characterized in that, described step C concrete realization step is as follows: Step C1, using the loss function as a constraint to optimize the single image raindrop removal convolutional neural network model, the specific formula is as follows:

where SSIM is the structural similarity loss function; assuming a given training image pair (X _i , Y _i ), where i=1, . . . , N, N is the total number of samples of training data, and X _i is damaged by raindrop The image block of the input image, Y _i is the image block of the corresponding clean image, and Y represents the clean image block after the raindrops predicted and generated by the network are removed when the training image pair (X _i , Y _i ) is used; Step C2: Randomly divide the image block data set into several batches, each batch contains the same number of image blocks, and train and optimize the designed network until the L value calculated in step C1 converges to the threshold or the number of iterations Reach the threshold, save the trained model, and complete the network training process.

Images