CN112733756B - Remote sensing image semantic segmentation method based on W divergence countermeasure network - Google Patents

Abstract

The invention relates to a remote sensing image semantic segmentation method based on a W divergence countermeasure network, which introduces a countermeasure training mechanism on a traditional U-net network architecture to solve the problem that the traditional convolutional neural network excessively depends on single pixel precision loss and ignores context connection. According to the invention, by improving the model network structure and adding a layer of deconvolution layer, the spatial resolution of the distribution of the segmented targets is recovered, and Wassertein divergence is introduced, so that the problem of spatial discontinuity of the joint distribution of different source data is solved, and the continuity of the training gradient is ensured. The segmentation precision of the model and the stability of model training are improved. All training and debugging works in the remote sensing image semantic segmentation method designed by the invention are carried out on line, and the model can be used for fast prediction of new on-line samples after training is finished.

Description

A Semantic Segmentation Method of Remote Sensing Image Based on W-Divergence Adversarial Network

technical field

The invention relates to the field of image processing, in particular to a remote sensing image semantic segmentation method based on W-divergence confrontation network.

Background technique

Generative Adversarial Network (GAN) is a new type of deep learning model that performs well in applications such as natural image synthesis, image translation, and style transfer. The GAN framework consists of a generator network and a discriminator network, in which the generator tries to deceive the discriminator by forging data, and the discriminator needs to improve its own discrimination ability to distinguish generated fake data from real data, through confrontation training , the generator and discriminator can learn the true distribution of the sample data. Due to the good data fitting ability of GAN, many studies try to introduce the GAN method into the traditional CNN semantic segmentation network. These methods use the original CNN segmentation network as the generator of the confrontation network, set another discriminator with a convolutional structure to compare the predicted segmentation map generated by the generator with the real label, and return the probability distribution of the predicted segmentation map as a loss. The function then performs adversarial training on the generator and the discriminator. Since the labels in the semantic segmentation task are discrete, the prediction distribution map generated by the generator is often continuous. This structural difference leads to the poor performance of the original GAN model in fitting the mapping relationship between continuous segmentation results and discrete labels. Specifically, When the JS divergence used in the two distributions is orthogonal or the overlap is small, the gradient will disappear and the training will not proceed normally.

Recently, Pix2pix used the CGAN (Conditional GAN) framework + U-net network to realize the translation of urban remote sensing images to maps. Unfortunately, in terms of semantic segmentation evaluation indicators, Pix2pix introduced the GAN framework. The scores obtained are not as good as the basic U-net network.

The existing remote sensing image segmentation technology has the following deficiencies:

1. The original GAN model using JS divergence as the loss function needs to carefully balance the learning capabilities of the generator and the discriminator during the training process, so as to avoid one party suppressing the other and causing the model to collapse. At present, GAN segmentation methods such as Pix2pix still use the original GAN framework. When the generator performs complex semantic segmentation tasks, the balance conditions between the generator and the discriminator are more subtle, making it difficult to train the model stably. In addition, JS divergence often has vanishing gradients when dealing with discrete labels for semantic segmentation tasks, which makes training impossible.

2. The discriminator of Pix2pix lacks the necessary upsampling layer, which leads to the reduction of the spatial resolution of the predicted distribution map, thus losing the location information of the segmentation target. In addition, the discriminator loss uses probability score statistics to predict the authenticity of the distribution map, which will lead to data distortion to a certain extent, resulting in a decrease in segmentation accuracy.

Therefore, how to improve the segmentation accuracy of remote sensing images is an urgent problem in the field of remote sensing image processing.

Contents of the invention

Aiming at the deficiencies in the prior art, the present invention proposes a remote sensing image semantic segmentation method based on Wasserstein divergence confrontation network, said method comprising:

Step 1: Establish a remote sensing image dataset;

Step 2: Establish a remote sensing image semantic segmentation model, which includes a generation network and a discriminator network;

Step 3: training described remote sensing image semantic segmentation model, Fig. 4 is the training flowchart of remote sensing image segmentation method of the present invention, specifically comprises:

Step 31: pair the RGB remote sensing images to be segmented in the training set with the corresponding binary image labels, and sequentially input a group of RGB remote sensing images and corresponding binary image labels to the input layer of the generator network, and randomly crop them to generate The rated input size of the convolutional network and position alignment;

Specifically, the RGB remote sensing image is randomly cropped to the rated input size of the generator network, the default is 256×256, and the same operation is performed on the corresponding position of the binary image label.

Step 32: The generator convolution stack performs feature extraction on the input RGB remote sensing image to be segmented, and outputs the first predicted segmentation map after feature extraction is completed;

Step 33: Comparing the labels of the first predicted segmentation image and the binary image, and counting the L1 loss of the predicted distribution, the mathematical expression is as follows:

表示L1距离损失，y表示二值图标签，G(x)表示预测分割图，

表示一次迭代的平均损失。in,

Represents the L1 distance loss, y represents the binary image label, G(x) represents the predicted segmentation map,

Indicates the average loss for an iteration.

可能在不同模型中计算范围不同。The input unit of iteration is batch, and a batch may contain multiple pictures, so

The calculation range may be different in different models.

Step 34: stack the first predicted segmentation image generated by the generator and the corresponding RGB image to be segmented, and simultaneously stack the binary image label and the RGB remote sensing image to be segmented, and input the two sets of stacked data into the discriminant in sequence In the device network, complete the discrimination and output the Wasserstein divergence of the first predicted segmentation map and the binary map label, the mathematical expression of the Wasserstein divergence is as follows:

表示Wasserstein散度损失，x表示待分割RGB遥感图像，y表示二值图标签，G(·)表示生成器网络输出，D(·)表示判别器网络输出，k、p分别为正则项的系数和指数，属于超参数，本方法默认k＝0.001,p＝3。in,

Represents the Wasserstein divergence loss, x represents the RGB remote sensing image to be segmented, y represents the binary image label, G( ) represents the output of the generator network, D( ) represents the output of the discriminator network, k and p are the coefficients of the regularization term and index are hyperparameters. This method defaults to k=0.001 and p=3.

Step 35: Use Wasserstein divergence as the discriminator loss function, and update the discriminator weight vector once through gradient descent backpropagation;

Step 36: Invert the generator loss term in the Wasserstein divergence polynomial and then add the weighted sum of the L1 loss (the default ratio is 1:100) as the generator loss function, and update the generator weight vector once through gradient descent backpropagation. The mathematical description of the optimal generator G ^* is:

Among them, λ represents the weight parameter.

Step 37. Repeat steps 31 to 36 until all samples in the training set participate in one training session;

Step 38. Repeat steps 31 to 37 until the number of iterations reaches the upper limit;

Step 39, the model pre-training is completed, and the pre-trained model is saved;

Step 4: Test the remote sensing image semantic segmentation model.

According to a preferred embodiment, the step 1 includes:

Step 11: Evenly cut the high-resolution remote sensing image and the corresponding binary image label into sub-images that fit the size of the computing resources, and filter out the samples with unbalanced number of target and background pixels;

Step 12: Divide the cut sub-images into a training set and a verification set according to a set ratio, ensuring that the training set and the verification set do not overlap and have the same distribution.

According to a preferred embodiment, the step 2 specifically includes:

Step 21: Establish a generator network based on the U-net architecture, including an input layer and a convolution stack, and the generator network is used for generation.

Step 22: Establish a discriminator network based on the FCN architecture, including a convolution stack, and the discriminator network is used for discrimination;

Step 23: Initialize the generator network and discriminator network weight vectors.

According to a preferred embodiment, the method for testing the semantic segmentation model of remote sensing images includes:

Step 41: Input the RGB remote sensing image to be segmented in the verification set into the pre-training model generator network to obtain the second pre-segmented image;

Step 42: Input the binary image label corresponding to the RGB remote sensing image to be segmented into the generator network, and calculate the segmentation accuracy of the verification set by comparing the difference between the second pre-segmented image and the binary image label, and the difference quantification can be Select parameters such as the DICE similarity coefficient IoU intersection and union ratio to quantitatively estimate;

Step 43: If the segmentation accuracy of the verification set does not reach the set value, perform steps 2 to 4 to adjust parameters to improve segmentation accuracy.

The beneficial effects of the present invention are:

1. The present invention uses the Wasserstein distance in the form of divergence to replace the JS divergence in the traditional GAN method. On the one hand, it solves the problem of gradient disappearance caused by the JS divergence, and on the other hand, it avoids the Lipschitz constraint that the optimization of the traditional Wasserstein distance needs to meet , which solves the problem of spatial discontinuity in the joint distribution of different source data between remote sensing images and binary image labels, and ensures the stability of the training gradient.

2. The present invention adds a deconvolution layer after the discriminator downsamples the convolution stack. Unlike JS divergence, which uses multi-pixel elections to return probability scores or probability distribution maps to reflect the prediction segmentation accuracy, the Wasserstein distance adopts Real-valued calculations, so our method does not need to compress the spatial resolution of the discriminator output, so a deconvolution layer is added to restore the spatial resolution of the segmentation target distribution and reduce the loss of position information.

Description of drawings

Fig. 1 is the flow chart of remote sensing image segmentation method of the present invention;

Fig. 2 is the network structural diagram of generator of the present invention;

Fig. 3 is the network structural diagram of discriminator of the present invention;

Fig. 4 is the training flowchart of remote sensing image segmentation method of the present invention;

Fig. 5 is the effect contrast figure of an example of the present invention; With

Fig. 6 is an effect comparison diagram of another example of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are exemplary only, and are not intended to limit the scope of the present invention. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present invention.

A detailed description will be given below in conjunction with the accompanying drawings.

It mainly solves the problem of lack of context in the traditional convolutional neural network (CNN) method in the semantic segmentation of remote sensing images. High-resolution remote sensing images contain rich ground information. Compared with the semantic segmentation of natural images, the segmentation targets of remote sensing images often have greater intra-class differences and more serious inter-class interference. higher requirements. Traditional CNN methods rely too much on the classification of individual pixels and ignore the internal integrity of target instances and the relationship between classes, resulting in mis-segmentation and mis-recognition. The traditional Patch AN summarizes the pixel value of a pixel block with a diameter smaller than the segmentation target into a probability score by compressing the spatial resolution, thereby obtaining the contextual consistency within the pixel block. In practice, it is found that this approach will lead to blurred segmentation results. The present invention After changing the JS divergence to the Wasserstein divergence, it is better to use the deconvolution layer to restore the spatial resolution in the experiment.

In order to introduce higher-order context relations into the CNN network and improve the segmentation performance of the CNN network, the present invention proposes an Adversarial Network (GAN) segmentation method based on Wasserstein divergence.

FIG. 1 is a flow chart of the segmentation method of the present invention. Now, with reference to FIG. 1 , the method of the present invention will be described in detail. The segmentation method of the present invention comprises:

Step 1: Establish a remote sensing image dataset, including:

Step 11: Evenly cut the high-resolution remote sensing image and the corresponding binary image label into sub-images that fit the size of the computing resources, and filter out samples with an unbalanced number of target and background pixels; the purpose of screening is to prevent the model from distorting pixels The classification result of the points is shifted to the side with a higher proportion of pixels. The default size is 600*600.

Step 12: Divide the cut sub-images into a training set and a verification set according to a set ratio, ensuring that the training set and the verification set do not overlap and have the same distribution. Generally, the ratio of training set: validation set is 3:1.

Step 2: Establish a remote sensing image semantic segmentation model, the semantic segmentation model includes a generator network and a discriminator network, specifically including:

Step 21: Establish a generator network based on the U-net architecture, including an input layer and a convolution stack, and the generator network is used to generate a prediction segmentation map. Fig. 2 is a network structure diagram of the generator of the present invention, and the specific structure is shown in Fig. 2 . The convolution stack includes convolutional layers and corresponding deconvolutional layers that run sequentially. The dotted line between the convolutional layer and the deconvolutional layer refers to the skip-connection layer, that is, the skipping layer. Its function is to fuse the data extracted by different convolutional layers. feature. Currently, there are two ways to use jump layers: addition and stacking. The present invention connects mirror layers in a stacked manner, and its main function is to enhance spatial alignment and prevent gradient disappearance.

Step 22: Establish a discriminator network based on the FCN architecture, including a convolution stack, and the discriminator network is used to calculate the wasserstein divergence loss of the predicted segmentation map generated by the generator; Figure 3 is the network structure of the discriminator of the present invention As shown in Figure 3, the discriminator consists of multiple convolutional layers and a deconvolutional layer that run sequentially. The present invention adds a layer of deconvolution layer after the downsampling convolution stack, which is different from JS divergence which uses the probability score or probability distribution map of statistical pixel blocks to reflect the accuracy of prediction segmentation. Wasserstein distance uses the actual value of a single pixel Value calculation, so our method does not need to compress the spatial resolution of the discriminator output, so a deconvolution layer is added to restore the spatial resolution of the segmentation target distribution and reduce the loss of position information.

Step 23: Initialize the generator network and discriminator network weight vectors;

Step 3: training described remote sensing image semantic segmentation model, Fig. 4 is the training flowchart of remote sensing image segmentation method of the present invention, specifically comprises:

Step 31: pair the RGB remote sensing images to be segmented in the training set with the corresponding binary image labels, and sequentially input a group of RGB remote sensing images and corresponding binary image labels to the input layer of the generator network, and randomly crop them to generate The rated input size of the convolutional network and position alignment;

Specifically, the RGB remote sensing image is randomly cropped to the rated input size of the generator network, the default is 256×256, and the same operation is performed on the corresponding position of the binary image label.

Step 32: The generator convolution stack performs feature extraction on the input RGB remote sensing image to be segmented, and outputs the first predicted segmentation map after feature extraction is completed;

Step 33: Comparing the labels of the first predicted segmentation image and the binary image, and counting the L1 loss of the predicted distribution, the mathematical expression is as follows:

表示L1距离损失，y表示二值图标签，G(x)表示预测分割图，

表示一次迭代的平均损失。in,

Represents the L1 distance loss, y represents the binary image label, G(x) represents the predicted segmentation map,

Indicates the average loss for an iteration.

可能在不同模型中计算范围不同。The input unit of iteration is batch, and a batch may contain multiple pictures, so

The calculation range may be different in different models.

Step 34: stack the first predicted segmentation image generated by the generator and the corresponding RGB image to be segmented, and simultaneously stack the binary image label and the RGB remote sensing image to be segmented, and input the two sets of stacked data into the discriminant in sequence In the device network, complete the discrimination and output the Wasserstein divergence of the first predicted segmentation map and the binary map label, the mathematical expression of the Wasserstein divergence is as follows:

表示Wasserstein散度损失，x表示待分割RGB遥感图像，y表示二值图标签，G(·)表示生成器网络输出，D(·)表示判别器网络输出，k、p分别为正则项的系数和指数，属于超参数，本方法默认k＝0.001,p＝3。in,

Represents the Wasserstein divergence loss, x represents the RGB remote sensing image to be segmented, y represents the binary image label, G( ) represents the output of the generator network, D( ) represents the output of the discriminator network, k and p are the coefficients of the regularization term and index are hyperparameters. This method defaults to k=0.001 and p=3.

Step 35: Use Wasserstein divergence as the discriminator loss function, and update the discriminator weight vector once through gradient descent backpropagation;

Step 36: Invert the generator loss term in the Wasserstein divergence polynomial and then add the weighted sum of the L1 loss (the default ratio is 1:100) as the generator loss function, and update the generator weight vector once through gradient descent backpropagation. The mathematical description of the optimal generator G ^* is:

Among them, λ represents the weight parameter.

Step 37. Repeat steps 31 to 36 until all samples in the training set participate in one training session;

Step 38. Repeat steps 31 to 37 until the number of iterations reaches the upper limit;

Step 39, the model pre-training is completed, and the pre-trained model is saved;

Step 4: Test the remote sensing image semantic segmentation model, specifically including:

Step 41: Input the RGB remote sensing image to be segmented in the verification set into the pre-training model generator network to obtain the second pre-segmented image;

Step 42: Input the binary image label corresponding to the RGB remote sensing image to be segmented into the generator network, and calculate the segmentation accuracy of the verification set by comparing the difference between the second pre-segmented image and the binary image label, and the difference quantification can be Select parameters such as the DICE similarity coefficient IoU intersection and union ratio to quantitatively estimate;

Step 43: If the segmentation accuracy of the verification set does not reach the set value, perform steps 2 to 4 to adjust parameters to improve segmentation accuracy.

The method of the present invention also includes: when the segmentation accuracy of the verification set reaches the set value, the online application is performed, that is, the new sample to be segmented is input into the pre-trained generator network, and the predicted segmentation result is output.

The parameter adjustment work of the present invention is mainly divided into two aspects: 1. Accuracy is improved by adjusting the network architecture, replacing the loss function, and adjusting the network structure; 2. By observing the convergence of the loss curve combined with the visual effect of the pre-segmented image and the verification set Accuracy to adjust hyperparameters, normalization, standardization, activation function settings and other operations to eliminate abnormalities.

The purpose of the present invention is to improve the performance of the segmentation model by introducing an adversarial network architecture into the traditional convolutional network. In order to verify the feasibility of the method in this paper, the following comparative experiments were carried out using U-net as an example.

In this paper, two sets of experiments on the basic U-net segmentation network and U-net+WGAN-div (the method of the present invention) are set up. The parameters used in the experiment, the training set and validation set, and the number of training iterations are all the same. Compared with the basic U-net, the method in this paper has not only achieved a greater improvement in the accuracy index, but also our method has better performance in terms of visual effects. The objective evaluation index adopted by the present invention is the DICE similarity coefficient and the IoU cross-over-union ratio. The experimental results show that the DICE value of the method of the present invention is 0.844, the IoU value is 0.729, the single-pixel precision is 0.951, and the DICE value of the basic U-net segmentation result It is 0.796, the IoU value is 0.663, and the single-pixel accuracy is 0.933.

Figure 5 and Figure 6 are from left to right the original RGB remote sensing image, the basic U-net segmentation result map, the segmentation result of the method of the present invention, and the real binary image label. It is obvious from Figure 5 that the GAN method has better intra-class consistency in the segmentation of individual instances, while the basic U-net has missing points when the building surface is not smooth. It can be seen from Figure 6 that the basic U-net is prone to misidentification caused by interference from open areas such as parking lots when segmenting buildings, while the method of the present invention is relatively more robust.

The main purpose of the present invention is to improve the lack of contextual connection in the semantic segmentation task of the convolutional neural network represented by the traditional U-net. The traditional convolutional network converts the semantic segmentation task into a pixel-by-pixel classification task, so that the classification of a single pixel is separated from the Semantic environment, in fact, the segmented target has continuity, if a pixel is judged as the target, then its area should have a higher probability of being the target than other areas. At present, most of the improvements for semantic segmentation revolve around the problem of contextual connection, and the confrontation network is one of the less used methods. The traditional JS divergence-based adversarial network mainly improves the contextual consistency by compressing the spatial resolution of the image. For example, through three downsampling convolutional layers, a probability score can be used to represent the pixel value in an 8*8 area. The side effect is that the high-frequency part (such as the boundary contour) of the segmentation result may be blurred. In the experiment, the traditional adversarial network method is not good at solving the problem of gradient disappearance from continuous RGB images to discrete binarized labels, and the segmentation results are not good, which is one of the reasons why the adversarial network is used less. The innovation of this paper is to use the wasserstein divergence to replace the JS divergence to avoid the gradient disappearance problem of the traditional adversarial network method, and introduce a deconvolution layer, which retains more spatial resolution and improves the accuracy of segmentation.

It should be noted that the above specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the scope of the disclosure of the present invention and fall within the scope of this disclosure. within the scope of protection of the invention. Those skilled in the art should understand that the description and drawings of the present invention are illustrative rather than limiting to the claims. The protection scope of the present invention is defined by the claims and their equivalents.

Claims (3)

1. a remote sensing image semantic segmentation method based on W divergence confrontation network, it is characterized in that, described method comprises: Step 1: Establish a remote sensing image dataset; Step 11: Evenly cut the high-resolution remote sensing image and the corresponding binary image label into sub-images that fit the size of the computing resources, and filter out the samples with unbalanced number of target and background pixels; Step 12: Dividing the cut sub-images into a training set and a verification set according to a set ratio, ensuring that the training set and the verification set do not overlap and have the same distribution; Step 2: Establish a remote sensing image semantic segmentation model, the semantic segmentation model includes a generator network and a discriminator network; Step 3: training the remote sensing image semantic segmentation model, specifically including: Step 31: pair the RGB remote sensing images to be segmented in the training set with the corresponding binary image labels, and sequentially input a group of RGB remote sensing images and corresponding binary image labels to the input layer of the generator network, and randomly crop them to generate The rated input size of the convolutional network and position alignment; Step 32: The generator convolution stack performs feature extraction on the input RGB remote sensing image to be segmented, and outputs the first predicted segmentation map after feature extraction is completed; Step 33: Comparing the labels of the first predicted segmentation image and the binary image, and counting the L1 loss of the predicted distribution, the mathematical expression is as follows:

in,

Represents the L1 distance loss, y represents the binary image label, G(x) represents the predicted segmentation map,

Indicates the average loss of an iteration; Step 34: stack the first predicted segmentation map generated by the generator and the corresponding RGB remote sensing image to be segmented, and simultaneously stack the binary image label and the RGB remote sensing image to be segmented, and input two sets of stacked data in sequence In the discriminator network, complete the discrimination and output the Wasserstein divergence of the first predicted segmentation map and the binary map label, the mathematical expression of the Wasserstein divergence is as follows:

in,

Represents the Wasserstein divergence loss, x represents the RGB remote sensing image to be segmented, y represents the binary image label, G( ) represents the output of the generator network, D( ) represents the output of the discriminator network, k and p are the coefficients of the regularization term and index; Step 35: Use Wasserstein divergence as the discriminator loss function, and update the discriminator weight vector once through gradient descent backpropagation; Step 36: Invert the generator loss term in the Wasserstein divergence polynomial and then add the weighted sum of the L1 loss as the generator loss function, and update the generator weight vector once through gradient descent backpropagation, the optimal generator G ^* The mathematical description is:

Among them, λ represents the weight parameter; Step 37: Repeat steps 31 to 36 until all samples in the training set participate in one training session; Step 38: Repeat steps 31 to 37 until the number of iterations reaches the upper limit; Step 39: Model pre-training is completed, save the pre-trained model; Step 4: Test the remote sensing image semantic segmentation model. 2. The remote sensing image semantic segmentation method according to claim 1, wherein said step 2 specifically comprises: Step 21: set up a generator network based on U-net architecture, including an input layer and a convolution stack, and the generator network is used to generate the first prediction segmentation map; Step 22: Establish a discriminator network based on the FCN architecture, including a convolution stack, and the discriminator network is used to calculate the wasserstein divergence loss of the predicted segmentation map generated by the generator; Step 23: Initialize the generator network and discriminator network weight vectors. 3. the remote sensing image semantic segmentation method as claimed in claim 1, is characterized in that, the method for testing described remote sensing image semantic segmentation model comprises: Step 41: Input the RGB remote sensing image to be segmented in the verification set into the pre-training model generator network to obtain the second pre-segmented image; Step 42: Input the binary image label corresponding to the RGB remote sensing image to be segmented into the generator network, calculate the segmentation accuracy of the verification set by comparing the difference between the second pre-segmented image and the binary image label, and select the difference quantification DICE similarity coefficient and IoU intersection and ratio parameters are used to quantitatively estimate; Step 43: If the segmentation accuracy of the verification set does not reach the set value, perform steps 2 to 4 to adjust parameters to improve segmentation accuracy.

Images