CN113505792B - Multi-scale semantic segmentation method and model for unbalanced remote sensing image - Google Patents

Abstract

The invention discloses a multi-scale semantic segmentation method and a multi-scale semantic segmentation model for an unbalanced remote sensing image, wherein the multi-scale semantic segmentation model for the unbalanced remote sensing image adopts a multi-level semantic segmentation network which can learn fine-grained local features, retain small-class information, learn whole global context semantic features and retain large-scale information; the whole network architecture is divided into three layers, each layer adopts different network structures to extract features of different scales, outputs segmented images of different resolutions, and fuses the features after images are fused in the same layer by adopting a Bayesian fusion method, so that the fusion of multi-scale segmented image information is realized, and the complementation of missing information is realized; the multi-scale semantic segmentation method for the unbalanced remote sensing image adopts an optimization algorithm which can enable pixels of different categories to be more separated and pixels of the same category to be more aggregated, so that the semantic segmentation network model can realize uniform segmentation on category unbalanced data.

Description

Multi-scale semantic segmentation method and model for unbalanced remote sensing images

Technical field

The invention belongs to the technical field of remote sensing image processing, and particularly relates to a multi-scale semantic segmentation method and model for unbalanced remote sensing images.

Background technique

With the development of earth observation technology and the advancement of image acquisition technology, remote sensing images provide massive research data for earth observation and discovery. Content analysis of remote sensing images through image processing and artificial intelligence technology is an effective method to fully mine remote sensing data. The main means include scene classification, target recognition, semantic segmentation, etc. Among them, semantic segmentation is one of the important technologies for content analysis of remote sensing images. It segments the targets and regions contained in the image by inferring the semantic categories of individual pixels in the image.

The current method used for image semantic segmentation is a semantic segmentation method based on deep learning. Classic deep learning semantic segmentation networks include fully convolutional networks (FCN), SegNet, U-Net, etc. By using all convolutional layers, FCN realizes an end-to-end segmentation network for the first time, which can accept input images of any size and output segmented images of the same size. However, the contextual information and the correlation between pixels are not comprehensively combined, and the segmentation accuracy is insufficient. The U-Net network realizes the cascade function in the channel dimension and can be suitable for segmentation of multi-scale and large-size images. However, U-Net has relatively high requirements on the computing power of the device, and the computing speed is relatively slow. The SegNet network can improve memory utilization and model segmentation efficiency by using the maximum pooling index saved in the encoding stage during the decoding stage. However, when performing pooling operations on low-resolution feature maps, the information of adjacent pixels will be ignored, resulting in a loss of accuracy. Pyramid Scene Parsing Network (PSPNet) can fully utilize global context information by learning multi-level features using pyramid pooling module. But the entire scene information is not fully utilized.

Due to the significant differences in resolution, spatial structure, and semantics between remote sensing images and ordinary images, it is difficult for traditional methods and ordinary neural network methods to achieve efficient segmentation based on the characteristics of remote sensing images. Semantic segmentation of remote sensing images still faces the following challenges:

First, the distribution of semantic categories processed by existing natural image segmentation methods is relatively balanced, and the phenomenon that a certain category accounts for a larger proportion of the image in remote sensing images is not considered. Due to differences in the distribution of real physical entities on the earth's surface, the foreground and background of remote sensing images are uneven, and the scales of objects of different categories vary greatly. Secondly, deep learning segmentation models suitable for natural images are insensitive to scale changes of objects, and directly used for semantic segmentation of remote sensing images will lose pixel accuracy. Compared with categories such as land and lakes, the volume of categories such as vehicles in remote sensing images is even negligible, and there are large scale changes between objects. Therefore, previous semantic segmentation methods are not suitable for direct application to remote sensing images, and corresponding segmentation algorithms need to be designed based on the characteristics of remote sensing images.

Due to the diversity of remote sensing image acquisition and the particularity of the data itself that is different from natural images, its semantic segmentation cannot solve the problem in the previous single way. Remote sensing images have the problem of multi-scale changes in objects. Large-scale objects dominate the segmentation and inhibit the learning of small-scale objects, making it difficult to identify small-scale objects. In addition, due to the high-resolution characteristics of remote sensing images, the information contained in the images is usually very dense, causing the problem of uneven distribution of image categories.

Contents of the invention

In view of the shortcomings of the existing technology, the present invention provides a multi-scale semantic segmentation method and model for unbalanced remote sensing images. The technical problems solved are: (1) The problem of large differences in object scales in remote sensing images. (2) The problem of unbalanced category distribution of remote sensing images. In response to the first problem, the present invention proposes a multi-scale semantic segmentation model for unbalanced remote sensing images, designs a multi-level semantic segmentation network, extracts features of different scales, and fuses these features at the same level to achieve missing information. Complementary, make full use of global context information, while retaining local detail information of the image, overcome the mutual influence of multi-scale objects, and improve the robustness and accuracy of remote sensing image segmentation. In response to the second problem, the present invention carries out algorithm design from two aspects: 1) constructing an inter-class loss function to maximize the class distance of samples of different categories; 2) constructing a category weight balanced distribution loss function to solve the problem of positive and negative samples of all categories Imbalance problem.

In order to solve the above technical problems, the technical solutions adopted by the present invention are described in detail as follows:

First, the present invention provides a multi-scale semantic segmentation model for unbalanced remote sensing images, using a multi-level model that can learn fine-grained local features, retain small category information, and learn the entire global contextual semantic features and retain large-scale information. Semantic segmentation network; the entire network architecture of the multi-level semantic segmentation network is divided into three layers. Each layer uses different network structures to extract features of different scales, outputs segmented images of different resolutions, and uses Bayesian methods to combine these features at the same level. This fusion method performs post-image fusion to achieve the fusion of multi-scale segmented image information and complement the missing information.

Further, the first level Level 1 of the multi-level semantic segmentation network model uses the original resolution data, the second level Level 2 uses the data after downsampling 2 times, and the third level Level 3 uses the data after 4 times downsampling. data;

The backbone network used in the multi-level semantic segmentation network model is the SegNet semantic segmentation network. The left side of the network is the encoder, which consists of 5 convolutional pooling processes. Each of the first two layers contains two convolutional layers. Each of the last three layers contains three convolutional layers;

The right side of the network is the decoder, which consists of 5 upsampling and convolution processes. The first and fourth layers on the right are one upsampling layer plus two convolutional layers, and the second and third layers are one upsampling layer plus three convolutions. The fifth layer is an upsampling layer plus two convolutional layers, and finally a Softmax layer.

Furthermore, each layer in the encoding stage corresponds to each layer in the decoding stage; each upsampling layer in the decoder corresponds to the maximum pooling layer of the encoder at the same level, and the upsampling layer of the decoder uses maximum pooling. The index retained in the encoding process is used to upsample the feature map, so that the features of image classification in the encoding stage can be reproduced to generate dense feature maps. Finally, these feature maps are restored to the same size as the original image, and then classified through the softmax layer. , which generates the final segmentation map.

Further, the three-layer network of the multi-level semantic segmentation network outputs segmentation maps O ₁ , O ₂ and O ₃ . When performing post-image fusion of the segmentation maps output by the three-layer network, the segmentation map output by any one of the layers is selected as the segmentation map. Priori O _i , i=1,2,3, calculate the likelihood of any segmentation map O _j except this segmentation map, j≠i,j={1,2,3}, so the calculation of posterior probability is :

n represents the category of the current pixel, m represents the number of categories; F _ni and B _ni respectively represent the foreground area and background area when the category is n; O _ni is used as a priori, i = 1, 2, 3, O _nj is used Calculate the likelihood, j≠i,j={1,2,3}; in each region, calculate the likelihood by comparing O _ni and O _nj in the foreground and background of each category.

Preferably, when the segmentation map output by the three-layer network is used for post-image fusion, the segmentation map output by the first two layers is first used for fusion; and then the segmentation map obtained by the fusion of the first two layers is fused with the segmentation map output by the third layer, specifically is:

First, use the segmentation map output by the first layer network as a priori, use the segmentation map output by the second layer network to calculate the likelihood, and then merge the information of the two segmentation maps based on the Bayesian formula;

Then, the two are exchanged, using the segmentation map output by the second layer as the prior, using the segmentation map output by the first layer to calculate the likelihood, and then integrating based on the Bayesian formula;

Finally, the segmentation maps of the first two network layers and the third network layer are fused in the same way to obtain the final integrated segmentation map.

Then, the present invention also provides a multi-scale semantic segmentation method for unbalanced remote sensing images, which includes the following steps:

1. Improve the semantic segmentation network architecture: build a multi-level semantic segmentation network model, each layer of the network outputs segmentation maps of different scales, and use the Bayesian fusion method to fuse multi-scale segmentation image information;

2. Equalization loss function: Use an optimization algorithm that can not only make pixels of different categories more separated, but also make pixels of the same category more aggregated. The details are as follows:

1) Construct a category weight balanced distribution loss function based on the focal loss function to solve the problem of imbalance between positive and negative samples of all categories;

2) Introduce the hinge loss function Hinge loss to construct an inter-class loss function to maximize the class distance of samples of different categories; 3) Balance loss function: Construct an overall loss function.

Further, the category weight balanced distribution loss function

Among them, p _t is the probability that the sample of category t is positive, M is the number of sample categories, t represents a certain category, γ is a hyperparameter, -log(p _t ) is the initial cross entropy loss function; λ is an adjustable hyperparameter , setting 0<λ<1, the purpose is to increase the adjustability of the classification accuracy of different samples, reduce the penalty weight of complex samples, and increase the penalty contribution of good samples.

Further, the inter-class loss function is

Hinge＝max(0,1+maxw _wrong -w _correct ) (11)

Among them, w _wrong is the number of incorrectly classified samples, w _correct is the number of correctly classified samples, and taking the maximum value of w _wrong means selecting the category with the most incorrect samples.

Further, the overall loss function is as follows

β is a hyperparameter that controls the contribution rate of the Hinge loss penalty term, β>0.

Compared with the prior art, the advantages of the present invention are:

(1) The present invention proposes a multi-level semantic segmentation network to extract features of different scales and fuse these features at the same level to achieve complementation of missing information. It can not only learn fine-grained local features and retain small category information, but also It can learn the semantic features of the entire global context, make full use of the global context information, and retain large-scale information; on the premise of retaining local detail information of the image, it can overcome the mutual influence of multi-scale objects and improve the robustness and accuracy of remote sensing image segmentation. accuracy.

(2) The present invention also provides a Bayesian-based multi-scale post-fusion semantic segmentation method. In view of the scale-dependent characteristics of remote sensing images, by studying the multi-scale post-fusion method, different scale results are modeled as prior and similarity respectively. However, Bayes' principle is used to make optimal decisions, and it is verified that this method can improve the accuracy of segmentation.

The Bayesian fusion method of the present invention can better identify the semantic information of objects. The outline of the entire object and the distribution of categories are relatively clearer. At the same time, the boundaries of different categories of objects are more obvious, and this method also improves the network output segmentation map. performance.

(3) The present invention designs an equalization loss function for semantic segmentation of unbalanced remote sensing images. In view of the uneven semantic distribution characteristics of remote sensing images, especially the imbalance of foreground and background caused by spatial differences, based on the idea of focusing loss, the difficulty is balanced. The loss weight of training samples reduces the weight of easy-to-learn categories and increases the weight of difficult-to-learn categories, which improves the stability of training; at the same time, hinge loss is introduced to expand the distance between classes and make the boundaries of samples of different categories more obvious, thereby improving segmentation Structural accuracy and clarity in classifying local information.

Through the equalization loss algorithm of the present invention, the semantic segmentation network model can also achieve uniform segmentation on category imbalanced data.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present invention more clearly, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. Those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is a multi-level semantic segmentation network architecture diagram of the present invention;

Figure 2 is the first layer segmentation network encoding and decoding structure of the present invention;

Figure 3 is the second layer segmentation network encoding and decoding structure of the present invention;

Figure 4 is the third layer segmentation network coding and decoding structure of the present invention;

Figure 5 is a schematic diagram of the Bayesian image fusion algorithm of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

Example 1

This embodiment designs a multi-level deep neural network model. Specifically, it proposes a multi-scale semantic segmentation model for non-equilibrium remote sensing images, which can learn fine-grained local features, retain small category information, and learn the entire global context semantics. Feature,multi-level semantic segmentation networks that preserve large-scale,information. The network architecture is shown in Figure 1. The entire network architecture of the multi-level semantic segmentation network is divided into three layers. The first level corresponds to Level 1 in Figure 1, using original resolution data; the second level corresponds to Level 2 in Figure 1, using 2 times downsampling. Data; the third level corresponds to Level 3 in Figure 1, using data after downsampling 4 times. By downsampling the remote sensing image twice, more local and global information can be obtained.

For multi-scale information, each layer uses different network structures to extract features of different scales, retain as much visual information as possible in the feature extraction process, and output segmented images of different resolutions; then, these features are used at the same level using Bayern The Yeasian fusion method performs post-image fusion to achieve the fusion of multi-scale segmented image information, complement the missing information, and solve the precise segmentation of remote sensing images.

The advantage of the multi-level network model for segmentation of the present invention and other classic deep neural network segmentation models is that the multi-level network can not only retain good local detail information, but also better retain global semantic information.

1. The multi-level network architecture of this embodiment is introduced in detail below:

The backbone network used in this embodiment is the SegNet semantic segmentation network, as shown in Figures 2, 3, and 4, which correspond to Level 1, Level 2, and Level 3 in Figure 1 respectively. The left side of the network is the encoder, which consists of 5 convolutional pooling processes. Each of the first two layers contains two convolutional layers, and each of the last three layers contains three convolutional layers. The features of the image are extracted through the convolutional layer, and then the pooling layer is used to reduce the size of the feature map and increase the receptive field. The pooling layer uses maximum pooling to achieve spatial invariance in small space movements, but it will cause loss of positioning accuracy and loss of spatial details.

The right side of the network is the decoder, which consists of 5 upsampling and convolution processes. The first and fourth layers on the right are one upsampling layer plus two convolutional layers, and the second and third layers are one upsampling layer plus three convolutions. The fifth layer is an upsampling layer plus two convolutional layers, and finally a Softmax layer.

Each layer in the encoding stage corresponds to each layer in the decoding stage, similar to the U-shaped structure of the U-net network. Each upsampling layer in the decoder corresponds to the max pooling layer of the encoder at the same level. The upsampling layer of the decoder uses the index retained by the max pooling process to upsample the feature map. The features of the image classification in the encoding stage are reproduced through the upsampling layer to generate dense feature maps. Finally, these feature maps are restored to the same size as the original image, and then classified through the softmax layer, which generates the final segmentation map. SegNet has fewer training parameters and takes up less computing memory. It can also ensure the accuracy of segmentation and is suitable for semantic segmentation of high-resolution remote sensing images.

2. The image post-fusion method of this embodiment is introduced in detail below:

The multi-scale network will output segmented images of different resolutions. Due to different resolutions, the segmentation effects are different. This patent proposes a multi-scale post-fusion semantic segmentation method based on Bayesian principle. In the saliency detection task, the posterior probability is calculated by integrating the saliency mapping:

Both S ₁ and S ₂ are saliency maps, one of which is used as the prior probability Si ( _i ={1,2}) and the other S _j (j≠i,j={1,2}) for calculation Likelihood rate; F _i and B _i represent the foreground and background areas respectively, and the likelihood rate in each area is calculated by the following formula:

in Represents the number of pixels in the foreground,/> is that its color feature falls into the foreground containing feature S _j /> The number of pixels in ;/> Represents the number of pixels in the background, /> Its color feature falls into the background bin containing feature S _j /> number of pixels in .

The three-layer network of the multi-level semantic segmentation network of the present invention outputs segmentation maps O ₁ , O ₂ and O ₃ . When performing image post-fusion on the segmentation maps output by the three-layer network, the segmentation map output by any one of the layers is selected as a priori O _i (i=1,2,3), any segmentation map O _j (j≠i,j={1,2,3}) except this segmentation map calculates the likelihood, so the posterior probability is calculated as :

Among them, n represents the category of the current pixel, m represents the number of categories; F _ni and B _ni respectively represent the foreground area and background area when the category is n; O _ni (i=1,2,3) is used as a priori, O _nj (j≠i,j={1,2,3}) is used to calculate the likelihood; in each region, the likelihood is calculated by comparing the foreground and background of O _ni and O _nj in each category:

Represents the number of foreground pixels of the nth category,/> is the number of pixels of color features in the foreground area containing feature O _nj (z). Use O _nj as the prior to calculate the posterior probability.

As a preferred embodiment, when the segmentation map output by the three-layer network is post-image fused, the segmentation map output by the first two layers is first used for fusion; then the segmentation map obtained by the fusion of the first two layers is combined with the segmentation map output by the third layer. Fusion is performed as shown in (6) and (7) below.

O _n4 (z)＝O _B (O _n1 (z),O _n2 (z))＝p(F _n1 |O _n2 (z))+p(F _n2 |O _n1 (z)) (6)

O(z)＝O _B (O _n3 (z),O _n4 (z))＝p(F _n3 |O _n4 (z))+p(F _n4 |O _n3 (z)) (7)

As shown in Figure 5, the details are:

First, use the segmentation map output by the first layer network as a priori, use the segmentation map output by the second layer network to calculate the likelihood, and then merge the information of the two segmentation maps based on the Bayesian formula;

Then, the two are exchanged, using the segmentation map output by the second layer as the prior, using the segmentation map output by the first layer to calculate the likelihood, and then integrating based on the Bayesian formula;

Finally, the segmentation maps of the first two network layers and the third network layer are fused in the same way to obtain the final integrated segmentation map.

Based on Bayesian fusion, it is repeatedly forced to use different output segmentation maps as a priori, which can fuse the effective information of segmentation maps of different resolutions and improve the accuracy of image segmentation.

Example 2

This embodiment provides a multi-scale semantic segmentation method for unbalanced remote sensing images, including the following steps:

1. Improved semantic segmentation network architecture

A multi-level semantic segmentation network model is constructed. Each layer of the network outputs segmentation maps of different scales, and the Bayesian fusion method is used to fuse multi-scale segmentation image information.

The semantic segmentation network can use a classic semantic segmentation network. As a preferred embodiment, the multi-level semantic segmentation network model can directly use the model described in Embodiment 1. For details, please refer to the records of Embodiment 1, here No longer.

2. Equalization loss function

Use an optimization algorithm that can both make pixels of different categories more separated and make pixels of the same category more aggregated, as follows:

1) Based on the focal loss function, a category weight balanced distribution loss function is constructed to solve the problem of imbalance between positive and negative samples of all categories.

2) Introduce the hinge loss function Hinge loss to construct an inter-class loss function to maximize the class distance of samples of different categories.

3) Equilibrium loss function: Construct an overall loss function.

The following are introduced respectively:

1. Balanced distribution of category weights

In solving multi-classification problems, the sample categories of the data set are unevenly distributed, the number of negative samples is too large, and most samples are highly differentiated, which often leads to ineffective learning during the training process. This problem is especially obvious in the segmentation problem of remote sensing images. The purpose of using the Focal loss loss function is mainly to address the extreme imbalance problem between the background and foreground of the target detection scenario. It is improved by adding a modulation factor on the basis of cross-entropy loss (you can directly refer to the existing technology). The specific formula is as follows:

In this formula, p _t is the probability that the sample of category t is positive, M is the number of sample categories, t represents a certain category, -log(p _t ) is the initial cross-entropy loss function; γ≥0 is an adjustable hyperparameter , (1-p _t ) ^γ is the modulation factor. For easy-to-learn samples, the value of p _t is close to 1, and the modulation factor tends to zero. However, for difficult samples and misclassified samples p _t have a small value, the value of the modulation factor will increase accordingly to balance the ineffective training problem caused by sample problems. This advantage can effectively solve the problem of training failure caused by category imbalance in remote sensing images. Therefore, for the multi-classification problem of remote sensing image segmentation, the inter-class loss function formula is adjusted as follows:

Among them, λ is an adjustable hyperparameter, and is set to 0<λ<1. The purpose is to increase the adjustability of the classification accuracy of different samples and further reduce the penalty term in complex sample training based on the original formula (8). The weight of the penalty term in simple sample classification is increased to achieve balanced consistency of samples between classes. For example, when p _t is higher, it means that the confidence in this type of sample is higher. If λ=1 is set, the penalty weight of the alignment will be smaller. Its contribution to training will be reduced. Similarly, if p _t is small, it means that the classification of the judgment sample is difficult and the sample is a complex sample. When λ is 1, the training weight is larger and its contribution to training will be higher. Set λ so that 0<λ <1, this will reduce the penalty weight of complex samples, increase the penalty contribution of good samples, and thereby improve the classification accuracy of benign samples. Therefore, the effective setting of λ can directly find a benign balance between complex samples and easy samples, and then Improve overall sample classification accuracy.

2. Inter-class balance

Since the difference between adjacent samples in remote sensing images is small, how to expand the difference between samples between classes is also a problem that needs to be solved. This embodiment introduces the Hinge loss (HL) loss function. HL is usually used in support vector machines (SVM). In the maximum margin classification task, the intra-class distance is reduced and the inter-class distance is increased to achieve the maximum boundary. For binary classification, the formula is as follows:

Hinge＝max(0,1-y*y _pre ) (10)

In the above formula, y is the label of a real sample, and its value can only be -1 or 1. y _pre is the predicted value. When the absolute value of this predicted value is greater than or equal to 1, the distance between the sample and the dividing line is greater than 1. In this case, there will be no reward, because the probability of correctly classifying the sample is quite high. Its multi-classification form is as follows:

Hinge＝max(0,1+maxw _wrong -w _correct )

(11)

Among them, w _wrong is the number of incorrectly classified samples, and w _correct is the number of correctly classified samples. Taking the maximum value of w _wrong means selecting the category with the most wrong samples. When the number of incorrectly classified samples is large, formula (11) will give a larger penalty term to the training data, prompting the training to continue. Only when there are less incorrect training data and more correct training samples, the penalty term will be Automatically reduces, thus speeding up the training process to end as quickly as possible. This can promote the consistency of samples within a class, expand the interval of samples between classes, and improve the accuracy of classification.

3. Balance algorithm

Finally, in order to solve the problem of category imbalance, while increasing the sample interval of different categories and improving the accuracy of segmentation, the overall loss function of the present invention is as follows:

Among them, β is the hyperparameter that controls the contribution rate of the Hinge loss penalty term, β>0. Finally, the present invention uses the classic gradient descent method to optimize the model parameters, obtain the optimal parameters, and then test the training samples.

In summary, the present invention improves the semantic segmentation network structure and designs the loss function equalization method, and uses a spatial multi-scale parallel post-fusion framework to achieve scale difference characterization, which can not only retain good local detail information, but also better retain Global semantic information; the loss function is designed based on the uneven distribution of pixel-level samples. Based on the idea of focused loss, the loss weight of difficult and easy training samples is balanced, improving the stability of training; at the same time, hinge loss is introduced to expand the distance between classes and improve The accuracy of semantic segmentation is improved; the equalization loss function is combined with the classic semantic segmentation network to improve the accuracy of segmentation structure and the clarity of local information classification.

Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Those of ordinary skill in the art should make changes, modifications, additions or substitutions within the essential scope of the present invention. belong to the protection scope of the present invention.

Claims (5)

1. A construction method of a multi-scale semantic segmentation model for unbalanced remote sensing images is characterized by adopting a multi-level semantic segmentation network which can learn fine-grained local features, retain small-class information, learn whole global context semantic features and retain large-scale information; the whole network architecture of the multi-level semantic segmentation network is divided into three layers, each layer adopts different network structures to extract features of different scales, segmented images with different resolutions are output, and the features are subjected to graph in the same level by adopting a Bayesian fusion methodAfter-image fusion, fusion of multi-scale segmentation image information is realized, and complementation of missing information is realized; three-layer network output segmentation map O of the multi-layer semantic segmentation network ₁ ,O ₂ And O ₃ When the segmentation graphs output by the three layers of networks are fused after the images are processed, selecting any one layer of segmentation graph output as priori O _i I=1, 2,3, any one of the divided maps O except the divided map _j The likelihood, j+.i, j= {1,2,3}, is calculated, so the posterior probability is calculated as:

n represents the class of the current pixel, m represents the number of classes; f (F) _ni And B _ni Respectively representing a foreground region and a background region when the category is n; o (O) _ni As a priori i=1, 2,3, o _nj For computing likelihood, j+.i, j= {1,2,3}; in each region, by comparing O _ni And O _nj Calculating likelihood at the foreground and background of each category;

when the segmentation graphs output by the three layers of networks are fused after images, the output segmentation graphs of the first two layers are fused; then fusing the segmentation map obtained by fusing the first two layers with the segmentation map output by the third layer, specifically:

firstly, using a segmentation map output by a first layer of network as a priori, calculating likelihood by using a segmentation map output by a second layer of network, and then merging information of the two segmentation maps based on a Bayesian formula;

then exchanging the two, taking the segmentation map output by the second layer as a priori, calculating likelihood ratio by using the segmentation map output by the first layer, and integrating based on a Bayesian formula;

finally, the segmentation maps of the first two network layers and the third network layer are fused in the same way to obtain the final integrated segmentation map.

2. The method for constructing the multi-scale semantic segmentation model for the unbalanced remote sensing image according to claim 1, wherein a first Level 1 of the multi-Level semantic segmentation network model adopts data with original resolution, a second Level 2 adopts data after downsampling by 2 times, and a third Level 3 adopts data after downsampling by 4 times;

the main network adopted by the multi-level semantic segmentation network model is a SegNet semantic segmentation network, the left side of the network is an encoder, the network is composed of 5 convolution pooling processes, each of the first two layers comprises two convolution layers, and each of the last three layers comprises three convolution layers;

the right of the network is a decoder which is composed of 5 up-sampling and convolution processes, the first layer and the fourth layer on the right are an up-sampling layer and two convolution layers, the second layer and the third layer are an up-sampling layer and three convolution layers, the fifth layer is an up-sampling layer and two convolution layers, and finally a Softmax layer is added.

3. The method for constructing the multi-scale semantic segmentation model for the unbalanced remote sensing image according to claim 2, wherein each layer of the encoding stage corresponds to each layer of the decoding stage one by one; each up-sampling layer of the decoder corresponds to the maximum pooling layer of the same-level encoder, the up-sampling layer of the decoder uses the index reserved in the maximum pooling process to up-sample the feature map, so that the features of the image classification in the encoding stage are reproduced, dense feature maps are generated, and finally the feature maps are restored to the same size as the original image, and classified by the softmax layer, namely the final segmentation map is generated.

4. The multi-scale semantic segmentation method for the unbalanced remote sensing image is characterized by comprising the following steps of:

1. improving semantic segmentation network architecture: constructing a multi-level semantic segmentation network model, outputting different scale segmentation graphs by each layer of network, and fusing multi-scale segmentation image information by adopting a Bayesian fusion method;

2. equalizing the loss function: the optimization algorithm which can separate pixels of different categories and aggregate pixels of the same category is adopted, and the optimization algorithm is specifically as follows:

1) Constructing class weight balanced distribution loss functions based on Focal loss functions, and solving the problem of unbalance of positive and negative samples of all classes; the class weight balanced distribution loss function is as follows:

wherein p is _t The probability of positive for a class tsample, M is the number of sample classes, t represents a class, γ is the hyper-parameter, -log (p _t ) Is an initial cross entropy loss function; lambda is the adjusted hyper-parameter, set 0<λ<1, in order to increase the adjustability of classification accuracy of different samples, reducing the punishment weight of complex samples and increasing the punishment contribution of good samples;

2) A Hinge loss function Hinge loss is introduced to construct an inter-class loss function, so that class spacing maximization of samples of different classes is realized; the inter-class loss function is:

Hinge＝max(0,1+max w _wrong -w _correct ) (11)

wherein w is _wrong The number of samples misclassified, w _correct The number of correctly classified samples is equal to w _wrong The maximum value is taken to represent the category with the most error samples;

3) Equalizing the loss function: and constructing an overall loss function.

5. The unbalanced remote sensing image oriented multi-scale semantic segmentation method of claim 4, wherein the overall loss function is as follows:

wherein, beta is a super parameter for controlling the contribution rate of the finger loss penalty term, and beta is more than 0.