CN110210485A - The image, semantic dividing method of Fusion Features is instructed based on attention mechanism - Google Patents

Abstract

The invention discloses an image semantic segmentation method based on attention mechanism to guide feature fusion, including the following steps: (10) Encoder basic network construction: use the improved ResNet-101 to generate a series of high-resolution low-semantic to low-resolution (20) Decoder feature fusion module construction: using a pyramid structure module based on three-layer convolution operations, extracting high-level semantics with strong consistency constraints, and then weighting and fusing low-level features layer by layer to obtain a preliminary Segmentation heat map; (30) Auxiliary loss function construction: add auxiliary supervision to each fusion output in the decoding stage, and then superimpose with the main supervision loss after sampling on the heat map, strengthen the layered training of the model, and obtain a semantic segmentation map. The image semantic segmentation method guided by feature fusion based on the attention mechanism of the present invention has high accuracy and clear boundary outline.

Description

Image Semantic Segmentation Method Based on Attention Mechanism Guided Feature Fusion

technical field

The invention belongs to the technical field of static image recognition, in particular to an image semantic segmentation method based on an attention mechanism guiding feature fusion with high accuracy and clear boundaries.

Background technique

Semantic segmentation, that is, pixel-level image understanding, is one of the important cornerstones in the field of computer vision and has a very wide range of application scenarios. Through fine-grained segmentation, it gives the machine the ability to separate different areas of the visual image at the pixel level. Semantic segmentation divides the pixel regions belonging to the same object in the image together, thereby expanding its application field.

Semantic segmentation combines the two problems of object classification and target positioning while performing pixel-level prediction. How to strike a balance between the two mutually constrained problems of high-level abstract object classification and low-level precise target positioning is the current semantic The core problem that segmentation methods have to face. Semantic segmentation methods can be broadly classified into two categories. The first is to generate the semantics of each object in the image by manually extracting features. This method often requires careful feature engineering methods, and then input into a classifier for pixel-level classification. The second is based on deep learning methods, which directly assign a semantic label to each pixel by building an end-to-end system combining feature extraction and classifiers.

Most of the traditional methods are machine learning methods that rely on manually extracting features and combining them with classifiers, such as the Boost method of Shotton et al., the random forest of Johnson et al., and the support vector machine of Soatto et al. These methods have made substantial progress by integrating rich information from contextual and structured prediction techniques. However, due to the limited expressive ability of manually extracted features, the performance of the image semantic segmentation system based on traditional machine learning methods is gradually saturated, unable to break through the bottleneck, and there is still a lot of room for improvement in the performance of segmentation accuracy.

In recent years, the deep learning revolution has revolutionized related fields, and many computer vision problems, including semantic segmentation, have begun to be solved using deep architectures. Based on the fully convolutional network method proposed by the deep convolutional neural network, the convolutional layer is used instead of the fully connected layer to construct a fully convolutional network and applied to semantic segmentation, which generates dense pixel-by-pixel labeling output and obtains higher segmentation accuracy. Zhao et al. proposed a pyramid scene parsing network method, using a pyramid pooling module to utilize global context information through context aggregation in different regions, and effectively producing high-quality segmentation results using global priors. Li et al. improved segmentation performance by first classifying shallow stage regions and focusing deeper stages on few difficult regions for adaptation and learning for difficult example recognition. Lin et al. propose a general multipath optimization network approach that explicitly exploits all available information during downsampling to achieve high-resolution pixel-wise predictions using long-range residual connections.

However, there are still two main problems in the existing technology in terms of semantic segmentation effect:

1. In the semantic segmentation of images based on deep fully convolutional networks, when using convolutional networks for feature extraction, the feature resolution is gradually reduced due to repeated combinations of convolution, maximum pooling, and downsampling operations, resulting in loss of context information. Semantic inconsistencies such as misrecognition of local areas of complex-looking targets in the segmentation results and misrecognition of small targets in multi-scale objects;

2. The success of convolutional networks is partly due to their inherent invariance to local transformations of images, which enhances the network's ability to learn hierarchical abstractions, which is exactly what is required for high-level vision tasks such as object classification. While semantic segmentation solves the classification problem, it also needs to face the spatial details such as the boundary contour of the positioning object in the segmentation. The simple pixel classification task often has the phenomenon that the boundary contour of the object in the segmentation result is blurred.

Contents of the invention

The purpose of the present invention is to provide an image semantic segmentation method based on attention mechanism to guide feature fusion, which has high accuracy and clear boundary outline.

The technical scheme that realizes the object of the present invention is:

An image semantic segmentation method based on an attention mechanism to guide feature fusion, comprising the following steps:

(10) Encoder basic network construction: use the improved ResNet-101 to generate a series of features ranging from high resolution and low semantics to low resolution and high semantics;

(20) Decoder feature fusion module construction: a pyramid structure module based on three-layer convolution operation is used to extract high-level semantics with strong consistency constraints, and then weighted and fused to low-level features layer by layer to obtain a preliminary segmentation heat map;

(30) Auxiliary loss function construction: add auxiliary supervision to each fusion output in the decoding stage, and then superimpose with the main supervision loss after sampling on the heat map, strengthen the layered training of the model, and obtain a semantic segmentation map.

Compared with the prior art, the present invention has the following points:

1. High accuracy: the method of the present invention integrates features of three different scales by implementing a high-level semantic information extraction module at the end similar to a pyramid structure, and additionally introduces a global pooling branch connected to the output features for subsequent processing, combining context information with Multiplying the original features after a simple convolution operation can capture strong semantic consistency features without introducing too much calculation, and reduce the probability of error in the recognition of local areas of objects;

2. The boundary outline is clear: according to the feature that the high-level features contain more semantic information and the low-level features contain more spatial detail information between adjacent features, the present invention first connects the two-level features to generate a channel attention vector, and selects it as a weight The most discriminative information in low-level features, using the strong semantic consistency constraints of high-level features to guide and refine its fusion with low-level features, capture rich context, and finally refine the segmentation boundary of objects to better fuse hierarchical features In order to restore the boundary details of objects in the segmentation map, the blurring of boundary contours is reduced.

Description of drawings

Fig. 1 is the main flowchart of the image semantic segmentation method based on the attention mechanism guiding feature fusion in the present invention.

Fig. 2 is a flowchart of the construction steps of the encoder basic network in Fig. 1 .

Fig. 3 is a flowchart of the construction steps of the decoder feature fusion module in Fig. 1 .

Figure 4 is an example of the terminal high-level semantic information extraction module.

Figure 5 is an example of the attention mechanism guiding the feature fusion module.

Detailed ways

As shown in Figure 1, the image semantic segmentation method of the present invention based on attention mechanism guiding feature fusion, comprises the following steps:

(10) Encoder basic network construction: use the improved ResNet-101 to generate a series of features ranging from high resolution and low semantics to low resolution and high semantics;

As shown in Figure 2, the (10) encoder basic network construction steps include:

(11) Re-deployment of the number of layers of building blocks: redeploy the number of building blocks owned by each stage from res-2 to res-5, and res-2 to res-5 of the original ResNet-101 {3, 4, 23, 3 } The number of building blocks is adjusted to {8, 8, 9, 8};

The purpose of the convolutional network encoder is to generate a series of features ranging from high-resolution low-semantic to low-resolution high-semantic. The base network usually uses existing convolutional neural network models, such as LeNet, AlexNet, VGG, GoogLeNet, ResNet, etc. Among them, ResNet-101 uses a large number of residual structures to solve the problem of gradient disappearance while the number of layers is deepened. Each residual structure also provides a new path for forward and backward propagation, so it has a strong expression ability. The present invention uses ResNet-101 as the encoder base network for semantic segmentation.

In the basic network, features are extracted from the end of each stage of the encoder part. For ResNet-101, there are four stages of res-2, res-3, res-4 and res-5, each of which The number of building blocks contained in the stages are {3, 4, 23, 3} respectively, and each building block consists of three convolutional layers. It can be seen that the first two encoding stages of ResNet-101 have only a small number of building blocks, so the shallow number of convolutional layers makes it impossible to extract deep semantic features, and the semantic quality of low-level features is poor. Starting from res-4, after a large number of deep convolutions, the output features have stronger semantics. The semantic quality gap between the two adjacent features extracted in the res-3 and res-4 stages is extremely large. In order to improve the semantic quality of low-level features and make them closer to supervision, a straightforward method is to redeploy the number of building blocks owned by res-2 to res-5 stages, balance the number of convolutional layers in each stage, and reduce res-3 and res-5. The semantic difference between the features output by the two stages of res-4. Among the number of building blocks in each stage of redeployment, the number of {3, 4, 23, 3} building blocks from res-2 to res-5 of the original ResNet-101 is adjusted to {8, 8, 9, 8}.

(12) Expand the receptive field: Change the traditional convolution of the res-5 stage in the basic network structure of ResNet-101 to an empty convolution with an expansion rate of 2.

The output resolution of semantic segmentation should be consistent with the input image. Although the semantic segmentation method based on the fully convolutional network can accept input images of any resolution, continuous convolution and pooling operations increase the receptive field while reducing the resolution of the feature. Although the reduced feature map can be restored to the original size of the image through upsampling, this process will inevitably cause the lost information to be irreversible, and the feature map restored by upsampling will lose its sensitivity to image details. Also, frequent upsampling operations require additional memory and time. The present invention overcomes this problem by adopting the atrous convolution method originally applied in wavelet transform analysis in the field of signal processing.

Upsample the original filter by a factor of 2 and insert zero values between filter values. Although the size of the effective filter is increased, there is no need to account for intervening zeros, i.e. holes, so the number of filter parameters and the number of operations per position remain the same. The size of the receptive field can be adaptively modified by changing the expansion rate parameter r, thereby effectively controlling the resolution of features in the convolutional network without learning additional parameters.

In the convolutional neural network, after three consecutive standard convolutions with a kernel size of 3×3, the receptive field sizes are 3×3, 5×5 and 7×7, respectively. If the kernel size of the continuous convolution operation is (2d+1)×(2d+1) and remains unchanged, the receptive field size of the nth layer is:

f _n =2dn+1 (1)

That is, the size of the receptive field under standard convolution increases linearly. Figure 2 shows dilated convolutions with a kernel size of 3×3 and expansion rates of 1, 2 and 4, and their receptive fields are 3×3, 7×7 and 15×15, respectively. Assuming that the kernel size is also (2d+1)×(2d+1) constant, the expansion rate of the nth layer in the continuous dilated convolution operation is r _n , then the size of the receptive field is:

f _n ＝f _n-1 +2dr _n (2)

Where n≥2 and f ₁ =2dr ₁ +1, recursively:

Let the expansion rate r _n =2 ^n-1 , then the size of the receptive field becomes:

f _n =2d(2 ⁿ -1)+1 (4)

Therefore, choosing an appropriate expansion rate for dilated convolution can make the receptive field grow exponentially. In the res-5 stage of the basic network structure, the hole convolution with an expansion rate of 2 is used. Because both res5a and res5c use a 1×1 filter kernel in this stage, only res5b is actually rapidly expanding the receptive field to extract dense feature.

(20) Decoder feature fusion module construction: a pyramid structure module based on three-layer convolution operation is used to extract high-level semantics with strong consistency constraints, and then weighted and fused to low-level features layer by layer to obtain a preliminary segmentation heat map;

A decoder module-based structure is used to recover image resolution, during which features from various levels are integrated to refine the final prediction. The decoder architecture mainly considers how to recover the spatial information lost due to successive pooling and downsampling operations. In the present invention, an end module is designed in the decoder architecture to extract high-level semantic information with the strongest consistency constraints, and use the attention mechanism to guide the fusion with low-level features and refine the output results.

As shown in Figure 3, the (20) decoder feature fusion module construction steps include:

(21) Extract high-level semantic information at the end: use a pyramid-like structural module based on three-layer convolution operations, use 3×3, 5×5 and 7×7 convolutions in the module, and fuse contexts of different scales, Get high-level semantics with the strongest intra-class semantic consistency;

Most of the previous models performed a series of scale-scale empty pyramid pooling or empty space pyramid modules at the end of the basic network. In the current semantic segmentation system, the pyramid structure can extract feature information at different scales and expand the receptive field at the pixel level, but this structure lacks global context priors, cannot select suitable elements by channel, and may lose important Pixel level information. For example, too frequent dilated convolutions can cause loss of local information and grid pooling is also detrimental to the local consistency of feature maps. The pyramid pooling module proposed in PSPNet often loses pixel positions in pooling operations of different scales.

The present invention uses a high-level semantic information extraction module as shown in FIG. 4 to extract high-level features with strong intra-class semantic consistency constraints from the end of the basic network.

This module fuses feature information of three different scales by implementing a pyramid-like structure. In order to better extract useful context from different scales, 3×3, 5×5 and 7×7 convolutions are used in the modules respectively. Since the high-level feature resolution is small, it will not bring too much calculation. burden. This module can more accurately combine the context features of adjacent scales by gradually fusing feature information of different scales. The output features from res-5 are multiplied by channels with the fusion features after 1×1 convolution. This module also additionally introduces a global pooling branch connected to the output features, which can further improve the performance of semantic segmentation in subsequent processing.

Benefiting from the pyramid-like structure, the terminal high-level semantic information extraction module can fuse contextual information of different scales while producing powerful semantic information for high-level features. Unlike the pyramid pooling module that connects features of different scales before the convolutional layer of the clipping channel, the terminal high-level semantic information extraction module multiplies the context information with the original features after a simple 1×1 convolution operation without introducing too much calculation.

(22) Fusion of contextual features: The channel attention vector is calculated by merging the features of the adjacent stages layer by layer, which is used as a weight to select the feature information with strong discriminative power in the low-level stage, and fused with the adjacent high-stage features to obtain Segmentation heatmap.

In the basic network, ResNet-101 consists of five stages, which generate features of corresponding scales. Different stages have different recognition capabilities, resulting in multiple consistent performances. In the low-level stage, the network encodes fine spatial information, and its small receptive field and lack of spatial context guidance make it contain only a small amount of semantic consistency. In the high-level stage, it has strong intra-class semantic consistency due to its large receptive field, but the spatial accuracy of prediction is very rough. In summary, the lower stages generate more accurate spatial predictions while the higher stages give more accurate semantic predictions. In this way, the respective advantages of the two can be combined, and the high-level semantic consistency can be used to guide and low-level feature fusion to obtain the best prediction. The present invention uses the attention mechanism shown in Figure 5 to guide feature fusion.

This design computes a channel attention vector as weights by incorporating features from adjacent stages. The high-level features provide strong consistent guidance, while the low-level stages provide information with different discriminative power in the features. The channel attention vector is used for weighting to select feature information with strong discriminative power. In the semantic segmentation architecture, the convolution operation finally outputs a score map, which provides the probability that each pixel belongs to a different class. The score in the final score map is obtained by summing all the channels in the feature map and the score in the final score map is obtained by summing all the channels in the feature map:

Where x represents the features of the network output, ω represents the convolution kernel, and D is the set of pixel positions.

In formula (6), p is the predicted probability, and N is the number of channels. As in formulas (5) and (6), the final predicted label is the category with the highest probability value. Suppose the prediction result of a class is And the real label is y, so a parameter α is introduced to change the highest probability value from becomes y, as shown in formula (7):

Among them, y is the new predicted output, and α=Sigmoid(ω,x), which is the Sigmoid output in Figure 4.

Based on the above analysis, we can see the deep meaning of the attention mechanism. Equation (5) implicitly reveals that the weights of different channels are equal. But as mentioned earlier, features at different stages have different degrees of discriminative ability, resulting in different prediction fine-grainedness. In order to obtain prediction results with fine object boundaries, discriminative features should be extracted as much as possible while weak discriminative features are suppressed. Therefore, applying the value of α in Equation (7) to the feature map x represents the feature selection of the attention mechanism. With this module, it can be refined layer by layer to output the best prediction results.

(30) Auxiliary loss function construction: add auxiliary supervision to each fusion output in the decoding stage, and then superimpose with the main supervision loss after sampling on the heat map, strengthen the layered training of the model, and obtain a semantic segmentation map.

The present invention improves the loss function of the commonly used semantic segmentation method, adopts a layer-by-layer label supervision strategy, and directly adds auxiliary supervision to the features of each fusion output in the decoding stage to improve the learning ability of each layer branch in the network model . In order to generate semantic output in the auxiliary branch, each fused feature as a high-level stage is forced to learn more semantics before going to the next step, hoping to be more helpful for the subsequent fusion. It should be noted that, like the redeployment of building block layers in the encoder stage, the layer-by-layer label supervision itself cannot improve the classification function of the convolutional network, but this measure can force the convolutional network to improve in the semantic segmentation task. Semantic quality of low-level stage features, which are more helpful to the output of the decoding stage.

When training the network, the auxiliary softmax loss of the equal-resolution annotation map is added to the end of the feature fusion module corresponding to res-2, res-3 and res-4. The final classification loss of the entire model is equivalent to the sum of the supervision of the final output and the supervision of the three auxiliary branches.

Given 3 branches and a total of T=4 supervisions in the final output, the number of feature channels output by each supervised object, that is, the number of categories in the training set is N. The spatial resolution of the up-sampled feature F ^t at the end of the t-th branch is W _t ×H _t , and its value corresponding to a specific coordinate position (w,h,n) is F ^t _w,h,n . The weighted softmax cross-entropy loss is added to the final output and the feature map supervision of each branch, and the corresponding weight is λ _t , where λ ₀ =1 is the loss weight of the final output, and the rest is the loss of auxiliary supervision. Input F ^t into the softmax function to calculate the probability that each pixel in the image belongs to a different category The specific formula of the softmax function layer is:

will predict Mapped to the real label P ^t _w,h,n , the final loss function used for training is shown in formula (9):

The layer-by-layer label supervision strategy makes the gradient optimization smoother and the model is easier to train. Each branch under supervision has a strong learning ability and can learn rich semantic features at each level. Through fusion, the accuracy of the final segmentation map does not depend on any individual branch.

A specific example is used below to verify that the method of the present invention can improve the accuracy of image semantic segmentation.

Several proposed correction modules are tested on two semantic segmentation datasets, PASCAL VOC 2012 and Cityscapes. The base network is ResNet-101 pre-trained on ImageNet. The experimental hardware platform is Core i7 processor, 3.6GHz main frequency, 48G memory, GPU is NVIDIA GTX 1080, and the code runs on the TensorFlow deep learning framework.

1. Ablation experiment

This section decomposes the proposed method step by step to verify the effectiveness of each building block. In the next experiment, we evaluate and compare the obtained data on the validation set of PASCAL VOC 2012. First, based on the original ResNet-101 as the base network, and directly upsample the output at the end, as shown in Table 1.

Table 1 The effect of random scaling and flipping the enhanced dataset

Subsequently, the basic network is extended to feature fusion based on the FCN encoding-decoding architecture, and the feature fusion strategy adopts simple summation by channels after cropping and upsampling. In order to test the effectiveness of this feature fusion, a series of feature subsets are selected to list the effect of feature fusion at each stage, and compared with the effect after re-deployment of building block layers at each stage, as shown in Table 2.

From the second column in Table 2, it can be clearly seen that fusing more hierarchical features does gradually improve the output quality of the segmentation system. However, when more low-level features are fused later, the overall performance quickly tends to saturate. The watershed lies in the res-4 stage of ResNet-101, which has a total of 23 building blocks and 69 convolutional layers, making a huge semantic gap between the low-level features output by the res-2 stage and the low-level features output by the res-3 stage , this gap makes the overall performance improvement almost zero when fusing the low-level features output by the res-3 stage, and the effect of continuing to fuse in the later stage is not significant.

Table 2 Effects of fusion features before and after redeployment of building block layers

Therefore, it is concluded that the fusion between features with huge differences is basically ineffective. The third column in the table shows the effect of feature fusion after redeployment of four building block layers. Initially, the segmentation quality of res-5 output feature upsampling is slightly inferior to that before re-deployment, but it is almost negligible, which confirms that the re-deployment of the number of building block layers does not strengthen the classification ability of the convolutional network itself. The difference from column 2 is that as more low-level features are fused later, the performance improves steadily, although the improvement pace is not stable, but it is not as fast saturated as in column 2. The re-deployment mechanism makes the number of building blocks in the original four stages of res2, res-3, res-4 and res-5 of ResNet-101 change from {3, 4, 23, 3} to {8, 8, 9, 8} , so that the gap between the features in the output features of each stage is relatively narrowed, and the feature fusion effect is better, and finally it is better than the performance before redeployment, which is 0.52 percentage points higher.

Table 3 shows the effectiveness of various components of the whole model.

Table 3 Comparison of ablation experiments on the PASCAL VOC2012 validation set

The semantic information extracted by the high-level end contains strong semantic consistency. Through strong semantic constraints, it is gradually fused to the low-level stage to obtain more detailed image semantic features, which improves the model performance by 1.1%. The attention mechanism is the most important improvement of the whole model. Different from the fusion method of simple summation by channel, the channel attention vector generated by this mechanism selects the most discriminative information in the low-level features, so as to refine the boundary of object segmentation well, and improves the model's performance by 2.06% on the basis of the foregoing. Performance, compared to other component modules, contributes the most. The final layer-by-layer label supervision refines the fused hierarchical features, making each fused feature closer to the supervision, improving the performance of the entire model by 0.43%. In addition to generating high-level semantic information, the end module also has a branch to output global pooling features. The global pooling feature is used to further constrain the output of the low-level feature fusion in the res-2 stage, which strengthens the semantic consistency of the entire model for all pixels of the target when processing images. The global pooling branch improves the performance of the model by 0.96%, which is of great value.

2. Qualitative analysis

Table 4 shows the visualization of image semantic segmentation for several comparison methods.

Table 4 Partial image segmentation effect visualization

In the third and fifth columns of the table, the FCN method shows the phenomenon of misidentification of object local regions. The objects in the original picture in the third column are three cows, two of which are relatively small in scale. The FCN basic network shows the problem of incorrect recognition of the local area of the target object. The front two feet of the large-scale cow are similar to the ground visual effect, and the appearance is slightly complicated. Although it has a certain segmentation effect, it is misclassified as the sole of the horse's foot. . For the two smaller cows, many pixel regions are misclassified, and the misclassified regions are also misidentified as horses. It is speculated that the scale of cattle in the training set is generally large, and the model cannot handle smaller similar targets well. . The present invention shows an almost perfect effect, and well copes with the problem that the FCN loses image details and causes errors in local pixel classification of ground objects. The image in the fifth column is a white horse beside the white railing. Visually, the horse's back and legs are blocked by the railing. Due to the similar colors, the FCN method does not directly recognize the part of the horse's back above the railing, and the horse's legs under the railing also show blurred results. The present invention is a little more perfect, and there is basically no problem except that a very small number of pixels have misjudgment.

In the first, second and fourth columns, the FCN method shows the ambiguity of the boundaries of the segmented objects. The input image in the first column is a sheep. In the black and white negative film, the part of the sheep’s body tone gloss is close to the ground background, which is the white with the highest brightness. In the FCN segmentation results, part of the bright-colored ground background is misidentified as a part of the goat's body, and the misidentified area is relatively scattered. The segmentation map obtained on the basis of the attention mechanism of the present invention eliminates the scattered misidentified areas. It is very clear and has an excellent constraint effect on segmentation. The same goes for the case plus monitor in the second column and the racehorse in the fourth column.

3. Quantitative evaluation

In the present invention, the quantitative analysis and comparison of the experimental results of several methods are carried out on the PASCAL VOC 2012 enhanced version and the Cityscapes data set. The test results are shown in Table 5.

Table 5 Class-by-category accuracy of the attention mechanism model on the PASCAL VOC 2012 test set

When the present invention is compared with DeepLab, the accuracy rate of about half of the categories is higher than that of DeepLab, and the accuracy rate of some categories is far higher, and the final total accuracy rate is slightly higher than that of DeepLab. When compared with the cutting-edge LRR method, the present invention has a higher accuracy rate on most categories, among which bicycle, boat, bottle, chair, potted plant, sofa, TV and other categories are 3% higher than LRR, and some Even higher by 15% to 20%, these categories are difficult to segment and easily confused categories, the method of the present invention is guided by high-level semantics to carefully integrate multiple low-level features, so when dealing with more semantic details Bicycles, chairs, potted plants and other categories have advantages in feature extraction, the segmented objects have strong semantic consistency, and there are few problems such as misidentification of local areas. For category objects with similar appearances such as cows, sheep, dogs, etc., complex semantic categories can also be distinguished.

Finally, the method of the present invention is also evaluated on the Cityscapes dataset. During training, each image is cropped to 800×800. It has been observed that for high-resolution images, large-scale cropping is useful. The performance of the model on the test set is shown in Table 6. Similar to the case of PASCAL VOC 2012, the present invention achieves the best results in the segmentation of most objects and outperforms other methods in the final score.

Table 6 Class-by-category accuracy of the attention mechanism model on the Cityscapes test set

The present invention uses a convolutional network encoder to embed semantic information of different levels into a feature map, and then uses a decoder to integrate each feature map to refine the output and generate a final segmentation result.

Encoders are pre-trained convolutional models used to extract image features, whose topmost features are highly semantic, but insufficient in reconstructing the precise details of segmentation maps due to insufficient resolution. While the features at the bottom of the encoder have high-resolution details but lack strong semantic information. The encoder redeploys the number of building blocks in each stage to balance the semantic discrepancy changes between features, and uses atrous convolution with a dilation rate of 2 in the res5b block. Powerful semantic consistency constraints are generated through the terminal high-level semantic information extraction module. In the decoding stage, low-resolution high-level features and high-resolution low-level features are fused layer by layer through the attention mechanism, and the strong semantic consistency of high-level features is used. Sexual guidance feature fusion to generate high-resolution semantic results.

Claims (3)

1. an image semantic segmentation method based on attention mechanism guidance feature fusion, is characterized in that, comprises the steps: (10) Encoder basic network construction: use the improved ResNet-101 to generate a series of features ranging from high resolution and low semantics to low resolution and high semantics; (20) Decoder feature fusion module construction: a pyramid structure module based on three-layer convolution operation is used to extract high-level semantics with strong consistency constraints, and then weighted and fused to low-level features layer by layer to obtain a preliminary segmentation heat map; (30) Auxiliary loss function construction: add auxiliary supervision to each fusion output in the decoding stage, and then superimpose with the main supervision loss after sampling on the heat map, strengthen the layered training of the model, and obtain a semantic segmentation map. 2. image semantic segmentation method according to claim 1, is characterized in that, The (10) encoder basic network construction steps include: (11) Re-deployment of the number of building block layers: redeploy the number of building blocks owned by each of res-2 to res-5, and res-2 to res-5 of the original ResNet-101 {3, 4, 23, 3} The number of building blocks is adjusted to {8, 8, 9, 8}; (12) Expand the receptive field: Change the traditional convolution of the res-5 stage in the basic network structure of ResNet-101 to an empty convolution with an expansion rate of 2. 3. image semantic segmentation method according to claim 1, is characterized in that, The (20) decoder feature fusion module construction steps include: (21) Extract high-level semantic information at the end: use a pyramid-like structural module based on three-layer convolution operations, use 3×3, 5×5 and 7×7 convolutions in the module, and fuse contexts of different scales, Get high-level semantics with the strongest intra-class semantic consistency; (22) Fusion of contextual features: The channel attention vector is calculated by merging the features of the adjacent stages layer by layer, which is used as a weight to select the feature information with strong discriminative power in the low-level stage, and fused with the adjacent high-stage features to obtain Preliminary segmentation heatmap.