CN114170599B - Abnormal object segmentation method based on distillation comparison - Google Patents

Abstract

The present invention discloses a method for segmenting abnormal objects based on distillation comparison: on a training set without abnormalities, a semantic segmentation network is trained, and the network is used as a teacher branch after removing the semantic classification head; the parameters of the teacher branch are fixed, and a student branch with a similar structure to the teacher branch is obtained by using semantic feature distribution distillation. The outputs of the two branches are consistent in the normal class and inconsistent in the abnormal class. An abnormal test image is input, and the two branches respectively perform multi-scale feature extraction and aggregation on the image. The extracted semantic features are compared position by position to obtain an abnormal score map. The abnormal score map is bilinearly interpolated and thresholded to divide all pixels in the image into normal and abnormal categories. This method introduces a new, simple and flexible distillation comparison network to perform abnormal object segmentation. The result of the semantic classification head is not used in the reasoning stage, which greatly reduces the misjudgment of normal category pixels with semantic segmentation errors, and achieves more accurate abnormal object segmentation.

Description

A method for abnormal object segmentation based on distillation comparison

Technical Field

The present invention belongs to the technical field of computer vision, and more specifically, relates to an abnormal object segmentation method based on distillation comparison.

Background Art

In recent years, abnormal object segmentation has become a research hotspot in some safety-critical fields, such as autonomous driving and medical image analysis. Deep convolutional neural networks have made significant breakthroughs in semantic segmentation tasks, whose goal is to assign a predefined semantic category label to each pixel. However, in real-world open scenes, if a pixel belongs to an unknown category, i.e., an abnormal category, it is dangerous to classify it as a class in the predefined normal category instead of the abnormal category. The abnormal object segmentation task is to segment out the abnormal object pixels in the test image that are not in the predefined normal category of the training image.

Existing abnormal object segmentation methods first use semantic segmentation networks (such as PSPNet) to obtain semantic segmentation predictions, and then use different strategies to obtain anomaly score maps based on the semantic segmentation predictions. Methods based on uncertainty estimation perform different post-processing on semantic segmentation predictions, such as softmax function, CRF (Conditional Random Field) algorithm, semantic segmentation prediction integration, etc. to obtain anomaly score maps; methods based on image reconstruction use semantic segmentation predictions to resynthesize the original image, and compare the resynthesized image with the original image to obtain anomaly score maps. Semantic segmentation prediction is a core step in the inference stage of existing abnormal segmentation methods.

Most existing abnormal object segmentation methods rely heavily on semantic segmentation prediction, but it is observed that pixels that are misclassified in semantic segmentation are very easy to be misdetected as abnormalities. This phenomenon seriously affects the accuracy of abnormal object segmentation, but is rarely discussed in existing methods.

Summary of the invention

In response to the above defects or improvement needs of the prior art, the present invention provides a more accurate abnormal object segmentation method based on distillation comparison, which avoids the adverse effects of the introduction of semantic segmentation errors on the results, greatly reduces the misjudgment of normal category pixels caused by semantic segmentation errors, and achieves more accurate segmentation of abnormal objects in images.

To achieve the above object, the present invention provides an abnormal object segmentation method based on distillation comparison, comprising the following steps:

Step S1, using training images without abnormal objects and their pixel-level semantic labels to train a semantic segmentation network, removing the classifier of the trained semantic segmentation network and retaining only the feature extraction and aggregation parts as the teacher branch;

Step S2, fixing the parameters of the teacher branch obtained in step S1, and using semantic feature distribution distillation to train a student branch with a similar structure to the teacher branch on training images without abnormal objects. The semantic feature distribution distillation ensures that the outputs of the two branches are consistent on the normal class and inconsistent on the abnormal class.

Step S3, input a test image with an abnormal object, use the teacher branch obtained in step S1 and the student branch obtained in step S2 to perform multi-scale feature extraction and aggregation on the image, and obtain high-level semantic features of the image;

In step S4, the high-level semantic features of the teacher branch and the student branch obtained in step S3 are compared position by position, and the difference in the semantic features of the two branches is calculated using a scoring function as the anomaly score of the position to obtain an anomaly score map. The anomaly score map is bilinearly upsampled to the original image size, and finally a suitable threshold is set to divide all pixels in the image into normal and abnormal categories.

In one embodiment of the present invention, the semantic segmentation network in step S1 adopts any existing semantic segmentation network.

In one embodiment of the present invention, the classifier part is removed from the semantic segmentation network as a teacher branch. The teacher branch includes an image feature extraction module and a feature aggregation module. When training the semantic segmentation network, the downsampling operation of the high-level feature extraction module is removed. At the same time, a hole convolution is introduced at the high-level of the feature extraction module to replace the ordinary convolution, so that the output feature map size of the teacher branch is 1/8 of the input image, the number of channels is C, and C is a preset value. Finally, each layer of channels is standardized to obtain the final output features of the teacher branch.

In one embodiment of the present invention, the student branch in step S2 adopts a structure similar to that of the teacher branch in step S1.

In one embodiment of the present invention, the student branch in step S2 is specifically as follows: the student branch includes an image feature extraction module and a feature aggregation module, the downsampling operation of the high-level feature extraction module is removed, and at the same time, a hole convolution is introduced at the high-level feature extraction module to replace the ordinary convolution, so that the output feature map size of the student branch is 1/8 of the input image, which is consistent with the output feature map size of the teacher branch, the number of channels is C, C is a preset value, and is consistent with the number of output feature channels of the teacher branch, and finally each layer of channels is standardized to obtain the final output features of the student branch.

In one embodiment of the present invention, in step S2, the student branch is trained using semantic feature distribution distillation on a training image without abnormal objects, and the optimized objective function is: Where M is the batch size, C is the number of channels, i,j are the position indexes in the feature map, and c is the channel index in the feature map. It is the difference between the feature values at the corresponding positions of the corresponding channels in the output feature maps of the teacher branch and the student branch.

In one embodiment of the present invention, during the training phase of the teacher branch and the student branch, the input training images are images without abnormal objects and only contain predefined normal categories.

In one embodiment of the present invention, the scoring function in step S4 is: Among them, C is the number of channels, i,j is the position index in the feature map, and c is the channel index in the feature map. It is the difference between the feature values at the corresponding positions of the corresponding channels in the output feature maps of the teacher branch and the student branch.

In one embodiment of the present invention, PSPNet with ResNet-50 as the backbone network is used as the semantic segmentation network.

In one embodiment of the present invention, M=4, C=512.

In general, the above technical solution conceived by the present invention has the following beneficial effects compared with the prior art:

(1) The present invention proposes a novel abnormal object segmentation method based on distillation comparison. In the training phase, the method only uses normal images without abnormalities. In the test phase, the differences in high-level semantic features extracted by the two branches are compared position by position to more effectively detect abnormalities. In the test phase, the prediction of the semantic segmentation network classification head is not used as an intermediate step to avoid the introduction of semantic segmentation errors that have a negative impact on the results. This greatly reduces the misjudgment of normal category pixels caused by semantic segmentation errors, and achieves more accurate segmentation of abnormal objects in images.

(2) The present invention proposes a simple and flexible abnormal object segmentation method based on distillation comparison, in which the teacher branch can be any semantic segmentation network without the classifier part. The structure of the student branch can be similar to that of the teacher branch. The model size can be selected according to actual requirements, and the method is more universal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is an overall flow chart of the abnormal object segmentation method based on distillation comparison provided by the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

The present invention provides a method for segmenting abnormal objects based on distillation comparison, as shown in FIG1 , comprising the following steps:

Step S1, using training images without abnormal objects (1) and their pixel-level semantic labels to train a semantic segmentation network, removing the classifier of the trained semantic segmentation network and retaining only the feature extraction and aggregation parts as the teacher branch (2);

Step S2, fix the parameters of the teacher branch obtained in step S1, and train a student branch (3) with a similar structure to the teacher branch using semantic feature distribution distillation on the training image (1) without abnormal objects. The semantic feature distribution distillation ensures that the outputs of the two branches are consistent on the normal class and inconsistent on the abnormal class.

Step S3, input the test image (5) with the abnormal object, use the teacher branch obtained in step S1 and the student branch obtained in step S2 to perform multi-scale feature extraction and aggregation on the image, and obtain high-level semantic features of the image;

In step S4, the high-level semantic features of the teacher branch and the student branch obtained in step S3 are compared position by position, and the difference in the semantic features of the two branches is calculated using the scoring function (6) as the anomaly score of the position to obtain an anomaly score map. The anomaly score map is bilinearly upsampled to the original image size, and finally a suitable threshold is set to divide all pixels in the image into normal and abnormal categories.

There are three main implementation parts: 1) teacher branch; 2) student branch; 3) objective function and scoring function. Next, the steps in the present invention are described in detail.

1. Teacher branch

The embodiment of the present invention adopts PSPNet with ResNet-50 as the backbone network as the semantic segmentation network. On the training set without abnormal objects, PSPNet is trained under the conventional settings of semantic segmentation. The teacher branch (2) in the embodiment removes the classifier part of the trained PSPNet, retains the feature extraction module ResNet-50 for image feature extraction, and the feature aggregation module uses a pyramid pooling module for multi-scale aggregation of high-level semantic features. In order not to lose too much resolution to avoid losing small object information, the embodiment removes the downsampling operation of the last two layers of convolution blocks of ResNet-50. In order to obtain a larger receptive field to obtain more contextual information, the fourth layer convolution block of ResNet-50 is replaced with a dilated convolution block with a dilated rate of 2, and the fifth layer convolution block is replaced with a dilated convolution block with a dilated rate of 4. The output image feature map size is 1/8 of the input image. The output features of the last layer of ResNet-50 are pooled at four levels: 1×1, 2×2, 3×3, and 6×6. The pooled features are bilinearly upsampled to 1/8 of the input image size and concatenated with the output features of the last layer of ResNet-50. A 1×1 convolution is used to convert the number of concatenated feature channels to C. Finally, each layer of channels is standardized to obtain the final output features of the teacher branch (2). In the embodiment, C is set to 512.

2. Student Branch

The student branch (3) used in the embodiment of the present invention is similar in structure to the teacher branch (2) in the embodiment. ResNet-34 is used as a feature extraction module to extract image features, and a pyramid pooling module is used as a feature aggregation module to perform multi-scale aggregation of high-level semantic features. In order not to lose too much resolution and avoid losing small object information, the downsampling operation of the last two layers of convolution blocks of ResNet-34 is removed in the embodiment. In order to obtain a larger receptive field to obtain more context information, the fourth layer of convolution blocks of ResNet-34 is replaced with a dilated convolution block with a dilated rate of 2, and the fifth layer of convolution blocks is replaced with a dilated convolution block with a dilated rate of 4. The size of the output image feature map is 1/8 of the input image. The output features of the last layer of ResNet-34 are pooled at four levels of 1×1, 2×2, 3×3, and 6×6. The pooled features are bilinearly upsampled to 1/8 of the size of the input image and connected with the output features of the last layer of ResNet-34. The number of feature channels after connection is changed to C using 1×1 convolution. Finally, each layer of channels is standardized to obtain the final output features of the student branch. In the embodiment, C is set to 512.

3. Objective function and scoring function

In the stage of training the student branch, the embodiment of the present invention fixes the parameters of the teacher branch, and trains the student branch using semantic feature distribution distillation on a training set without abnormal objects. The optimized objective function (4) is:

Where M is the batch size, C is the number of channels, i,j are the position indexes in the feature map, and c is the channel index in the feature map. It is the difference between the feature values at the corresponding positions of the corresponding channels in the output feature graphs of the teacher branch and the student branch. In this embodiment of the present invention, M=4 and C=512 are set.

In the test phase of the embodiment of the present invention, a test image with an abnormal object is input, and the teacher branch (2) and the student branch (3) respectively extract high-level semantic features of the test image (5) with an abnormal object after aggregation and compare them position by position. The difference of the semantic features of the two branches is calculated using the scoring function (6) as the abnormal score of the position to obtain an abnormal score map. The scoring function (6) is:

Among them, C is the number of channels, i,j is the position index in the feature map, and c is the channel index in the feature map. It is the difference between the feature values at the corresponding positions of the corresponding channels in the output feature maps of the teacher branch and the student branch. In the embodiment of the present invention, C=512 is set. The semantic feature distribution distillation ensures that the outputs of the two branches are consistent in the normal class, because the training process is only performed on images that only contain predefined normal categories without abnormal objects. For untrained abnormal category pixels, the semantic feature distribution extracted by the two branches is arbitrary, so the output features of the two branches are inconsistent in the abnormal class. For normal category pixels, the abnormal score calculated by the scoring function is small, while for abnormal category pixels, the abnormal score calculated by the scoring function is large.

The present invention proposes a method for segmenting abnormal objects based on distillation comparison. The distillation comparison network includes a teacher branch and a student branch. On a training set without abnormal objects, the teacher branch is trained under the conventional setting of semantic segmentation, and the student branch is obtained by distilling the semantic feature distribution of the teacher branch. The semantic feature distribution distillation ensures that the outputs of the two branches remain consistent on the normal class and inconsistent on the abnormal class. The difference in semantic features between the two branches is used to effectively detect anomalies in the test phase. The distillation comparison network is simple and flexible. The prediction of the semantic segmentation network classification head is not used as an intermediate step in the test phase, so as to avoid the introduction of semantic segmentation errors that have a negative impact on the results, greatly reduce the misjudgment of normal category pixels with semantic segmentation errors, and achieve more accurate segmentation of abnormal objects in the image.

It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A method for segmenting abnormal objects based on distillation comparison, characterized in that it comprises the following steps: Step S1, using training images without abnormal objects and their pixel-level semantic labels to train a semantic segmentation network, removing the classifier of the trained semantic segmentation network and retaining only the feature extraction and aggregation parts as the teacher branch; Step S2, fixing the parameters of the teacher branch obtained in step S1, and using semantic feature distribution distillation to train a student branch with a similar structure to the teacher branch on training images without abnormal objects. The semantic feature distribution distillation ensures that the outputs of the two branches are consistent on the normal class and inconsistent on the abnormal class. Step S3, input a test image with an abnormal object, use the teacher branch obtained in step S1 and the student branch obtained in step S2 to perform multi-scale feature extraction and aggregation on the image, and obtain high-level semantic features of the image; In step S4, the high-level semantic features of the teacher branch and the student branch obtained in step S3 are compared position by position, and the difference in the semantic features of the two branches is calculated using a scoring function as the anomaly score of the position to obtain an anomaly score map. The anomaly score map is bilinearly upsampled to the original image size, and finally a suitable threshold is set to divide all pixels in the image into normal and abnormal categories. 2. The abnormal object segmentation method based on distillation comparison as described in claim 1 is characterized in that the semantic segmentation network in step S1 adopts any existing semantic segmentation network. 3. The abnormal object segmentation method based on distillation comparison as described in claim 1 or 2 is characterized in that the classifier part is removed as a teacher branch in the semantic segmentation network, and the teacher branch includes an image feature extraction module and a feature aggregation module. When training the semantic segmentation network, the downsampling operation of the high-level feature extraction module is removed, and at the same time, a hole convolution is introduced at the high-level of the feature extraction module to replace the ordinary convolution, so that the output feature map size of the teacher branch is 1/8 of the input image, the number of channels is C, C is a preset value, and finally each layer of channels is standardized to obtain the final output features of the teacher branch. 4. The abnormal object segmentation method based on distillation comparison as described in claim 1 or 2 is characterized in that the student branch in step S2 adopts a structure similar to that of the teacher branch in step S1. 5. The abnormal object segmentation method based on distillation comparison according to claim 4, characterized in that the student branch in step S2 is specifically: The student branch includes an image feature extraction module and a feature aggregation module. The downsampling operation of the high-level feature extraction module is removed, and atrous convolution is introduced at the high-level feature extraction module to replace ordinary convolution, so that the output feature map size of the student branch is 1/8 of the input image, which is consistent with the output feature map size of the teacher branch. The number of channels is C, which is a preset value and consistent with the number of output feature channels of the teacher branch. Finally, each layer of channels is standardized to obtain the final output features of the student branch. 6. The abnormal object segmentation method based on distillation comparison according to claim 1 or 2, characterized in that in step S2, the student branch is trained by using semantic feature distribution distillation on the training image without abnormal objects, and the optimized objective function is: Where M is the batch size, C is the number of channels, i,j are the position indexes in the feature map, and c is the channel index in the feature map. It is the difference between the feature values at the corresponding positions of the corresponding channels in the output feature maps of the teacher branch and the student branch. 7. The abnormal object segmentation method based on distillation comparison as described in claim 1 or 2 is characterized in that, in the training stage of the teacher branch and the student branch, the input training images are images without abnormal objects and only contain predefined normal categories. 8. The abnormal object segmentation method based on distillation comparison according to claim 1 or 2, characterized in that the scoring function in step S4 is: Among them, C is the number of channels, i,j is the position index in the feature map, and c is the channel index in the feature map. It is the difference between the feature values at the corresponding positions of the corresponding channels in the output feature maps of the teacher branch and the student branch. 9. The abnormal object segmentation method based on distillation comparison as described in claim 1 or 2 is characterized in that PSPNet with ResNet-50 as the backbone network is used as the semantic segmentation network. 10. The abnormal object segmentation method based on distillation comparison as described in claim 6 is characterized in that M=4 and C=512.