CN117409413A - A small-sample semantic segmentation method and system based on background information mining - Google Patents

Abstract

The present invention proposes a small-sample semantic segmentation method and system based on background information mining. The method includes: mining the potential information of the background part in the small-sample semantic segmentation task through an offline background marking algorithm network to obtain a pseudo-class data set , and then through joint training, the generalization ability and performance of the semantic segmentation model in the face of new classes are greatly improved, and the base class bias problem of the model is greatly alleviated. The small-sample semantic segmentation method based on background information mining proposed by this invention inputs the base class image data set into the offline background labeling algorithm network, and obtains prototype features from the unsupervised image segmentation algorithm subnetwork and backbone network to obtain pseudo-class data through clustering. The set is jointly trained based on the pseudo-class data set and the base class image data set to perform the segmentation task of the new class image data set, which greatly improves the generalization ability and performance of the semantic segmentation model when facing new classes, and greatly improves the generalization ability and performance of the semantic segmentation model when facing new classes. Alleviates the base class bias problem of the model.

Description

A small-sample semantic segmentation method and system based on background information mining

Technical field

The invention relates to the field of image recognition, and in particular to a small sample semantic segmentation method and system based on background information mining.

Background technique

With the rapid development of the artificial intelligence industry, deep learning is widely used in various industries, and its application is particularly prominent in computer vision. Among them, semantic segmentation algorithms are used in target detection, image classification, instance segmentation, and pose estimation. It plays an important role in a series of image recognition tasks and is one of the key research directions in the field of image recognition.

Currently existing small-sample semantic segmentation methods usually face the "base class bias problem". This is because a large amount of base class data is used in the training stage, resulting in the model's segmentation effect declining when facing new class objects in the test stage. If the image There are both new class and base class objects in the model, and it is easy to cause mis-segmentation. In the existing technology, some use the method of updating only part of the values in the model network to solve this problem, and use the singular value decomposition method to decompose the weight of the backbone network. matrix to find the singular values that must be updated, then freeze other weight parameters, update only the singular values, keep other values unchanged, and finally re-transform the updated values into the weight matrix of the model, while only updating a small number of parameters The generalization ability of the model architecture to new classes is improved, and some add an additional base class learner branch to accurately segment base class objects and correct the final prediction results.

However, among these current methods, whether they only update some parameters of the model or accurately segment base class objects through additional base class learner branches, it is still difficult to alleviate the base class bias problem of the model.

Contents of the invention

Based on this, the purpose of the present invention is to provide a small-sample semantic segmentation method and system based on background information mining. By inputting base class images into the offline background labeling algorithm network, the potential information of the background part in the small-sample semantic segmentation task is mined. , obtain the pseudo-class data set, and then conduct joint training based on the pseudo-class data set and the original data set, which greatly improves the generalization ability and performance of the semantic segmentation model when facing new classes, and greatly eases the base class of the model. Bias problem.

The small sample semantic segmentation method based on background information mining proposed by this invention includes:

Input the preset base class image data set into the offline background labeling algorithm network;

Obtain the pre-segmentation sub-region mask and high-level semantic features of the base class image through the unsupervised image segmentation algorithm sub-network and backbone network to extract the prototype features of the background area in the sub-region;

Perform clustering according to the prototype features to divide multiple different pseudo-classes, and make the pseudo-classes into a pseudo-class data set;

The semantic segmentation model is jointly trained according to the pseudo class data set and the base class image data set, so as to perform the segmentation task of the new class image data set through the trained semantic segmentation model.

In summary, according to the above-mentioned small-sample semantic segmentation method based on background information mining, by inputting the base class image into the offline background labeling algorithm network, the potential information of the background part in the small-sample semantic segmentation task is mined to obtain the pseudo-class data set, and then Through joint training based on the pseudo-class data set and the original data set, the generalization ability and performance of the semantic segmentation model in the face of new classes are greatly improved, and the base class bias problem of the model is greatly alleviated. Specifically, the preset base class image data set is input into the offline background labeling algorithm network, the foreground area and background area in the base class image are set, and then the base class is obtained through the unsupervised image segmentation algorithm subnetwork and backbone network. The pre-segmented sub-region mask and high-level semantic features of the image are used to extract the prototype features of the background area in the sub-region, clustering is performed based on the prototype features to divide multiple different pseudo-classes, and the pseudo-classes are Made into a pseudo-class data set, because the data is expanded through the offline background labeling algorithm network based on the original data, the training phase of the model will not only involve the base class information, but also the generated background pseudo-class information, so the model is very sensitive to new classes. The generalization ability is significantly improved, and the semantic segmentation model is jointly trained according to the pseudo-class data set and the base class image data set, so as to perform the segmentation task of the new class image data set through the trained semantic segmentation model, It greatly improves the generalization ability and performance of the semantic segmentation model when facing new classes, and greatly alleviates the base class bias problem of the model.

Further, the step of inputting the preset base class image data set into the offline background labeling algorithm network includes:

Select the current base class target in the preset base class image data set, set the base class area in the base class image as the foreground area, and set the non-base class area in the base class image as the background area.

Further, the step of obtaining the pre-segmented sub-region mask and high-level semantic features of the base class image through the unsupervised image segmentation algorithm sub-network and the backbone network to extract the prototype features of the background area in the sub-region includes:

Scale the preset base class images in the base class image data set to a preset image size threshold to be segmented;

Pre-segment the base class image through the unsupervised image segmentation algorithm sub-network in the offline background labeling algorithm network to obtain multiple pre-segmented sub-region masks;

The unsegmented original image of the base class image is passed through the backbone network in the offline background labeling algorithm network, and after an upsampling operation is performed on the unsegmented original image, high-level semantic features of the unsegmented original image are extracted.

Further, the step of obtaining the pre-segmented sub-region mask and high-level semantic features of the base class image through the unsupervised image segmentation algorithm sub-network and the backbone network to extract the prototype features of the background area in the sub-region also includes :

Perform a mask inversion operation through the base class target mask to remove the background area of the current base class target and suppress the foreground area;

According to the pre-segmentation sub-region mask and high-level semantic features, the Hadamard product is calculated by the following formula:

in, is the high-level semantic feature, where/> ,/> ,/> are respectively the height, width, and channel dimensions of the high-level semantic features,/> Represents broadcasting the mask along the channel, /> Represents Hadama Product,/> is the pseudo mask covered by the mask, where /> is the number of obtained pre-segmented sub-region masks,/> Temporary pseudo-mask annotation for the background area of a single image, where /> ,/> are the number of foreground base classes and background prototype features corresponding to the background mask, /> Mask the real background;

Obtain prototype features through masked average pooling.

Further, the step of clustering according to the prototype features to divide a plurality of different pseudo-classes and making the pseudo-classes into a pseudo-class data set includes:

Annotate the background areas of all base class images to obtain pseudo class prototype features and pseudo masks;

All pseudo-class prototype features are clustered through an unsupervised clustering algorithm to perform pseudo-class classification;

Classify the pre-segmented sub-regions of the background area into corresponding pseudo-classes, and label the pseudo-mask with corresponding pseudo-class labels to create a pseudo-class data set.

Further, the step of jointly training a semantic segmentation model based on the pseudo-class data set and the base class image data set to perform a segmentation task on the new class image data set through the trained semantic segmentation model includes: :

Input the base class data set and pseudo class data set into the joint training backbone network respectively;

Feature map extraction is performed through the joint training backbone network to obtain support feature maps and query feature maps;

After performing multi-scale feature extraction through the feature enrichment module, compare and integrate the support feature map and the query feature map;

The integrated feature map is convolved and passed through the classifier to obtain the final prediction result.

Further, the step of convolving the integrated feature map and passing it through the classifier to obtain the final prediction result also includes:

After obtaining the final prediction result through the classifier, calculate the loss function of the original data composed of the base class according to the following formula :

Then calculate the loss function of pseudo-class data according to the following formula :

Calculate the overall loss function L according to the following formula:

In the above formula, is the predicted query image segmentation result,/> is the spatial position of the corresponding pixel,/> is the true value mask of the query image, /> and/> The segmentation prediction mask and pseudo class mask for the pseudo class after passing through the small sample segmentation network, /> is a hyperparameter.

The present invention proposes a small-sample semantic segmentation system based on background information mining, including:

The background mining module is used to input the preset base class image data set into the offline background labeling algorithm network, and obtain the pre-segmented sub-region mask and high-level semantics of the base class image through the unsupervised image segmentation algorithm sub-network and backbone network. Features to extract prototype features of the background area in the sub-region, perform clustering according to the prototype features to divide multiple different pseudo-classes, and make the pseudo-classes into a pseudo-class data set;

A joint training module, configured to jointly train a semantic segmentation model according to the pseudo-class data set and the base class image data set, so as to perform the segmentation task of the new class image data set through the trained semantic segmentation model.

On the other hand, the present invention also provides a storage medium, including the storage medium storing one or more programs. When the program is executed, the small sample semantic segmentation method based on background information mining is implemented as described above.

Another aspect of the present invention also provides a computer device, the computer device includes a memory and a processor, wherein:

The memory is used to store computer programs;

The processor is configured to implement the above-mentioned small-sample semantic segmentation method based on background information mining when executing the computer program stored on the memory.

Description of the drawings

Figure 1 is a flow chart of the small-sample semantic segmentation method based on background information mining proposed by the first embodiment of the present invention;

Figure 2 is a flow chart of a small-sample semantic segmentation method based on background information mining proposed in the second embodiment of the present invention;

Figure 3 is a schematic structural diagram of a small-sample semantic segmentation system based on background information mining proposed in the third embodiment of the present invention;

Figure 4 is a flow chart of the unsupervised image segmentation algorithm of the small-sample semantic segmentation method based on background information mining proposed in the first embodiment of the present invention.

The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings.

Detailed ways

In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the invention are shown in the drawings. However, the invention may be embodied in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

It should be noted that when an element is referred to as being "mounted" on another element, it can be directly on the other element or intervening elements may also be present. When an element is said to be "connected" to another element, it can be directly connected to the other element or there may also be intervening elements present. The terms "vertical," "horizontal," "left," "right" and similar expressions are used herein for illustrative purposes only.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which the invention belongs. The terminology used herein in the description of the invention is for the purpose of describing specific embodiments only and is not intended to limit the invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Please refer to Figure 1, which is a flow chart of a small-sample semantic segmentation method based on background information mining proposed by the first embodiment of the present invention. This small-sample semantic segmentation method based on background information mining includes steps S01 to step S04, where :

Step S01: Input the preset base class image data set into the offline background labeling algorithm network;

It should be noted that in this embodiment, the data set used is PASCAL-5i. There are a total of 20 categories in the PASCAL-5i data set, and every 5 categories are counted as one group. In this embodiment, three groups of 15 categories are selected during training. As base class data, another set of 5 categories are used as new classes for testing.

Step S02: Obtain the pre-segmentation sub-region mask and high-level semantic features of the base class image through the unsupervised image segmentation algorithm sub-network and backbone network to extract the prototype features of the background area in the sub-region;

It should be noted that in this embodiment, the image is first scaled to the commonly used size of 473×473, and the original image is pre-segmented through an unsupervised segmentation algorithm. The upper limit of the sub-region segmented by the pre-segmentation algorithm is set to 10. This segmentation method It will generate multiple segmented sub-regions in the image. At the same time, the original image passes through the backbone network, upsamples it to the same size to extract its high-level semantic features, extracts the background area of the current category through mask inversion, suppresses the foreground area of the feature map, and then The Hadamard product Hadamard is calculated by taking the sub-region mask and image feature map obtained by pre-segmentation. The unsupervised pre-segmentation method in this embodiment uses the unsupervised image segmentation algorithm of back propagation. First, the scaled original image is obtained, using The foreground mask is inverted to obtain the background area of the image, and then a graph-based segmentation algorithm is used to pre-segment the image to replace the Mask-SLIC algorithm in the original back propagation unsupervised segmentation method to generate segmented areas and their labels. The specific process steps of the algorithm See Figure 4, where Represents the set of pixels of the input original image (a total of N pixels), through an unsupervised algorithm/> Get/> prototypes and a set of pixels corresponding to each superpixel/> , then input the neural network for t iterations, convolutional neural network/> Take an image as input to generate an image feature map/> , of which/> ,/> ,/> Represents the height, width and channel dimensions of the feature map, expressed as a set of pixels/> , according to the feature map, if the /> If the maximum value is obtained for each channel, the pixel will be marked as/> , corresponding to each pixel set/> , we find the label with the most occurrences, and record the pixels of the entire set as this label/> , and then use the Softmax loss function to calculate the model loss to make it close to the pre-segmentation result, and finally obtain the segmentation prediction of each pixel/> , this method assigns the same semantic label to small areas with the same semantic information under the pre-classification algorithm, and then uses a neural network model to classify the input image, so that the classification results output by the network are as close as possible to the pre-classification of the image segmentation algorithm. As a result, on the basis of the pre-classification results, small blocks with the same semantic information are merged to obtain the final segmentation effect. The temporary pseudo-mask annotation of the background area of a single image obtained through the unsupervised segmentation algorithm is recorded as /> , of which/> ,/> are the foreground base class and the number of background feature prototypes corresponding to the background pseudo mask respectively. By corresponding to the base class of the original data/> The corresponding real foreground mask is inverted to obtain the real background mask/> , obtain the high-level semantic features of the original image through the backbone network/> , of which/> ,/> ,/> are the width, height, and channel dimensions of the high-level semantic feature map respectively. On this basis,/> ,/> The mask is bilinearly interpolated to the feature map size and broadcast along the channel, changing the dimensionality from/> Become/> , and/> To do the Hadamard product, the formula is as follows:

in, Represents broadcasting the mask along the channel, /> Represents the Hadamard product and obtains a set of pseudo masks covered by the mask/> , and among them/> is the number of pre-segmentation pseudo-masks obtained.

Step S03: Cluster according to prototype features to divide multiple different pseudo-classes, and make the pseudo-classes into pseudo-class data sets;

It should be noted that in this embodiment, the background area of each image is marked in order, and each image is obtained pseudo-class prototype and mask, marked as/> , generally take U equal to 5 and save it. Finally, when the annotation is completed, all the prototype vectors obtained are re-clustered, and finally all vectors are divided into/> pseudo-class, and then classify the background sub-region of each image to this/> pseudo-class, label the pseudo-mask with the corresponding pseudo-class label, and obtain the final pseudo-class data set. The specific process is to obtain/> After that, add/> Perform mask average pooling to obtain a background prototype collection of a picture/> , and then run the above process on the entire data set to obtain a prototype set of all images , in/> Running the k-means algorithm based on cosine similarity, the final convergence is/> pseudo-class, usually 100. Based on this, re-label the background category of the original image as/> .

Step S04: Jointly train the semantic segmentation model according to the pseudo-class data set and the base class image data set, so as to perform the segmentation task of the new class image data set through the trained semantic segmentation model;

It should be noted that in this embodiment, the overall data is first divided into base class data sets according to and new class data sets/> , where,/> Data used for training/> for testing,/> Extract support image set/> and query image set , this embodiment adopts the "1-Way-1-Shot" setting method, that is, when the supported image set only extracts an image from one category as a supporting image, query the image set/> It is used to test the model performance in the testing phase. In this embodiment, the PFENet framework is used to extract the feature map from the middle layer of the backbone network. After comparing the features through the support feature map and the query feature map, it is then convolved through the final classifier to obtain the final The result is that while using mid-level features, high-level semantic features are also used to calculate a prior mask as the probability prediction map of the foreground, and a feature enrichment module FEM is designed to establish links after aggregating features at each scale. , extract information from features at various scales, and finally obtain the final prediction result through the classifier, which improves the overall segmentation capability of the framework.

In summary, according to the above-mentioned small-sample semantic segmentation method based on background information mining, by inputting the base class image into the offline background labeling algorithm network, the potential information of the background part in the small-sample semantic segmentation task is mined to obtain the pseudo-class data set, and then Through joint training based on the pseudo-class data set and the original data set, the generalization ability and performance of the semantic segmentation model in the face of new classes are greatly improved, and the base class bias problem of the model is greatly alleviated. Specifically, the preset base class image data set is input into the offline background labeling algorithm network, the foreground area and background area in the base class image are set, and then the base class is obtained through the unsupervised image segmentation algorithm subnetwork and backbone network. The pre-segmented sub-region mask and high-level semantic features of the image are used to extract the prototype features of the background area in the sub-region, clustering is performed based on the prototype features to divide multiple different pseudo-classes, and the pseudo-classes are Made into a pseudo-class data set, because the data is expanded through the offline background labeling algorithm network based on the original data, the training phase of the model will not only involve the base class information, but also the generated background pseudo-class information, so the model is very sensitive to new classes. The generalization ability is significantly improved, and the semantic segmentation model is jointly trained according to the pseudo-class data set and the base class image data set, so as to perform the segmentation task of the new class image data set through the trained semantic segmentation model, It greatly improves the generalization ability and performance of the semantic segmentation model when facing new classes, and greatly alleviates the base class bias problem of the model.

Please refer to Figure 2, which is a flow chart of a small-sample semantic segmentation method based on background information mining proposed by the second embodiment of the present invention. This small-sample semantic segmentation method based on background information mining includes steps S11 to step S15, where :

Step S11: Select the current base class target in the preset base class image data set, and set the foreground area and background area;

Step S12: Scale the base class image and perform pre-segmentation through the unsupervised image segmentation algorithm sub-network to obtain multiple pre-segmented sub-region masks, and then upsample the unsegmented original image of the base class image through the offline background The backbone network in the tagging algorithm network extracts high-level semantic features;

Step S13: After calculating the Hadamard product based on the pre-segmented sub-region mask and high-level semantic features, obtain the prototype features through mask average pooling;

Step S14: Annotate the background areas of all base class images to obtain pseudo class prototype features and pseudo masks, and then perform clustering through an unsupervised clustering algorithm to divide pseudo classes and create pseudo class data sets;

Step S15: Perform joint training based on the base class data set and the pseudo class data set to perform the segmentation task of the new class image data set through the trained semantic segmentation model;

It should be noted that the offline background marking algorithm OBAA (Offline BackgroundAnnotation Algorithm) proposed by this invention is applied to three small sample segmentation frameworks: PFENet, BAM, and the MSANet model improved based on the BAM model. Under the same settings and different base class segmentation, Its performance is compared with the performance of the original framework without using the method of the present invention as follows:

Table 1

Table 2

It can be clearly seen from the results in the above table that the average intersection-over-union ratio (MIoU) of each model after using the background mining algorithm for data expansion under two conditions has been significantly improved. As can be seen from Table 1 above, In the "1-Way-1-Shot" setting, VGG-16 was used as the backbone network and applied to the PFENet framework. The results showed that its average intersection ratio increased by 1.2% compared to the original model. ResNet was then used as the backbone. network, the average intersection and union ratio increased by 0.78%, and after using the OBAA algorithm for the BAM model and MSANet, the average intersection and union ratio and the foreground-background-intersection and union ratio on the PASCAL-5i data set also improved significantly, and the average intersection and union ratio also improved significantly. The ratios increased by 0.54% and 0.6% respectively. As can be seen from Table 2 above, under the "1-Way-5-Shot" condition, the BAM model has an average intersection and union ratio on the PASCAL-5i data set after using the OBAA algorithm. The intersection-to-merge ratio of foreground and background has also been significantly improved. The average intersection-to-merge ratio has increased by 0.48%, and the foreground and background-to-intersection ratio has increased by 1.06%. Under the "1-Way-1-Shot" setting conditions, the COCO- Tests were also conducted on the 20i data set, and the performance comparison results are as follows:

table 3

The overall segmentation difficulty of the COCO-20i data set is greater than that of the PASCAL-5i data set. As can be seen from Table 3 above, the method proposed by the present invention has been improved when four types of sets are selected on the COCO-20i data set. Overall The average intersection ratio has increased by 1.67%, and the segmentation effect has been improved even more obviously.

In summary, according to the above-mentioned small-sample semantic segmentation method based on background information mining, by inputting the base class image into the offline background labeling algorithm network, the potential information of the background part in the small-sample semantic segmentation task is mined to obtain the pseudo-class data set, and then Through joint training based on the pseudo-class data set and the original data set, the generalization ability and performance of the semantic segmentation model in the face of new classes are greatly improved, and the base class bias problem of the model is greatly alleviated. Specifically, the preset base class image data set is input into the offline background labeling algorithm network, the foreground area and background area in the base class image are set, and then the base class is obtained through the unsupervised image segmentation algorithm subnetwork and backbone network. The pre-segmented sub-region mask and high-level semantic features of the image are used to extract the prototype features of the background area in the sub-region, clustering is performed based on the prototype features to divide multiple different pseudo-classes, and the pseudo-classes are Made into a pseudo-class data set, because the data is expanded through the offline background labeling algorithm network based on the original data, the training phase of the model will not only involve the base class information, but also the generated background pseudo-class information, so the model is very sensitive to new classes. The generalization ability is significantly improved, and the semantic segmentation model is jointly trained according to the pseudo-class data set and the base class image data set, so as to perform the segmentation task of the new class image data set through the trained semantic segmentation model, It greatly improves the generalization ability and performance of the semantic segmentation model when facing new classes, and greatly alleviates the base class bias problem of the model.

Please refer to Figure 3, which is a schematic structural diagram of a small-sample semantic segmentation system based on background information mining proposed in the third embodiment of the present invention. The system includes:

The background mining module 10 is used to input the preset base class image data set into the offline background labeling algorithm network, and obtain the pre-segmented sub-region mask and high-level layer of the base class image through the unsupervised image segmentation algorithm sub-network and backbone network. Semantic features to extract prototype features of the background area in the sub-region, cluster according to the prototype features to divide multiple different pseudo-classes, and make the pseudo-classes into a pseudo-class data set;

The joint training module 20 is used to jointly train a semantic segmentation model according to the pseudo class data set and the base class image data set, so as to perform the segmentation task of the new class image data set through the trained semantic segmentation model.

Further, the background mining module 10 includes:

The feature extraction unit 101 is used to input the preset base class image data set into the offline background labeling algorithm network, and obtain the pre-segmented sub-region mask and high-level layer of the base class image through the unsupervised image segmentation algorithm sub-network and backbone network. Semantic features to extract prototype features of the background area in the sub-region;

The pseudo-class dividing unit 102 is configured to perform clustering according to the prototype features to divide a plurality of different pseudo-classes, and produce the pseudo-classes into a pseudo-class data set.

Further, the joint training module 20 includes:

The joint training unit 201 is used to jointly train a semantic segmentation model according to the pseudo class data set and the base class image data set, so as to perform the segmentation task of the new class image data set through the trained semantic segmentation model.

On the other hand, the present invention also provides a computer storage medium on which one or more programs are stored, which implement the above-mentioned small sample semantic segmentation method based on background information mining when executed by the processor.

Another aspect of the present invention also proposes a computer device, including a memory and a processor, wherein the memory is used to store computer programs, and the processor is used to execute the computer programs stored on the memory to implement the above background-based A small-sample semantic segmentation method for information mining.

It will be understood by those skilled in the art that the logic and/or steps represented in the flowchart diagrams or otherwise described herein may, for example, be considered a sequenced list of executable instructions for implementing the logical functions, which may be embodied in in any computer-readable medium for use by an instruction execution system, device, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from the instruction execution system, device, or device), or Used in connection with these instruction execution systems, devices or equipment. For the purposes of this specification, a "computer-readable medium" may be any device that can contain a program that stores, communicates, propagates, or transports a program for use by or in connection with an instruction execution system, apparatus, or device.

More specific examples (non-exhaustive list) of computer readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk cartridge (magnetic device), random access memory (RAM), Read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, and subsequently edited, interpreted, or otherwise suitable as necessary. process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a gate circuit with a logic gate circuit for implementing a logic function on a data signal; Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

In the description of this specification, reference to the terms "one embodiment," "some embodiments," "an example," "specific examples," or "some examples" or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

The above-mentioned embodiments only express several implementation modes of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of the present invention. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the scope of protection of the patent of the present invention should be determined by the appended claims.

Claims (10)

1. The small sample semantic segmentation method based on background information mining is characterized by comprising the following steps of:

inputting a preset base class image data set into an offline background marking algorithm network;

acquiring a pre-segmentation subarea mask and high-level semantic features of the base class image through an unsupervised image segmentation algorithm sub-network and a backbone network so as to extract prototype features of a background area in the subarea;

clustering according to the prototype features to divide a plurality of different pseudo classes, and manufacturing the pseudo classes into a pseudo class data set;

and carrying out joint training on the semantic segmentation model according to the pseudo-class data set and the base class image data set so as to carry out segmentation tasks of the new class image data set through the trained semantic segmentation model.

2. The method for semantic segmentation of small samples based on background information mining according to claim 1, wherein the step of inputting a pre-set base class image dataset into an offline background labeling algorithm network comprises:

selecting a current base object in a preset base image data set, setting a base region in the base image as a foreground region, and setting a non-base region in the base image as a background region.

3. The background information mining-based small sample semantic segmentation method according to claim 1, wherein the step of acquiring pre-segmented sub-region masks and high-level semantic features of the base class image through an unsupervised image segmentation algorithm sub-network and a backbone network to extract prototype features of background regions in the sub-regions comprises:

scaling the preset base class image in the base class image dataset to a preset image size threshold to be segmented;

pre-segmenting the base class image through an unsupervised image segmentation algorithm sub-network in an offline background marking algorithm network to obtain a plurality of pre-segmented sub-region masks;

and carrying out up-sampling operation on the undivided original image of the base image through a backbone network in the offline background marking algorithm network, and extracting high-level semantic features of the undivided original image.

4. The background information mining-based small sample semantic segmentation method according to claim 1, wherein the step of acquiring pre-segmented sub-region masks and high-level semantic features of the base class image through an unsupervised image segmentation algorithm sub-network and a backbone network to extract prototype features of background regions in the sub-regions further comprises:

performing mask negation operation through the base class target mask so as to take out the background area of the current base class target and inhibit the foreground area;

and carrying out Hadamard product calculation according to the pre-segmentation subarea mask and the high-level semantic features by the following formula:

wherein,for said high-level semantic features, wherein +.>，/>，/>The height, width and channel dimension of the high-level semantic features are respectively +.>Representing broadcasting the mask along the channel, +.>Representing Hadamard product, ->Is a pseudo mask covered by a mask, wherein +.>For the number of pre-divided sub-area masks obtained,/->Temporary pseudo mask labeling for background area of single image, wherein +.>，/>The number of foreground base class and background prototype feature corresponding to the background mask respectively, ++>Is a true background mask;

prototype features were obtained by mask averaging pooling.

5. The background information mining-based small sample semantic segmentation method according to claim 1, wherein the step of clustering according to the prototype features to divide a plurality of different pseudo-classes and making the pseudo-classes into a pseudo-class dataset comprises:

labeling background areas of all base class images to obtain pseudo-class prototype features and pseudo masks;

clustering all the pseudo-class prototype features through an unsupervised clustering algorithm to perform pseudo-class division;

classifying the pre-divided subareas of the background area into corresponding pseudo classes, and marking the pseudo masks with corresponding pseudo class labels to manufacture and acquire pseudo class data sets.

6. The background information mining-based small sample semantic segmentation method according to claim 1, wherein the step of jointly training a semantic segmentation model according to the pseudo-class data set and the base-class image data set to perform a segmentation task of a new-class image data set through the trained semantic segmentation model comprises:

respectively inputting the basic class data set and the pseudo class data set into a joint training backbone network;

extracting feature images through the joint training backbone network to obtain a support feature image and a query feature image;

after multi-scale feature extraction is carried out through a feature enrichment module, the support feature map and the query feature map are compared and integrated;

and convolving the integrated feature images and obtaining a final prediction result through a classifier.

7. The background information mining-based small sample semantic segmentation method according to claim 6, wherein after the step of convolving the integrated feature map and obtaining a final prediction result through a classifier, further comprises:

after the final prediction result is obtained by the classifier, the loss function of the original data composed of the base classes is calculated according to the following formula：

Then, the loss function of the pseudo-class data is calculated according to the following formula：

Calculating the overall loss function according to the following formulaL：

In the above formula, the water content of the water-soluble polymer,segmentation of the result for the predicted query image, +.>For the spatial position of the corresponding pixel point, +.>Is the true value mask of the query image, +.>And->A split prediction mask and a pseudo class mask after the pseudo class has passed through the small sample split network,is a super parameter.

8. A small sample semantic segmentation system based on background information mining, comprising:

the background mining module is used for inputting a preset base class image data set into an offline background marking algorithm network, acquiring a pre-segmentation subarea mask and high-level semantic features of the base class image through an unsupervised image segmentation algorithm sub-network and a backbone network, extracting prototype features of a background area in the subarea, clustering according to the prototype features, dividing a plurality of different pseudo classes, and manufacturing the pseudo classes into a pseudo class data set;

and the joint training module is used for carrying out joint training on the semantic segmentation model according to the pseudo-class data set and the base class image data set so as to carry out segmentation tasks of the new class image data set through the trained semantic segmentation model.

9. A storage medium, comprising: the storage medium stores one or more programs which, when executed by a processor, implement the small sample semantic segmentation method based on background information mining of any of claims 1-7.

10. A computer device comprising a memory and a processor, wherein:

the memory is used for storing a computer program;

the processor is configured to implement the small sample semantic segmentation method based on background information mining of any one of claims 1-7 when executing a computer program stored on the memory.