CN116229063B - Semantic segmentation network model and its training method based on category colorization technology - Google Patents

Abstract

The invention discloses a training method for a semantic segmentation network model based on category colorization technology. It first obtains the ADE20K data set, divides the data set into a training set and a test set, generates a category color array with a dimension of C×3, and uses The training set trains the image feature extractor, discrete feature number classifier and auxiliary pixel classifier to obtain the weights of the trained image feature extractor and discrete feature number classifier and the image feature extractor generated in multiple iterative processes. and the weights of the discrete feature ordinal classifier. The learnable color patch-category mapping module is trained using the training set, the image feature extractor generated in multiple iterations, and the weights of the discrete feature ordinal classifier. The present invention can solve the technical problem that the existing semantic segmentation method based on the discriminant model cannot achieve optimal accuracy due to insufficient knowledge and insufficient information, resulting in poor accuracy and poor generalization.

Description

Semantic segmentation network model and its training method based on category colorization technology

Technical field

The invention belongs to the technical field of image data processing, and more specifically, relates to a semantic segmentation network model based on category colorization technology and a training method thereof.

Background technique

Nowadays, semantic segmentation (Semantic Segmentation) has been increasingly widely used in the field of computer vision, including autonomous driving, robots, matting software, etc. Since each image contains rich semantic entities, how to store richer semantic knowledge in the model has become the key to improving the detection performance of the semantic model.

Existing semantic segmentation methods are mainly divided into discriminative models and generative models.

The discriminative semantic segmentation model first uses the feature extraction network to obtain the feature matrix of the entire image. Then, a pixel-level classifier is used to convert the features at each pixel position on these features into probability values of the categories. Among them, the number of probability values is equal to the total number of categories. Finally, the category with the largest category probability at each pixel position is used as the category of the pixel to obtain the semantic segmentation result. In the process of offline training, the discriminative model directly uses the cross-entropy loss function to supervise the offline training of the model. The training process and inference process of this method are relatively simple and easy for business use.

The generative model first uses the feature extraction network to obtain the feature matrix of the entire image. Then, the feature matrix is input into a classifier of discrete feature numbers to obtain the number matrix. Subsequently, the discrete features corresponding to each discrete feature serial number are queried in the discrete feature code table, and all the queried discrete features are spliced into a discrete feature matrix according to the spatial position of the elements. Finally, the discrete feature matrix is input into the decoder of VQ-VAE to obtain the semantic segmentation results. In the process of offline training of the generative semantic segmentation model, the VQ-VAE encoder needs to be used to map the true value of the semantic mask into a feature matrix, and each feature in the feature matrix is the discrete feature of VQ-VAE. A matrix of discrete feature serial numbers composed of serial numbers in the code table. The generative model uses the discrete feature number matrix as the supervision signal and cross-entropy loss as the loss function to supervise the offline training of the model. This method can learn rich semantic information and detailed features due to the introduction of the idea of generative models.

However, existing semantic segmentation methods based on discriminative models and semantic segmentation methods based on generative models have some shortcomings that cannot be ignored:

First, because the pixel-level classifier in the discriminative semantic segmentation model essentially learns features of different categories and their discriminative boundaries, it does not learn enough semantic information and detailed features, which results in the existing discriminative semantic segmentation It is difficult for the model to achieve the best mean intersection over Union (mIoU) accuracy, which will lead to poor final accuracy and generalization.

Second, the existing pixel-level classifiers based on discriminative semantic segmentation methods use the characteristics of each pixel position to predict the category of the pixel, and do not fully utilize the information of the local area, resulting in the pixel prediction of the local area being too independent. It is easy to form small noise points in the semantic segmentation results, which in turn affects the accuracy of the final semantic segmentation.

Third, because the existing semantic segmentation methods based on generative models do not properly handle the unlabeled areas that are commonly found in the true values of the semantic masks in the training samples during the training process, resulting in unlabeled areas being input to VQ- The layer-by-layer diffusion in the VAE encoder process results in the true values of a large number of discrete feature number matrices containing information about unlabeled areas, which in turn causes the trained model to incorrectly predict some difficult-to-predict categories as unlabeled categories. , ultimately reducing the accuracy of the semantic segmentation model.

Contents of the invention

In view of the above defects or improvement needs of the existing technology, the present invention provides a semantic segmentation network model based on category colorization technology and its training method. Its purpose is to solve the problem of insufficient knowledge of the existing semantic segmentation method based on the discriminant model. , The information is not rich enough, resulting in the method not being able to achieve optimal accuracy, which in turn leads to technical problems such as poor accuracy and generalization, and the existing semantic segmentation methods based on discriminant models do not fully utilize local information, resulting in local areas. The category prediction of pixels is too independent, resulting in noise in the semantic segmentation results, which in turn leads to technical problems with poor accuracy, and existing semantic methods based on generative models cannot properly handle uncategorized areas in the semantic masks of training samples. , leading to the introduction of erroneous information in the training process, which in turn leads to technical problems such as poor accuracy in semantic segmentation results.

In order to achieve the above object, according to one aspect of the present invention, a semantic segmentation network model based on category colorization technology is provided, including an image feature extractor, a discrete feature serial number classifier, a discrete feature code table, and a semantic image decoding connected in sequence. , learnable color block-category mapping module, auxiliary pixel classifier, unlabeled region completion module, category-color mapping module, and semantic image encoder, 24 Swin Transformer modules, used to receive input dimensions of bs× For a 3×h×w image, the output is four feature matrices. The dimensions of the feature matrices are and/> Among them, bs is the batch data size preset during the offline training process, h and w are the number of pixels on the long side and the number of pixels on the short side of the image respectively.

The discrete feature serial number classifier includes a feature aggregation layer, a feature processing layer and a classifier module.

The feature aggregation layer is a convolution module whose input is the four feature matrices output by the Swin Transformer network, and the output is an aggregated feature matrix whose dimensions are

The feature processing layer consists of two layers of Swin Transformer modules. Its input is the feature matrix aggregated by the feature aggregation layer. The output dimension is the feature matrix, whose dimensions are

The input of the classifier module is the feature matrix output by the feature processing layer, and the output is a probability matrix whose dimensions are Each element in the probability matrix is the probability that 8192 discrete features appear in that element.

The discrete feature code table is a collection of multiple discrete features. The dimension of each discrete feature is 128. Each discrete feature has a unique serial number in the discrete feature code table, that is, the discrete feature serial number.

The input of the discrete feature code table is a probability matrix, and the output is a discrete feature matrix whose dimensions are

The semantic image decoder specifically uses the decoder of DALL-E's VQ-VAE model. The input is the discrete feature matrix output from the discrete feature code table, and the dimension is The output is the predicted semantic image, whose dimensions are bs × 3 × h × w.

The learnable color block-category mapping module is a single-layer Swin Transformer module. The input of the learnable color patch-category mapping module is the predicted semantic image with dimension bs×3×h×w output by the semantic image decoder, and the output is a pixel category probability matrix with dimension bs×C×h×w.

The auxiliary pixel classifier is a convolution module whose input is the aggregated feature matrix output by the feature aggregation layer of the discrete feature classifier, and its dimension is The output is an auxiliary semantic mask with dimensions bs×1×h×w.

The unlabeled region completion module is used to replace the unlabeled pixel regions in the ground truth of the semantic mask with the categories predicted by the auxiliary pixel classifier. The inputs to the unlabeled region completion module are the ground-truth values of the semantic masks from the dataset and the auxiliary semantic masks output by the auxiliary pixel classifier, both of which have dimensions bs×1×h×w. The output of the unlabeled region completion module is the true value of the semantic mask of the full pixel annotation, and its dimension is bs×1×h×w.

The category-color mapping module is used to obtain the color corresponding to each pixel according to its category. Its input is the true value of the full-pixel annotated semantic mask output by the unlabeled area completion module, and its dimension is bs×1×h. ×w, the output is the ground truth of the semantic image with dimensions bs×3×h×w.

The semantic image encoder specifically uses the encoder of DALL-E's VQ-VAE model. Its input is the true value of the semantic image output by the category-color mapping module. The dimension is bs×3×h×w, and the output is the dimension of The true value of the equivalent characteristic matrix of .

Preferably, the discrete feature code table is to find the discrete feature sequence number corresponding to the discrete feature with the highest probability in each element of the probability matrix. The sequence numbers corresponding to all elements in the probability matrix form a sequence number matrix. Then, according to each element in the sequence number matrix The elements query the corresponding discrete features in the discrete feature code table, and all discrete features corresponding to all elements form a discrete feature matrix.

The learnable color block-category mapping module first traverses all pixels on the semantic image, targeting all pixels {L(p,q) of the square local area where the pixel L(i,j) in the i-th row and j-th column is the center point )|p∈[i-6,i+6],p∈[j-6,j+6]}, the learnable color block-category mapping module predicts C of its center point pixel L(i,j) Category probability, where C is the total number of categories, i∈[1,h], j∈[1,w]. Finally, the category probabilities of all pixels L(i,j) in the semantic image are spliced according to their spatial positions to obtain the predicted pixel category probability matrix.

According to another aspect of the present invention, a method for training a semantic segmentation network model based on category colorization technology is provided, including the following steps:

(1) Obtain the ADE20K data set, divide the 25574 sets of images in the ADE20K data set and the true values of their corresponding semantic masks into a training set, and divide the 2000 sets of images in the ADE20K data set and the true values of their corresponding semantic masks. divided into validation sets.

(2) Generate a category color array with dimension C×3.

(3) Use the training set of the ADE20K data set obtained in step (1) to train the image feature extractor, discrete feature serial number classifier and auxiliary pixel classifier to obtain the trained image feature extractor and discrete feature serial number classification The weights of the image feature extractor and the discrete feature number classifier generated during multiple iterations.

(4) Use the training set of the ADE20K data set obtained in step (1), one of the weights of the image feature extractor and discrete feature number classifier generated in the multiple iteration processes obtained in step (4), to learn the The color block-category mapping module is trained to obtain a trained and learnable color block-category mapping module.

(5) The weights of the trained image feature extractor and discrete feature number classifier obtained in step (3), and the weights of the trained learnable color block-category mapping module obtained in step (4) are obtained from the network. The downloaded discrete feature code table and the weights of the semantic image decoder are saved to obtain the weights of the semantic segmentation network model based on category colorization technology. Among them, the weight of the discrete feature code table and the weight of the semantic image decoder are respectively the weight of the discrete feature code table and the weight of the decoder of DALL-E's VQ-VAE model downloaded from the Internet.

Preferably, step (2) includes the following sub-steps:

(2-1) Generate three one-dimensional arrays A _R , A _G , and A _B . Preferably, each element of the array is and/> And there are k1∈[1, the total number of elements in the array A _R ], k2∈[1, the total number of elements in the array A _G ], k3∈[1, the total number of elements in the array A _B ].

(2-2) Set counters k1=1, k2=1, k3=1, and initialize RGB color array A _RGB to be an empty array.

(2-3) Determine whether k1 is greater than the preset maximum number of cycles J (its value is equal to the total number of elements in the array A _R ). If so, go to step (2-13), otherwise go to step (2-4 ).

(2-4) Determine whether k2 is greater than the preset maximum number of cycles K (its value is equal to the total number of elements in the array A _G ). If so, go to step (2-3), otherwise go to step (2-5) .

(2-5) Determine whether k3 is greater than the preset maximum number of cycles Q (its value is equal to the total number of elements in array A _B ). If so, go to step (2-4), otherwise go to step (2-6) .

(2-6) Generate a random integer r _R between -15 and 15, and update the k1th element of the A _R array To get the k1th element/> of the updated A _R array

(2-7) Generate a random integer r _G between -15 and 15, and update the k2th element of the A _G array To get the k2th element/> of the updated A _G array

(2-8) Generate a random integer r _B between -15 and 15, and update the k3th element of the A _B array To get the k3th element/> of the updated A _B array

(2-9) Add the k1th element of the updated A _R array obtained in step (2-6) The k2th element of the updated A _G array obtained in step (2-7)/> And the k3th element/> of the updated A _B array obtained in step (2-8) Make up a three-dimensional element/> and add three-dimensional elements/> Insert into the end of the RGB color array A _RGB to get the updated RGB color array A _RGB .

(2-10) Set k1=k1+1 and return to step (2-5)

(2-11) Set k2=k2+1 and return to step (2-4)

(2-12) Set k3=k3+1 and return to step (2-3)

(2-13) Obtain the first C three-dimensional elements in the updated RGB color array A _RGB obtained in step (2-9) to obtain the category color array.

Preferably, step (3) includes the following sub-steps:

(3-1) Initialize the weights of the image feature extractor, discrete feature number classifier, auxiliary pixel classifier, category-color mapping module, semantic image encoder and discrete feature code table to obtain the initialized image feature extraction , discrete feature number classifier, auxiliary pixel classifier, category-color mapping module, semantic image encoder and discrete feature code table.

(3-2) Set counter i=1 and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process.

(3-3) Obtain the true values of multiple images and their corresponding semantic masks from the training set of the ADE20K data set obtained in step (1).

(3-4) Perform data preprocessing on the true values of multiple images and semantic masks obtained in step (3-3) to obtain multiple preprocessed images and true values of semantic masks.

(3-5) Use the sequentially connected image feature extractor and discrete feature number classifier to map the multiple preprocessed images obtained in step (3-4) into a probability matrix of multiple discrete feature numbers, with the dimension and multiple aggregated feature matrices, whose dimensions are/>

(3-6) Input the multiple aggregated feature matrices obtained in step (3-5) into the auxiliary pixel classifier to obtain multiple auxiliary semantic masks, whose dimensions are bs×1×h×w .

(3-7) Input the true values of the multiple preprocessed semantic masks obtained in step (3-4) and the auxiliary semantic masks obtained in step (4-7) into the unlabeled area completion module , to obtain the true values of multiple full-pixel annotated semantic masks, whose dimensions are bs×1×h×w.

(3-8) Use the category-color mapping module, semantic image encoder, and discrete feature code table to map the true values of multiple full-pixel semantic masks obtained in step (3-7) into multiple discrete feature number matrices 's true value.

(3-9) Input the probability matrices of multiple discrete feature numbers obtained in step (3-5) and the true values of the multiple discrete feature number matrices obtained in step (4-7) into the cross entropy loss function to obtain the semantics Feature loss value.

(3-10) Use the semantic feature loss value obtained in step (3-9) to perform backpropagation to obtain the gradient of the image feature extractor and discrete feature number classifier.

(3-11) Use the learning rate set in step (3-2), the AdamW optimizer and the gradient obtained in step (3-9) to update the weights of the image feature extractor and discrete feature number classifier to obtain a new image Weights for feature extractors and discrete feature ordinal classifiers.

(3-12) Counter i=i+1, and set the gradient of the image feature extractor and discrete feature number classifier to 0.

(3-13) Determine whether i is an integer multiple of the preset model saving interval t. If so, save the weights of the image feature extractor and the discrete feature serial number classifier, and name them as the image feature extractor of the i-th iteration. and the weight of the discrete feature serial number classifier, otherwise go to step (3-14).

(3-14) Determine whether i is greater than the preset maximum number of iterations n, if so, go to step (3-15), otherwise return to step (3-3).

(3-15) Save the weights of the image feature extractor and discrete feature number classifier to obtain the weights of the trained image feature extractor and discrete feature number classifier.

Preferably, step (3-1) includes the following sub-steps:

(3-1-1) Load the pre-trained weights of the image feature extractor into the image feature extractor.

(3-1-2) Initialize the weight of the discrete feature serial number classifier to a random value, and set the weight of the discrete feature serial number classifier to a gradient weight, that is, the weight of the image feature extractor will be optimized during the offline training process.

(3-1-3) Initialize the weight of the auxiliary pixel classifier to a random value, and set the weight of the pixel classifier to a gradient weight.

(3-1-4) Load the preset category color array obtained in step (2) into the category-color mapping module, and set the weight of the category-color mapping module to a gradient-free weight.

(3-1-5) Add the pre-trained weights of the semantic image encoder to the semantic image encoder, and set the weights of the semantic image encoder to gradient-free weights.

(3-1-6) Load the discrete signature table and training weights into the discrete signature table, and set the weight of the discrete signature table to a gradient-free weight.

Preferably, step (3-8) includes the following sub-steps:

(3-8-1) Input the true values of multiple full-pixel annotated semantic masks obtained in step (3-7) into the category-color mapping module to obtain the true values of multiple semantic images.

(3-8-2) Input the true values of multiple semantic images obtained in step (3-8-1) to the initialized semantic image encoder obtained in step (3-1-5) to obtain multiple equal values. The true value of the valence characteristic matrix.

(3-8-3) For the true value of each equivalent characteristic matrix obtained in step (3-8-2), query the equivalent value in the initialized discrete characteristic code table obtained in step (3). The discrete features and their discrete feature serial numbers that are closest to the true value of the feature matrix are spliced together according to the spatial position of all the queried discrete feature serial numbers to obtain the true value of the discrete feature serial number matrix, whose dimensions are

Preferably, step (4) includes the following sub-steps:

(4-1) Initialize the weights of the image feature extractor, discrete feature number classifier, discrete feature code table, semantic image decoder and learnable color block-category mapping module to obtain the initialized image feature extractor , discrete feature number classifier, discrete feature code table, semantic image decoder and learnable color block-category mapping module.

(4-2) Set counter i=1 and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process.

(4-3) Obtain the true values of multiple images and their corresponding semantic masks from the training set of the ADE20K data set obtained in step (1).

(4-4) Perform data preprocessing on the true values of multiple images and semantic masks obtained in step (5-3) to obtain multiple preprocessed images and true values of semantic masks.

(4-5) Use the sequentially connected image feature extractor and discrete feature number classifier to map the multiple preprocessed images obtained in step 4-4) into a probability matrix of multiple discrete feature numbers, with the dimension

(4-6) Input the probability matrices of multiple discrete feature numbers obtained in step (4-5) into the discrete feature code table to obtain multiple discrete feature matrices, the dimensions of which are

(4-7) Input the multiple discrete feature matrices obtained in step (4-6) into the initialized semantic image decoder obtained in step (6-1) to obtain multiple predicted semantic images whose dimensions It is bs×3×h×w.

(4-8) Input the multiple predicted semantic images obtained in step (4-7) into the learnable color block-category mapping module to obtain multiple predicted pixel category probability matrices, whose dimensions are bs×C ×h×w. Among them, C is the total number of categories.

(4-9) Input the multiple predicted pixel category probability matrices obtained in step (4-8) and the true values of the multiple preprocessed semantic masks obtained in step (4-4) into the cross entropy loss function, To get the pixel classification loss value.

(4-10) Use the pixel classification loss value obtained in step (4-9) to perform backpropagation to obtain the gradient of the learnable color block-category mapping module.

(4-11) Utilize the learning rate set in step (4-2), the AdamW optimizer and the gradient obtained in step (4-10) to update the weight of the learnable color block-category mapping module to obtain a new learnable The weight of the color block-category mapping module.

(4-12) Counter i=i+1, and set the learnable color block-category mapping module to 0.

(4-13) Determine whether i is greater than the preset maximum number of iterations n, if so, go to step (4-14), otherwise return to step (4-3).

(4-14) Save the weight of the learnable color block-category mapping module to obtain the weight of the trained learned color block-category mapping module.

Preferably, step (4-1) includes the following sub-steps:

(4-1-1) Load the weight of the k-th iteration of the image feature extractor obtained in step (4) to the image feature extractor, and set the weight of the image feature extractor to a gradient-free weight.

(4-1-2) Load the weight of the k-th iteration of the discrete feature number classifier obtained in step (4) to the image feature extractor, and set the weight of the discrete feature number classifier to a gradient-free weight.

(4-1-3) Load the pre-trained weights of the discrete signature table into the discrete signature table, and set the weight of the discrete signature table to a gradient-free weight.

(4-1-4) Load the pre-trained weights of the semantic image decoder into the semantic image decoder, and set the weights of the semantic image decoder to gradient-free weights.

(4-1-5) Initialize the weight of the learnable color block-category mapping module to a random value, and set the weight of the learnable color block-category mapping module to a gradient weight.

Generally speaking, compared with the prior art, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

(1) Because the present invention uses a discrete feature serial number classifier, a discrete feature code table, and a semantic image decoder. It converts the colors of categories into colors and models semantic segmentation from the task of pixel classification to the task of image generation, effectively taking advantage of the generative model's ability to contain richer semantic information. Therefore, the present invention solves the limitation of insufficient richness of semantic information learned by existing discriminative semantic segmentation methods, greatly improving the accuracy and generalization of semantic segmentation.

(2) Because this method uses a learnable color block-category mapping module. It can determine the category of the central pixel based on the colorized category information of the local area, avoiding the practice of determining the category of a single pixel based on its characteristics in isolation. Therefore, the present invention solves the technical problem in the existing discriminative semantic segmentation method that the category prediction of pixels in a local area is too independent, resulting in noise in the semantic segmentation results, which in turn leads to poor accuracy.

(3) Because this method uses an auxiliary pixel classifier and an unlabeled area completion module. During the offline training process, it inputs the image features into the auxiliary pixel classifier to obtain the auxiliary semantic mask, and inputs the auxiliary semantic mask into the unlabeled area completion module to calculate the semantic mask containing the unlabeled area. The true value is completed to obtain the true value of the semantic mask of the full pixel annotation. This process avoids information from unlabeled areas being input into the VQ-VAE encoder, thereby obtaining erroneous supervision signals. Therefore, the present invention solves the technical problem that the existing generative semantic segmentation method cannot properly handle the uncategorized area in the semantic mask of the training sample, resulting in the introduction of erroneous information in the training process, which in turn leads to poor accuracy of the semantic segmentation results. .

Description of the drawings

Figure 1 is a schematic diagram of the semantic segmentation network model based on category colorization technology of the present invention.

Figure 2 is a flow chart of the training method of the semantic segmentation network model based on category colorization technology of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

As shown in Figure 1, the present invention provides a semantic segmentation network model based on category colorization technology, which includes an image feature extractor, a discrete feature serial number classifier, a discrete feature code table, a semantic image decoder, and a The learned color block-category mapping module, auxiliary pixel classifier, unlabeled region completion module, category-color mapping module and semantic image encoder, in which the category-color mapping module, semantic image encoder and auxiliary pixel classifier only use For offline training of semantic segmentation network models based on category colorization technology.

The image feature extractor is a Swin Transformer network, which includes 24 SwinTransformer modules connected in sequence. The image feature extractor receives an image with input dimensions of bs×3×h×w, and outputs four feature matrices. The dimensions of the feature matrices are and/> Among them, bs is the batch data size preset during the offline training process, h and w are the number of pixels on the long side and the number of pixels on the short side of the image respectively.

The discrete feature serial number classifier includes a feature aggregation layer, a feature processing layer and a classifier module.

The feature aggregation layer is a convolution module whose input is the four feature matrices output by the Swin Transformer network, and the output is an aggregated feature matrix whose dimensions are

The feature processing layer consists of two layers of Swin Transformer modules. Its input is the feature matrix aggregated by the feature aggregation layer. The output dimension is the feature matrix, whose dimensions are

The input of the classifier module is the feature matrix output by the feature processing layer, and the output is a probability matrix whose dimensions are Each element in the probability matrix is the probability that 8192 discrete features appear in that element.

The discrete feature code table is a collection of multiple discrete features. The dimension of each discrete feature is 128. Each discrete feature has a unique serial number in the discrete feature code table, that is, the discrete feature serial number.

The input of the discrete feature code table is a probability matrix, and the output is a discrete feature matrix whose dimensions are Specifically, the first step is to find the discrete feature sequence number corresponding to the discrete feature with the highest probability in each element of the probability matrix. The sequence numbers corresponding to all elements in the probability matrix form a sequence number matrix. Then, according to each element in the sequence number matrix, Query the corresponding discrete features in the discrete feature code table, and all discrete features corresponding to all elements form a discrete feature matrix.

The semantic image decoder specifically uses the decoder of DALL-E's VQ-VAE model. The input is the discrete feature matrix output from the discrete feature code table, and the dimension is The output is the predicted semantic image, whose dimensions are bs × 3 × h × w.

The weights of the semantic image decoder are the weights of the decoder of DALL-E's VQ-VAE model downloaded from the network and loaded into the semantic image decoder.

The learnable color block-category mapping module is a single-layer Swin Transformer module. The input of the learnable color patch-category mapping module is the predicted semantic image with dimension bs×3×h×w output by the semantic image decoder, and the output is a pixel category probability matrix with dimension bs×C×h×w. Specifically, the learnable color block-category mapping module will first traverse all pixels on the semantic image, targeting all pixels in the square local area where the pixel L(i,j) in the i-th row and j-th column is the center point { L(p,q)|p∈[i-6,i+6],p∈[j-6,j+6]}, the learnable color block-category mapping module predicts its center point pixel L(i, C category probabilities of j), where C is the total number of categories, i∈[1,h], j∈[1,w]. Finally, the category probabilities of all pixels L(i,j) in the semantic image are spliced according to their spatial positions to obtain the predicted pixel category probability matrix.

The auxiliary pixel classifier is a convolution module whose input is the aggregated feature matrix output by the feature aggregation layer of the discrete feature classifier, and its dimension is The output is an auxiliary semantic mask with dimensions bs×1×h×w. The auxiliary semantic mask is used to complete the pixel areas of unlabeled categories in the true value of the semantic mask of the samples in the training set, and then obtain the true value of the fully pixel-labeled semantic mask. Among them, the true value of the semantic segmentation mask comes from the data set used during offline training. The value range of each pixel y _gt in the true value of the semantic mask is y _gt ∈[1,C]∪255, where C is the total number of categories. y _gt ∈ [1,C] indicates that the pixel has a category label, and y _gt =255 indicates that the pixel has no category label.

The unlabeled region completion module is used to replace the unlabeled pixel regions in the ground truth of the semantic mask with the categories predicted by the auxiliary pixel classifier. The inputs to the unlabeled region completion module are the ground-truth values of the semantic masks from the dataset and the auxiliary semantic masks output by the auxiliary pixel classifier, both of which have dimensions bs×1×h×w. The output of the unlabeled region completion module is the true value of the semantic mask of the full pixel annotation, and its dimension is bs×1×h×w.

The category-color mapping module is used to obtain the color corresponding to each pixel according to its category. Its input is the true value of the full-pixel annotated semantic mask output by the unlabeled area completion module, and its dimension is bs×1×h. ×w, the output is the ground truth of the semantic image with dimensions bs×3×h×w. The semantic image encoder specifically uses the encoder of DALL-E's VQ-VAE model. Its input is the true value of the semantic image output by the category-color mapping module. The dimension is bs×3×h×w, and the output is the dimension of The true value of the equivalent characteristic matrix of .

The semantic segmentation network model based on category colorization technology is trained using the following steps:

(1) Obtain the ADE20K data set, divide the 25574 sets of images in the ADE20K data set and the true values of their corresponding semantic masks into a training set, and divide the 2000 sets of images in the ADE20K data set and the true values of their corresponding semantic masks. divided into validation sets.

(2) Generate a category color array with dimension C×3.

Specifically, the category color array is an array of C RGB color values preset for C categories respectively.

This step includes the following sub-steps:

(2-1) Generate three one-dimensional arrays A _R , A _G , and A _B . Preferably, each element of the array is and/> And there are k1∈[1, the total number of elements in the array A _R ], k2∈[1, the total number of elements in the array A _G ], k3∈[1, the total number of elements in the array A _B ].

Specifically, in this example, the total number of elements in the arrays A _R , A _G , and A _B are 5, 6, and 5 respectively.

The advantages of this step are: first, the arrays A _R , A _G , and A _B of each channel are all arithmetic sequences, and the elements are evenly distributed between 0 and 255, making full use of the color space to maximize the absolute color of the category. distance, reducing the possibility of misclassification of pixels. Second, the initial elements of the arrays A _R , A _G , and A _B of each channel are different, and the sequence differences are also different, which avoids the same values of different array elements and reduces the possibility of pixel misclassification.

(2-2) Set counters k1=1, k2=1, k3=1, and initialize RGB color array A _RGB to be an empty array.

(2-3) Determine whether k1 is greater than the preset maximum number of cycles J (its value is equal to the total number of elements in the array A _R ). If so, go to step (2-13), otherwise go to step (2-4 ).

(2-4) Determine whether k2 is greater than the preset maximum number of cycles K (its value is equal to the total number of elements in the array A _G ). If so, go to step (2-3), otherwise go to step (2-5) .

(2-5) Determine whether k3 is greater than the preset maximum number of cycles Q (its value is equal to the total number of elements in array A _B ). If so, go to step (2-4), otherwise go to step (2-6) .

(2-6) Generate a random integer r _R between -15 and 15, and update the k1th element of the A _R array To get the k1th element/> of the updated A _R array

(2-7) Generate a random integer r _G between -15 and 15, and update the k2th element of the A _G array To get the k2th element/> of the updated A _G array

(2-8) Generate a random integer r _B between -15 and 15, and update the k3th element of the A _B array To get the k3th element/> of the updated A _B array

The advantage of the above steps (2-6) to (2-8) is that each element is added with an independent random integer before being inserted into the RGB color array A _RGB , which avoids duplication of element values and reduces pixel errors. Classification possibilities.

(2-9) Add the k1th element of the updated A _R array obtained in step (2-6) The k2th element of the updated A _G array obtained in step (2-7)/> And the k3th element/> of the updated A _B array obtained in step (2-8) Make up a three-dimensional element/> and add three-dimensional elements/> Insert into the end of the RGB color array A _RGB to get the updated RGB color array A _RGB .

(2-10) Set k1=k1+1 and return to step (2-5)

(2-11) Set k2=k2+1 and return to step (2-4)

(2-12) Set k3=k3+1 and return to step (2-3)

(2-13) Obtain the first C three-dimensional elements in the updated RGB color array A _RGB obtained in step (2-9) to obtain the category color array.

(3) Use the training set of the ADE20K data set obtained in step (1) to train the image feature extractor, discrete feature serial number classifier and auxiliary pixel classifier to obtain the trained image feature extractor and discrete feature serial number classification The weights of the image feature extractor and the discrete feature number classifier generated during multiple iterations.

This step contains the following sub-steps:

(3-1) Initialize the weights of the image feature extractor, discrete feature number classifier, auxiliary pixel classifier, category-color mapping module, semantic image encoder and discrete feature code table to obtain the initialized image feature extraction , discrete feature number classifier, auxiliary pixel classifier, category-color mapping module, semantic image encoder and discrete feature code table.

This step includes the following sub-steps:

(3-1-1) Load the pre-trained weights of the image feature extractor into the image feature extractor.

Preferably, the backbone network structure of the large weight version (Large version) of Swin Transformer is used as the structure of the image feature extractor. Download the weights of the ImageNet-22K pre-trained Swin Transformer backbone network from the Internet, load it into the image feature extractor, and set the weight of the image feature extractor to a gradient weight (that is, the weight of the image feature extractor will be in optimized during offline training).

(3-1-2) Initialize the weight of the discrete feature serial number classifier to a random value, and set the weight of the discrete feature serial number classifier to a gradient weight (that is, the weight of the image feature extractor will be optimized during the offline training process) .

(3-1-3) Initialize the weight of the auxiliary pixel classifier to a random value, and set the weight of the pixel classifier to a gradient weight.

(3-1-4) Load the preset category color array obtained in step (2) into the category-color mapping module, and set the weight of the category-color mapping module to a gradient-free weight.

(3-1-5) Add the pre-trained weights of the semantic image encoder to the semantic image encoder, and set the weights of the semantic image encoder to gradient-free weights.

Specifically, this step is to download the weights of the encoder of DALL-E's VQ-VAE model from the Internet and load them into the semantic image encoder.

(3-1-6) Load the discrete signature table and training weights into the discrete signature table, and set the weight of the discrete signature table to a gradient-free weight.

Specifically, this step is to download the weights of the discrete signature table of DALL-E's VQ-VAE model from the Internet and load them into the discrete signature table.

(3-2) Set counter i=1 and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process.

Hyperparameters of the training process include the maximum number of loops, batch size bs, learning rate and optimizer.

Specifically, the batch size bs is set to 32, the initial learning rate is set to 0.00001, and the optimizer is set to the AdamW optimizer.

(3-3) Obtain the true values of multiple images and their corresponding semantic masks from the training set of the ADE20K data set obtained in step (1).

(3-4) Perform data preprocessing on the true values of multiple images and semantic masks obtained in step (3-3) to obtain multiple preprocessed images and true values of semantic masks.

Specifically, the preprocessing process of this step includes unified scale interpolation, random horizontal flipping, random cropping, and normalization. The dimensions of the multiple preprocessed images are bs×3×h×w, which correspond to The dimension of the preprocessed semantic mask is bs×1×h×w, and bs is the batch size set in step (3-2).

(3-5) Use the sequentially connected image feature extractor and discrete feature number classifier to map the multiple preprocessed images obtained in step (3-4) into a probability matrix of multiple discrete feature numbers, with the dimension and multiple aggregated feature matrices, whose dimensions are/>

This step includes the following sub-steps:

(3-5-1) Input each preprocessed image obtained in step (3-4) into the image feature extractor to obtain image features of four different spatial scales and channel numbers corresponding to each image.

(3-5-2) Input the image features of four different spatial scales and channel numbers corresponding to each image obtained in step (3-5-1) into the discrete feature serial number classifier to obtain the discrete number corresponding to each image. The probability matrix of feature numbers and the aggregated feature matrix.

(3-6) Input the multiple aggregated feature matrices obtained in step (3-5) into the auxiliary pixel classifier to obtain multiple auxiliary semantic masks, whose dimensions are bs×1×h×w .

(3-7) Input the true values of the multiple preprocessed semantic masks obtained in step (3-4) and the auxiliary semantic masks obtained in step (4-7) into the unlabeled area completion module , to obtain the true values of multiple full-pixel annotated semantic masks, whose dimensions are bs×1×h×w.

Specifically, this step is to use the pixels with a pixel value of 255 on the true values of the multiple semantic masks obtained in step (2), and use the corresponding positions on the multiple auxiliary semantic masks obtained in step (3-6). pixels to obtain the true values of multiple full-pixel annotated semantic masks.

The advantage of the above steps (3-5) to (3-6) is that the auxiliary pixel classifier uses the aggregated features to generate auxiliary semantic masks. The unlabeled region completion module completes the ubiquitous unlabeled regions in the ground truth of the semantic mask to obtain the ground truth of the semantic mask with category labels on all pixels. This prevents subsequent semantic image encoders from encoding information without category annotations, and avoids these erroneous information from interfering with the true values of discrete feature numbers. Therefore, the problem of uncategorized areas in the true value of the semantic mask interfering with model training, which in turn leads to a decrease in model accuracy, is solved.

(3-8) Use the category-color mapping module, semantic image encoder, and discrete feature code table to map the true values of multiple full-pixel semantic masks obtained in step (3-7) into multiple discrete feature number matrices 's true value.

Specifically, this step is to map the true values of multiple preprocessed semantic masks with dimensions bs×1×h×w into multiple dimensions with The true value of the discrete characteristic number matrix.

This step includes the following sub-steps:

(3-8-1) Input the true values of multiple full-pixel annotated semantic masks obtained in step (3-7) into the category-color mapping module to obtain the true values of multiple semantic images.

(3-8-2) Input the true values of multiple semantic images obtained in step (3-8-1) to the initialized semantic image encoder obtained in step (3-1-5) to obtain multiple equal values. The true value of the valence characteristic matrix.

(3-8-3) For the true value of each equivalent characteristic matrix obtained in step (3-8-2), query the equivalent value in the initialized discrete characteristic code table obtained in step (3). The discrete features and their discrete feature serial numbers that are closest to the true value of the feature matrix are spliced together according to the spatial position of all the queried discrete feature serial numbers to obtain the true value of the discrete feature serial number matrix, whose dimensions are

(3-9) Input the probability matrices of multiple discrete feature numbers obtained in step (3-5) and the true values of the multiple discrete feature number matrices obtained in step (4-7) into the cross entropy loss function to obtain the semantics Feature loss value.

(3-10) Use the semantic feature loss value obtained in step (3-9) to perform backpropagation to obtain the gradient of the image feature extractor and discrete feature number classifier.

(3-11) Use the learning rate set in step (3-2), the AdamW optimizer and the gradient obtained in step (3-9) to update the weights of the image feature extractor and discrete feature number classifier to obtain a new image Weights for feature extractors and discrete feature ordinal classifiers.

(3-12) Counter i=i+1, and set the gradient of the image feature extractor and discrete feature number classifier to 0.

(3-13) Determine whether i is an integer multiple of the preset model saving interval t. If so, save the weights of the image feature extractor and the discrete feature serial number classifier, and name them as the image feature extractor of the i-th iteration. and the weight of the discrete feature serial number classifier, otherwise go to step (3-14).

Specifically, the preset model saving interval t ranges from 1,000 to 10,000, preferably 16,000.

(3-14) Determine whether i is greater than the preset maximum number of iterations n, if so, go to step (3-15), otherwise return to step (3-3).

Specifically, the preset maximum number of cycles n ranges from 1,000 to 1,000,000, preferably 160,000.

(3-15) Save the weights of the image feature extractor and discrete feature number classifier to obtain the weights of the trained image feature extractor and discrete feature number classifier.

(4) Use the training set of the ADE20K data set obtained in step (1), one of the weights of the image feature extractor and discrete feature number classifier generated in the multiple iteration processes obtained in step (4), to learn the The color block-category mapping module is trained to obtain a trained and learnable color block-category mapping module.

This step contains the following sub-steps:

(4-1) Initialize the weights of the image feature extractor, discrete feature number classifier, discrete feature code table, semantic image decoder and learnable color block-category mapping module to obtain the initialized image feature extractor , discrete feature number classifier, discrete feature code table, semantic image decoder and learnable color block-category mapping module.

This step includes the following sub-steps:

(4-1-1) Load the weight of the k-th iteration of the image feature extractor obtained in step (4) to the image feature extractor, and set the weight of the image feature extractor to a gradient-free weight.

Specifically, the preset range of k is an integer less than the maximum number of iterations n and the model saving interval t, preferably 32000.

(4-1-2) Load the weight of the k-th iteration of the discrete feature number classifier obtained in step (4) to the image feature extractor, and set the weight of the discrete feature number classifier to a gradient-free weight.

The advantage of the above steps (4-1-1) to (4-1-2) is that the weights of the image feature extractor and the discrete feature sequence number classifier generated by the intermediate process are used to make the predicted semantic image more accurate. Loud noise. Compared with accurate semantic images, semantic images containing noise are a kind of data enhancement for the learnable color patch-category mapping module. Training with it can effectively improve the accuracy of the color patch-category mapping module using local information to infer the center pixel. performance and robustness. Therefore, the problem of existing discriminative semantic segmentation methods is solved, in which the category prediction of pixels in local areas is too independent, resulting in noise in the semantic segmentation results, which in turn leads to poor accuracy.

(4-1-3) Load the pre-trained weights of the discrete signature table into the discrete signature table, and set the weight of the discrete signature table to a gradient-free weight.

Specifically, this step is to download the weights of the discrete signature table of DALL-E's VQ-VAE model from the Internet and load them into the discrete signature table.

(4-1-4) Load the pre-trained weights of the semantic image decoder into the semantic image decoder, and set the weights of the semantic image decoder to gradient-free weights.

Specifically, this step is to download the weights of the decoder of DALL-E's VQ-VAE model from the network and load them into the semantic image decoder.

(4-1-5) Initialize the weight of the learnable color block-category mapping module to a random value, and set the weight of the learnable color block-category mapping module to a gradient weight (that is, the learnable color block-category The weights of the mapping module will be optimized during offline training).

(4-2) Set counter i=1 and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process.

Hyperparameters of the training process include the maximum number of loops, batch size bs, learning rate and optimizer.

Specifically, the batch size bs is set to 32, the initial learning rate is set to 0.00001, and the optimizer is set to the AdamW optimizer.

(4-3) Obtain the true values of multiple images and their corresponding semantic masks from the training set of the ADE20K data set obtained in step (1).

(4-4) Perform data preprocessing on the true values of multiple images and semantic masks obtained in step (5-3) to obtain multiple preprocessed images and true values of semantic masks.

Specifically, the preprocessing process of this step includes unified scale interpolation, random horizontal flipping, random cropping, and normalization. The dimensions of the multiple preprocessed images are bs×3×h×w, which correspond to The dimension of the preprocessed semantic mask is bs×1×h×w, and bs is the batch size set in step (4-2).

(4-5) Use the sequentially connected image feature extractor and discrete feature number classifier to map the multiple preprocessed images obtained in step 4-4) into a probability matrix of multiple discrete feature numbers, with the dimension

This step includes the following sub-steps:

(4-5-1) Input each preprocessed image obtained in step (4-4) into the image feature extractor to obtain image features of four different spatial scales and channel numbers corresponding to each image.

(4-5-2) Input the image features of four different spatial scales and channel numbers corresponding to each image obtained in step (4-5-1) into the discrete feature number classifier to obtain the probability matrix of the discrete feature number .

(4-6) Input the probability matrices of multiple discrete feature numbers obtained in step (4-5) into the discrete feature code table to obtain multiple discrete feature matrices, the dimensions of which are

(4-7) Input the multiple discrete feature matrices obtained in step (4-6) into the initialized semantic image decoder obtained in step (6-1) to obtain multiple predicted semantic images whose dimensions It is bs×3×h×w.

(4-8) Input the multiple predicted semantic images obtained in step (4-7) into the learnable color block-category mapping module to obtain multiple predicted pixel category probability matrices, whose dimensions are bs×C ×h×w. Among them, C is the total number of categories.

(4-9) Input the multiple predicted pixel category probability matrices obtained in step (4-8) and the true values of the multiple preprocessed semantic masks obtained in step (4-4) into the cross entropy loss function, To get the pixel classification loss value.

(4-10) Use the pixel classification loss value obtained in step (4-9) to perform backpropagation to obtain the gradient of the learnable color block-category mapping module.

(4-11) Utilize the learning rate set in step (4-2), the AdamW optimizer and the gradient obtained in step (4-10) to update the weight of the learnable color block-category mapping module to obtain a new learnable The weight of the color block-category mapping module.

(4-12) Counter i=i+1, and set the learnable color block-category mapping module to 0.

(4-13) Determine whether i is greater than the preset maximum number of iterations n, if so, go to step (4-14), otherwise return to step (4-3).

Specifically, the preset maximum number of cycles n ranges from 1,000 to 1,000,000, preferably 40,000.

(4-14) Save the weight of the learnable color block-category mapping module to obtain the weight of the trained learned color block-category mapping module.

(5) The weights of the trained image feature extractor and discrete feature number classifier obtained in step (3), and the weights of the trained learnable color block-category mapping module obtained in step (4) are obtained from the network. The downloaded discrete feature code table and the weights of the semantic image decoder are saved to obtain the weights of the semantic segmentation network model based on category colorization technology. Among them, the weight of the discrete feature code table and the weight of the semantic image decoder are respectively the weight of the discrete feature code table and the weight of the decoder of DALL-E's VQ-VAE model downloaded from the Internet.

It is easy for those skilled in the art to understand that the above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements, etc., made within the spirit and principles of the present invention, All should be included in the protection scope of the present invention.

Claims (6)

1. A semantic segmentation network model based on category colorization technology, including a sequentially connected image feature extractor, a discrete feature serial number classifier, a discrete feature code table, a semantic image decoder, and a learnable color block-category mapping module. Auxiliary pixel classifier, unlabeled region completion module, category-color mapping module, and semantic image encoder, characterized by: The image feature extractor is a Swin Transformer network, which includes 24 Swin Transformer modules connected in sequence. It is used to receive images with input dimensions of bs×3×h×w. The output is four feature matrices. The dimensions of the feature matrix are and Among them, bs is the batch data size preset during the offline training process, h and w are the number of pixels on the long side and the number of pixels on the short side of the image respectively; The discrete feature serial number classifier includes a feature aggregation layer, a feature processing layer and a classifier module; The feature aggregation layer is a convolution module whose input is the four feature matrices output by the Swin Transformer network, and the output is an aggregated feature matrix whose dimensions are The feature processing layer consists of two layers of Swin Transformer modules. Its input is the feature matrix aggregated by the feature aggregation layer. The output dimension is the feature matrix, whose dimensions are The input of the classifier module is the feature matrix output by the feature processing layer, and the output is a probability matrix whose dimensions are Each element in the probability matrix is the probability that 8192 discrete features appear in that element; The discrete feature code table is a collection of multiple discrete features. The dimension of each discrete feature is 128. Each discrete feature has a unique serial number in the discrete feature code table, that is, the discrete feature serial number; the discrete feature code table is found The discrete feature sequence number corresponding to the discrete feature with the highest probability in each element of the probability matrix. The sequence numbers corresponding to all elements in the probability matrix form a sequence number matrix. Then, the corresponding sequence number is queried in the discrete feature code table based on each element in the sequence number matrix. The discrete features of , all discrete features corresponding to all elements constitute a discrete feature matrix; The input of the discrete feature code table is a probability matrix, and the output is a discrete feature matrix whose dimensions are The semantic image decoder specifically uses the decoder of DALL-E's VQ-VAE model. The input is the discrete feature matrix output from the discrete feature code table, and the dimension is The output is the predicted semantic image, whose dimension is bs×3×h×w; The learnable color patch-category mapping module is a single-layer Swin Transformer module; the input of the learnable color patch-category mapping module is the predicted semantic image with dimensions bs×3×h×w output by the semantic image decoder, The output is a pixel category probability matrix with dimensions bs×C×h×w; the learnable color block-category mapping module first traverses all pixels on the semantic image, targeting the pixel L(i,j ) is all the pixels in the square local area with the center point {L(p,q)|p∈[i-6,i+6],p∈[j-6,j+6]}, the learnable color block - The category mapping module predicts C category probabilities for its center point pixel L(i,j), where C is the total number of categories, i∈[1,h], j∈[1,w]; finally, all pixels in the semantic image are The category probabilities of L(i,j) are spliced according to the spatial position to obtain the predicted pixel category probability matrix; The auxiliary pixel classifier is a convolution module whose input is the aggregated feature matrix output by the feature aggregation layer of the discrete feature classifier, and its dimension is The output is an auxiliary semantic mask whose dimension is bs×1×h×w; The unlabeled region completion module is used to replace the unlabeled pixel regions in the ground truth of the semantic mask with the categories predicted by the auxiliary pixel classifier; the input of the unlabeled region completion module is the ground truth of the semantic mask from the data set. value and the auxiliary semantic mask output by the auxiliary pixel classifier, both of which have dimensions bs × 1 × h × w; the output of the unlabeled region completion module is the true value of the semantic mask with full pixel annotation, and its dimensions is bs×1×h×w; The category-color mapping module is used to obtain the color corresponding to each pixel according to its category. Its input is the true value of the full-pixel annotated semantic mask output by the unlabeled area completion module, and its dimension is bs×1×h. ×w, the output is the true value of the semantic image with dimensions bs×3×h×w; The semantic image encoder specifically uses the encoder of DALL-E's VQ-VAE model. Its input is the true value of the semantic image output by the category-color mapping module. The dimension is bs×3×h×w, and the output is the dimension of The true value of the equivalent characteristic matrix of . 2. A training method for a semantic segmentation network model based on category colorization technology according to claim 1, characterized in that it includes the following steps: (1) Obtain the ADE20K data set, divide the 25574 sets of images in the ADE20K data set and the true values of their corresponding semantic masks into a training set, and divide the 2000 sets of images in the ADE20K data set and the true values of their corresponding semantic masks. Divide into validation set; (2) Generate a category color array with dimension C×3; step (2) includes the following sub-steps: (2-1) Generate three one-dimensional arrays A _R , A _G , and A _B ; each element of the array is and/> And there are k1∈[1, the total number of elements in the array A _R ], k2∈[1, the total number of elements in the array A _G ], k3∈[1, the total number of elements in the array A _B ]; (2-2) Set counters k1=1, k2=1, k3=1, and initialize RGB color array A _RGB to be an empty array; (2-3) Determine whether k1 is greater than the preset maximum number of cycles J, whose value is equal to the total number of elements in the array A _R. If so, go to step (2-13), otherwise go to step (2-4) ; (2-4) Determine whether k2 is greater than the preset maximum number of cycles K, whose value is equal to the total number of elements in the array A _G. If so, go to step (2-3), otherwise go to step (2-5); (2-5) Determine whether k3 is greater than the preset maximum number of cycles Q, whose value is equal to the total number of elements in array A _B. If so, go to step (2-4), otherwise go to step (2-6); (2-6) Generate a random integer r _R between -15 and 15, and update the k1th element of the A _R array To get the k1th element/> of the updated A _R array (2-7) Generate a random integer r _G between -15 and 15, and update the k2th element of the A _G array To get the k2th element/> of the updated A _G array (2-8) Generate a random integer r _B between -15 and 15, and update the k3th element of the A _B array To get the k3th element/> of the updated A _B array (2-9) Add the k1th element of the updated A _R array obtained in step (2-6) The k2th element of the updated A _G array obtained in step (2-7)/> And the k3th element/> of the updated A _B array obtained in step (2-8) Make up a three-dimensional element/> and add three-dimensional elements/> Insert into the end of RGB color array A _RGB to get the updated RGB color array A _RGB ; (2-10) Set k1=k1+1 and return to step (2-5); (2-11) Set k2=k2+1 and return to step (2-4); (2-12) Set k3=k3+1 and return to step (2-3); (2-13) Obtain the first C three-dimensional elements in the updated RGB color array A _RGB obtained in step (2-9) to obtain the category color array; (3) Use the training set of the ADE20K data set obtained in step (1) to train the image feature extractor, discrete feature serial number classifier and auxiliary pixel classifier to obtain the trained image feature extractor and discrete feature serial number classification The weight of the extractor and the weight of the image feature extractor and discrete feature number classifier generated during multiple iterations; step (3) includes the following sub-steps: (3-1) Initialize the weights of the image feature extractor, discrete feature number classifier, auxiliary pixel classifier, category-color mapping module, semantic image encoder and discrete feature code table to obtain the initialized image feature extraction , discrete feature number classifier, auxiliary pixel classifier, category-color mapping module, semantic image encoder and discrete feature code table; (3-2) Set counter i=1 and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process; (3-3) Obtain the true values of multiple images and their corresponding semantic masks from the training set of the ADE20K data set obtained in step (1); (3-4) Perform data preprocessing on the true values of multiple images and semantic masks obtained in step (3-3) to obtain multiple preprocessed images and true values of semantic masks; (3-5) Use the sequentially connected image feature extractor and discrete feature number classifier to map the multiple preprocessed images obtained in step (3-4) into a probability matrix of multiple discrete feature numbers, with the dimension and multiple aggregated feature matrices, whose dimensions are/> (3-6) Input the multiple aggregated feature matrices obtained in step (3-5) into the auxiliary pixel classifier to obtain multiple auxiliary semantic masks, whose dimensions are bs×1×h×w ; (3-7) Input the true values of the multiple preprocessed semantic masks obtained in step (3-4) and the auxiliary semantic masks obtained in step (4-7) into the unlabeled area completion module , to obtain the true values of multiple full-pixel annotated semantic masks, whose dimensions are bs×1×h×w; (3-8) Use the category-color mapping module, semantic image encoder, and discrete feature code table to map the true values of multiple full-pixel semantic masks obtained in step (3-7) into multiple discrete feature number matrices true value; (3-9) Input the probability matrices of multiple discrete feature numbers obtained in step (3-5) and the true values of the multiple discrete feature number matrices obtained in step (4-7) into the cross entropy loss function to obtain the semantics Feature loss value; (3-10) Use the semantic feature loss value obtained in step (3-9) to perform backpropagation to obtain the gradient of the image feature extractor and discrete feature number classifier; (3-11) Use the learning rate set in step (3-2), the AdamW optimizer and the gradient obtained in step (3-9) to update the weights of the image feature extractor and discrete feature number classifier to obtain a new image The weights of the feature extractor and the discrete feature number classifier; (3-12) Counter i=i+1, and set the gradient of the image feature extractor and discrete feature number classifier to 0; (3-13) Determine whether i is an integer multiple of the preset model saving interval t. If so, save the weights of the image feature extractor and the discrete feature serial number classifier, and name them as the image feature extractor of the i-th iteration. and the weight of the discrete feature serial number classifier, otherwise go to step (3-14); (3-14) Determine whether i is greater than the preset maximum number of iterations n, if so, go to step (3-15), otherwise return to step (3-3); (3-15) Save the weights of the image feature extractor and discrete feature number classifier, and obtain the weights of the trained image feature extractor and discrete feature number classifier; (4) Use the training set of the ADE20K data set obtained in step (1), and one of the weights of the image feature extractor and discrete feature number classifier generated in multiple iterations obtained in step (3) to learn the The color block-category mapping module is trained to obtain a trained and learnable color block-category mapping module; (5) The weights of the trained image feature extractor and discrete feature number classifier obtained in step (3), and the weights of the trained learnable color block-category mapping module obtained in step (4) are obtained from the network. The downloaded discrete feature code table and the weight of the semantic image decoder are saved to obtain the weight of the semantic segmentation network model based on category colorization technology; where the weight of the discrete feature code table and the weight of the semantic image decoder are obtained from The weights of the discrete feature code table and the decoder of DALL-E's VQ-VAE model downloaded from the Internet. 3. The training method of semantic segmentation network model based on category colorization technology according to claim 2, characterized in that step (3-1) includes the following sub-steps: (3-1-1) Load the pre-trained weights of the image feature extractor into the image feature extractor; (3-1-2) Initialize the weight of the discrete feature serial number classifier to a random value, and set the weight of the discrete feature serial number classifier to a gradient weight, that is, the weight of the image feature extractor will be optimized during the offline training process; (3-1-3) Initialize the weight of the auxiliary pixel classifier to a random value, and set the weight of the pixel classifier to a gradient weight; (3-1-4) Load the preset category color array obtained in step (2) into the category-color mapping module, and set the weight of the category-color mapping module to a gradient-free weight; (3-1-5) Add the pre-trained weights of the semantic image encoder to the semantic image encoder, and set the weights of the semantic image encoder to gradient-free weights; (3-1-6) Load the discrete signature table and training weights into the discrete signature table, and set the weight of the discrete signature table to a gradient-free weight. 4. The training method of the semantic segmentation network model based on category colorization technology according to claim 3, characterized in that step (3-8) includes the following sub-steps: (3-8-1) Input the true values of multiple full-pixel annotated semantic masks obtained in step (3-7) into the category-color mapping module to obtain the true values of multiple semantic images; (3-8-2) Input the true values of multiple semantic images obtained in step (3-8-1) to the initialized semantic image encoder obtained in step (3-1-5) to obtain multiple equal values. The true value of the valence characteristic matrix; (3-8-3) For the true value of each equivalent characteristic matrix obtained in step (3-8-2), query the equivalent value in the initialized discrete characteristic code table obtained in step (3). The discrete features and their discrete feature serial numbers that are closest to the true value of the feature matrix are spliced together according to the spatial position of all the queried discrete feature serial numbers to obtain the true value of the discrete feature serial number matrix, whose dimensions are 5. The training method of the semantic segmentation network model based on category colorization technology according to claim 4, characterized in that step (4) includes the following sub-steps: (4-1) Initialize the weights of the image feature extractor, discrete feature number classifier, discrete feature code table, semantic image decoder and learnable color block-category mapping module to obtain the initialized image feature extractor , discrete feature number classifier, discrete feature code table, semantic image decoder and learnable color block-category mapping module; (4-2) Set counter i=1 and initialize the hyperparameters of the training process to obtain the initialized hyperparameters of the training process; (4-3) Obtain the true values of multiple images and their corresponding semantic masks from the training set of the ADE20K data set obtained in step (1); (4-4) Perform data preprocessing on the true values of multiple images and semantic masks obtained in step (5-3) to obtain multiple preprocessed images and true values of semantic masks; (4-5) Use the sequentially connected image feature extractor and discrete feature number classifier to map the multiple preprocessed images obtained in step 4-4) into a probability matrix of multiple discrete feature numbers, with the dimension (4-6) Input the probability matrices of multiple discrete feature numbers obtained in step (4-5) into the discrete feature code table to obtain multiple discrete feature matrices, the dimensions of which are (4-7) Input the multiple discrete feature matrices obtained in step (4-6) into the initialized semantic image decoder obtained in step (6-1) to obtain multiple predicted semantic images whose dimensions is bs×3×h×w; (4-8) Input the multiple predicted semantic images obtained in step (4-7) into the learnable color block-category mapping module to obtain multiple predicted pixel category probability matrices, whose dimensions are bs×C ×h×w; where C is the total number of categories; (4-9) Input the multiple predicted pixel category probability matrices obtained in step (4-8) and the true values of the multiple preprocessed semantic masks obtained in step (4-4) into the cross entropy loss function, To get the pixel classification loss value; (4-10) Use the pixel classification loss value obtained in step (4-9) to perform backpropagation to obtain the gradient of the learnable color block-category mapping module; (4-11) Utilize the learning rate set in step (4-2), the AdamW optimizer and the gradient obtained in step (4-10) to update the weight of the learnable color block-category mapping module to obtain a new learnable The weight of the color block-category mapping module; (4-12) Counter i=i+1, and set the learnable color block-category mapping module to 0; (4-13) Determine whether i is greater than the preset maximum number of iterations n, if so, go to step (4-14), otherwise return to step (4-3); (4-14) Save the weight of the learnable color block-category mapping module to obtain the weight of the trained learned color block-category mapping module. 6. The training method of semantic segmentation network model based on category colorization technology according to claim 5, characterized in that step (4-1) includes the following sub-steps: (4-1-1) Load the weight of the k-th iteration of the image feature extractor obtained in step (4) to the image feature extractor, and set the weight of the image feature extractor to a gradient-free weight; (4-1-2) Load the weight of the k-th iteration of the discrete feature number classifier obtained in step (4) to the image feature extractor, and set the weight of the discrete feature number classifier to a gradient-free weight; (4-1-3) Load the pre-trained weights of the discrete signature table into the discrete signature table, and set the weight of the discrete signature table to a gradient-free weight; (4-1-4) Load the pre-trained weights of the semantic image decoder into the semantic image decoder, and set the weights of the semantic image decoder to gradient-free weights; (4-1-5) Initialize the weight of the learnable color block-category mapping module to a random value, and set the weight of the learnable color block-category mapping module to a gradient weight.