CN114863348B - Video target segmentation method based on self-supervision - Google Patents

Abstract

The invention discloses a video target segmentation method based on self-supervision, which mainly solves the problems of lower segmentation precision and larger influence of target shielding and tracking drift in the prior art. The scheme comprises the following steps: 1) Acquiring a video sequence from a video target segmentation data set, preprocessing the video sequence, and dividing the video sequence to obtain a training, verifying and testing sample set; 2) Constructing and training an image reconstruction neural network model, and extracting target features by adopting a self-supervision learning method based on a multi-pixel scale image reconstruction task; 3) Building and training a side output edge detection network model; 4) Constructing and training an edge correction network model based on self supervision; 5) Combining the three trained models to obtain a video target segmentation model; 6) And inputting the test set into a video target segmentation model to obtain a target segmentation result. The video object segmentation method can effectively improve generalization and accuracy of video object segmentation, and can be used in the fields of automatic driving, intelligent monitoring, unmanned aerial vehicle intelligent tracking and the like.

Description

Video object segmentation method based on self-supervision

Technical Field

The present invention belongs to the field of computer vision technology, and further relates to a video target segmentation technology, specifically a video target segmentation method based on self-supervision, which can be used in the fields of autonomous driving, intelligent monitoring, and drone intelligent tracking.

Background Art

Computer vision aims to imitate the process of human visual perception and is a key link in the development of artificial intelligence technology. Computer vision algorithms seek to simulate human visual behavior to the greatest extent possible with higher accuracy and provide perceptual information for downstream tasks. In the human perception system, image transformation is a continuous process. In the current context of visual technology, the image storage method that is closest to human perception is video. Therefore, computer vision algorithms that process video tasks are more capable of simulating human visual behavior.

Video target segmentation is an important topic in video processing tasks, and its purpose is to segment the target of interest from the background in a series of video sequences. In recent years, due to the excellent performance of deep learning technology in computer vision tasks (such as image recognition, target tracking, action recognition, etc.), the video target segmentation algorithm based on deep learning has become the mainstream method for solving video target segmentation tasks. The performance of the video target segmentation algorithm based on deep learning depends on the scale of the neural network it uses. The performance of the neural network depends on a large amount of training data. The larger the scale of the training data set, the better the generalization and robustness of the trained neural network. Under the paradigm of supervised learning, the production process of the video target segmentation training data set is costly and time-consuming. It is necessary not only to annotate every pixel in the image in space, but also to annotate every frame in the video sequence in time. The performance of the video target segmentation model is also closely related to the structure. By rationally optimizing the inference process of the video target segmentation model, the errors in the video target segmentation process can be effectively reduced.

The research goal of self-supervised learning is to train deep learning models without any manual annotation, so that they can extract effective visual representation information from a large number of unlabeled pictures or video data sets, and the extracted information is fine-tuned for use in downstream tasks. Video object segmentation based on self-supervised learning is designed for the specific task of semi-supervised video object segmentation. The video object segmentation model is trained through self-supervised learning methods. The trained model can be directly used for video object segmentation tasks, and no manual annotation data set is required during the entire training process.

The research on self-supervised video target segmentation methods can be basically divided into two lines. One is to design a better pre-task training model to enable the model to have better representation extraction capabilities; the other is to introduce more mechanisms to reduce the impact of target occlusion and tracking drift for the problem of semi-supervised video target segmentation. In 2018, Colorizer et al. published an article entitled "Tracking emerges by colorizing videos" in the European Conference on Computer Vision. They proposed a self-supervised video tracking colorization model. The model uses the natural temporal coherence of color to learn to color grayscale videos, further improving the effect of self-supervised video tracking technology. However, since it is based on the propagation of the previous frame, it is less robust to target occlusion and tracking drift. In 2019, Corrflow et al. published an article entitled "Self supervised video representation learning for correspondence flow" at the British Machine Vision Conference. By introducing a restricted attention mechanism, the resolution of the model input is improved without increasing the burden on the computing device, and the accuracy of segmentation is improved. However, this method does not consider the generalization of feature extraction of targets of different scales, and performs poorly when the target scales differ too much.

Summary of the invention

The purpose of the present invention is to propose a video target segmentation method based on self-supervision in view of the above-mentioned deficiencies in the prior art, so as to solve the technical problems existing in the prior art such as low segmentation accuracy, large influence of target occlusion and tracking drift.

The idea of implementing the present invention is: first, a self-supervised learning method based on a multi-pixel scale image reconstruction task is used to extract target features, so that the video target segmentation model can take into account the features of both large targets and small targets to obtain better generalization performance. Then, in view of the error accumulation problem generated by the video target segmentation model when performing target segmentation, it is proposed to use image semantic edges to correct the target segmentation mask; finally, a self-supervised edge fusion network is designed to obtain a more accurate target segmentation mask. Compared with the traditional self-supervised video target segmentation method, the present invention effectively improves the generalization and accuracy of video target segmentation.

To achieve the above object, the technical solution adopted by the present invention includes the following steps:

(1) Obtain training sample set, verification sample set and test sample set:

Obtain a video sequence from a video target segmentation dataset, perform preprocessing, obtain a frame sequence set V, divide the frame sequences in the set, and obtain a training sample set V _train , a verification sample set V _val , and a test sample set V _test ;

(2) Build and train the image reconstruction neural network model R:

(2a) constructing an image reconstruction neural network model R consisting of a feature extraction network, wherein the feature extraction network adopts a residual network including multiple convolutional layers, multiple pooling layers, multiple residual unit modules and a single fully connected layer connected in sequence;

(2b) Define the loss function of the image reconstruction neural network model R:

_Lmix ＝ _αLcls +(1-α) _Lreg

in, Represents the cross entropy loss function for the quantitative image reconstruction task, for the training sample set Select E cluster centroids {μ ₁ ,μ ₂ ,...,μ _E }, and E≤50; calculate the category to which the sample belongs based on the distance between the training sample and the cluster centroid, and set the number of target categories contained in the frame sequence set V to be C; modify the position of the cluster centroid so that the labels of the same target in the frame are the same, and the labels of different targets are different, where, Indicates the category to which the i-th pixel of a given frame image I _t belongs, represents the prediction result using the K-means algorithm, L _reg represents the regression loss function of the RGB image reconstruction task, in, in is the real target frame pixel, To reconstruct the target frame pixels, α represents the weight coefficient, 0.1≤α≤0.9;

(2c) setting feature extraction network parameters and a maximum number of iterations N, iteratively training the image reconstruction neural network model R according to the loss function of the image reconstruction neural network model R and using the target frame images in the training sample set V _train to obtain a trained image reconstruction neural network model R;

(3) Build and train the side output edge detection network model Q:

(3a) constructing an edge detection network model Q including a side output edge detection layer SODL and a side output edge fusion layer SOFL connected in sequence, wherein the side output edge detection layer SODL includes a deconvolution layer and a convolution layer with a convolution kernel size of 1×1 and an output channel number of 1, and the side output edge fusion layer SOFL is a convolution layer with a convolution kernel size of 1×1 and a channel number of 1;

(3b) Define the loss function of the side output edge detection network model Q:

L _edge = L _side + L _fuse

Among them, L _side represents the side output edge detection loss function, Among them, β _i represents the weight coefficient of the i-th side output edge detection network, The loss function of the prediction result of the i-th side output edge detection network is shown as:

in, Among them, e represents the true value of the target edge of the input image, |e ^- | represents the number of pixels that are edges in the true value of the target edge of the image, |e ⁺ | represents the number of pixels that are not edges in the true value of the target edge of the image, ω _i represents the parameters of the convolution layer, and L _fuse represents the edge fusion loss function:

(3c) setting a maximum number of iterations I, iteratively training the side output edge detection network model Q according to the loss function of the side output edge detection network model Q and using the feature map set output by each structural layer of the feature extraction network in the image reconstruction neural network model R, and obtaining a trained side output edge detection network model Q;

(4) Build and train the edge correction network model Z:

(4a) sequentially connecting the dilated spatial convolutional pooling pyramid model _Fγ and the softmax activation function output layer, wherein the dilated spatial convolutional pooling pyramid model _Fγ is composed of a plurality of sequentially connected convolutional layers and pooling layers, and obtaining an edge correction network model Z;

(4b) Define the loss function of the edge correction network model Z:

in, The rough segmentation result of the target frame output by the edge detection layer, is the prediction result of the dilated spatial convolutional pooling pyramid model F _γ , in, Represents the image edge obtained by the Canny algorithm, and M represents the mask The number of categories of pixels in , Indicates mask The total number of pixels in

(4c) setting a maximum number of iterations H, iteratively training the edge correction network model Z according to the loss function of the edge correction network model Z and using the output results of the image reconstruction network model R and the edge detection network model Q to obtain a trained edge correction network model Z;

(5) The video target segmentation model based on the image target edge correction segmentation result is obtained by combining the trained image reconstruction neural network R, the side output edge detection network Q and the edge correction network model Z;

(6) Obtain self-supervised video target segmentation results:

The test set The frame images in are used as the input of the video target segmentation model for forward propagation to obtain the predicted segmentation labels of all test frame images, and the final segmentation result image is obtained based on the predicted segmentation labels of the test frame images.

Compared with the prior art, the present invention has the following advantages:

First, since this paper uses the reconstruction task of multi-pixel scale images as the prerequisite task of self-supervised learning, the features extracted by the trained model have better generalization for large and small targets in the video segmentation task, thus having better performance in the overall video target segmentation task.

Second, the present invention uses edge restoration of targets in video images to repair target masks, integrates feature maps extracted from each layer of feature extraction networks in video target segmentation models through a side output edge detection network, predicts candidate target edges in target frames, and uses a self-supervised edge fusion model to fuse the segmentation results output by the video target segmentation model with the target edges output by the side output edge detection network, thereby correcting the segmentation mask according to the target edges to obtain more accurate segmentation results.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of the implementation of the present invention.

DETAILED DESCRIPTION

The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Embodiment 1: Referring to FIG. 1 , a video object segmentation method based on self-supervision proposed by the present invention specifically comprises the following steps:

Step 1: Get the training sample set, validation sample set, and test sample set:

Obtain a video sequence from the video target segmentation dataset and perform preprocessing to obtain a frame sequence set V. The frame sequences in the set are divided to obtain a training sample set V _train , a verification sample set V _val and a test sample set V _test . The implementation is as follows:

(1a) Obtain S multi-category video sequences from the video object segmentation dataset and obtain a frame sequence set after preprocessing S≥3000; represents the kth frame sequence consisting of preprocessed image frames, represents the nth image frame in the kth frame sequence, M ≥ 30;

(1b) Randomly select more than half of the frame sequences from the frame sequence set V to form a training sample set Where S/2＜N＜S, for each frame sequence in the training sample set Each target frame image to be segmented Scale the image blocks to p×p×h size and convert the image format from RGB to Lab; extract half of the frame sequences from the remaining frame sequences to form a verification sample set Where J≤S/4; the other half constitutes the test sample set T≤S/4, and convert the image format from RGB to Lab.

Step 2: Build and train the image reconstruction neural network model R:

(2a) constructing an image reconstruction neural network model R consisting of a feature extraction network, wherein the feature extraction network adopts a residual network including multiple convolutional layers, multiple pooling layers, multiple residual unit modules and a single fully connected layer connected in sequence;

(2b) Define the loss function of the image reconstruction neural network model R:

_Lmix ＝ _αLcls +(1-α) _Lreg

in, Represents the cross entropy loss function for the quantitative image reconstruction task, for the training sample set Select E cluster centroids {μ ₁ ,μ ₂ ,...,μ _E }, and E≤50; calculate the category to which the sample belongs based on the distance between the training sample and the cluster centroid, and set the number of target categories contained in the frame sequence set V to be C; modify the position of the cluster centroid so that the labels of the same target in the frame are the same, and the labels of different targets are different, where, Indicates the category to which the i-th pixel of a given frame image I _t belongs, represents the prediction result using the K-means algorithm, L _reg represents the regression loss function of the RGB image reconstruction task, in, in is the real target frame pixel, To reconstruct the target frame pixels, α represents the weight coefficient, 0.1≤α≤0.9;

(2c) Setting the feature extraction network parameters and the maximum number of iterations N, iteratively training the image reconstruction neural network model R according to the loss function of the image reconstruction neural network model R and using the target frame images in the training sample set V _train to obtain the trained image reconstruction neural network model R; the implementation is as follows:

(2c1) Let the network hyperparameter of the feature extraction network be θ, the maximum number of iterations be N ≥ 150000, n represents the current number of iterations; let n = 1, the number of initial iterations;

(2c2) The target frame images in the training sample set V _train are used as the input of the image reconstruction neural network model R for forward propagation:

For each target frame I _t to be segmented, select the q frames in front of it as reference frames {I' ₀ , I ₁ ', ..., I' _q }, where 2≤q≤5, the target frame I _t and its corresponding reference frame set are used as inputs of the feature extraction network Φ(.; θ), the feature extraction network extracts features from I _t and each of its reference frame images, and obtains the target frame image feature f _t =Φ(I _t ; θ), the reference frame image feature f′ ₀ =Φ(I′ ₀ ; θ), ..., f′ _q =Φ(I′ _q ; θ), the target frame {I _t |0≤t≤N} in the training sample set is used as the input of the K-means algorithm, and the quantized image reconstruction loss value L _cls is obtained. The reconstructed target frame I _t and the real target frame I _t are used as the input of the RGB image reconstruction task, and the RGB image reconstruction loss value L _reg is obtained;

(2c3) Using the loss function L _mix , the loss value of the image reconstruction neural network is calculated through the cross entropy loss L _cls and the regression loss L _reg Back propagation is used to calculate the network parameter gradient g(θ), and then the network parameter θ is updated by the gradient descent method;

(2c4) Determine whether n=N. If so, obtain the trained image reconstruction neural network R; otherwise, set n=n+1 and return to step (2c2).

Step 3: Build and train the side output edge detection network model Q:

(3a) constructing an edge detection network model Q including a side output edge detection layer SODL and a side output edge fusion layer SOFL connected in sequence, wherein the side output edge detection layer SODL includes a deconvolution layer and a convolution layer with a convolution kernel size of 1×1 and an output channel number of 1, and the side output edge fusion layer SOFL is a convolution layer with a convolution kernel size of 1×1 and a channel number of 1;

(3b) Define the loss function of the side output edge detection network model Q:

L _edge = L _side + L _fuse

Among them, L _side represents the side output edge detection loss function, Among them, β _i represents the weight coefficient of the i-th side output edge detection network, Represents the loss function of the prediction result of the i-th side output edge detection network:

in, Among them, e represents the true value of the target edge of the input image, |e ^- | represents the number of pixels that are edges in the true value of the target edge of the image, |e ⁺ | represents the number of pixels that are not edges in the true value of the target edge of the image, ω _i represents the parameters of the convolution layer, and L _fuse represents the edge fusion loss function:

(3c) Set the maximum number of iterations I, and iteratively train the side output edge detection network model Q according to the loss function of the side output edge detection network model Q and the feature map set output by each structural layer of the feature extraction network in the image reconstruction neural network model R to obtain the trained side output edge detection network model Q, which is implemented as follows:

(3c1) Assume that the maximum number of iterations I≥150000, the current number of iterations is i; let i=1, and initialize the number of iterations;

(3c2) The feature map set output by each structural layer of the feature extraction network in the image reconstruction network model Perform forward propagation as input to the side output edge detection network:

(3c3) The output edge detection layer obtains the rough edge of the target from the feature map set, thereby obtaining the rough edge corresponding to each feature map

(3c4) The rough edge set output by the side output edge detection layer SODL is used as the input of the side output edge fusion layer SOFL, and the rough edges are weighted and fused to obtain the final predicted edge. in, represents the feature map formed by merging coarse edges, ω _fuse represents the parameters of the side output edge fusion layer;

(3c5) Using the loss function L _edge , the loss value of the edge detection network is calculated by the side output edge detection loss L _side and the side output edge fusion loss L _fuse Back propagation is used to calculate the network parameter gradient g(ω), and then the network parameter ω is updated by the gradient descent method;

(3c6) Determine whether i=I holds. If so, obtain the trained side output edge detection network model Q; otherwise, set i=i+1 and return to execute step (3c2).

Step 4: Build and train the edge correction network model Z:

(4a) sequentially connecting the dilated spatial convolutional pooling pyramid model _Fγ and the softmax activation function output layer, wherein the dilated spatial convolutional pooling pyramid model _Fγ is composed of a plurality of sequentially connected convolutional layers and pooling layers, and obtaining an edge correction network model Z;

(4b) Define the loss function of the edge correction network model Z:

in, The rough segmentation result of the target frame output by the edge detection layer, is the prediction result of the dilated spatial convolutional pooling pyramid model F _γ , in, Represents the image edge obtained by the Canny algorithm, and M represents the mask The number of categories of pixels in , Indicates mask The total number of pixels in

(4c) Set the maximum number of iterations H, and iteratively train the edge correction network model Z according to the loss function of the edge correction network model Z and the output results of the image reconstruction network model R and the edge detection network model Q to obtain the trained edge correction network model Z, which is implemented as follows:

(4c1) Assume that the maximum number of iterations is H ≥ 150000, and the current number of iterations is h; let h = 1, and initialize the number of iterations;

(4c2) The target frame rough segmentation result output by the image reconstruction network model R and the edge detection result output by the edge detection network model Q are used as the input of the edge correction network model Z for forward propagation:

(4c2.1) The edge correction network first roughly segments the target frame And edge detection results Merge in the channel dimension to obtain a feature map of size H×W×(K+1)

(4c2.2) The feature map As the input of the dilated spatial convolutional pooling pyramid model F _γ , the prediction result of the expanded receptive field is obtained;

(4c2.3) The predicted result of the expanded receptive field is used as the input of the output layer of the softmax activation function. The segmentation label of the pixel is determined according to the probability that each pixel in the feature map belongs to each category, so as to obtain a more accurate target segmentation mask after edge fusion correction of the target frame target segmentation mask. Where O _t represents the predicted segmentation label of the target frame I _t ;

(4c3) Use the loss function L _corr to calculate the loss value of the edge correction network Back propagation is used to calculate the network parameter gradient g(c), and then the network parameter c is updated by the gradient descent method;

(4c4) Determine whether h=H holds. If so, obtain the trained edge correction network model Z; otherwise, set h=h+1 and return to execute step (4c2).

Step 5: The trained image reconstruction neural network R, the side output edge detection network Q and the edge correction network model Z are combined to obtain a video target segmentation model based on the image target edge correction segmentation result. Specifically, the combination is performed in the following manner: the intermediate feature map extracted by the image reconstruction neural network R is input into the side output edge detection network Q to obtain a target edge prediction map. The target segmentation mask prediction map output by the image reconstruction neural network R and the target edge prediction map output by the side output edge detection network Q are used as inputs of the edge correction network model Z to obtain a trained video target segmentation model based on the image target edge correction segmentation result.

Step 6: Obtain self-supervised video object segmentation results:

The test set The frame images in are used as the input of the video target segmentation model for forward propagation to obtain the predicted segmentation labels of all test frame images, and the final segmentation result image is obtained based on the predicted segmentation labels of the test frame images.

Embodiment 2: The overall steps of this embodiment are the same as those of Embodiment 1, and specific values are given for the settings of some parameters to further describe the implementation process of the present invention:

Step 1) Obtain training sample set, validation sample set and test sample set:

Step 1a) Obtain S multi-category video sequences from a video object segmentation dataset, and obtain a frame sequence set V after preprocessing. In this embodiment, the multi-category video sequences are obtained from a YouTube VOS dataset, S=4453, M=50;

Step 1b) Assume that the number of target categories contained in the frame sequence V is C=94, and the category set Class={c _num ,1≤num≤C}. Multiple categories of targets may appear in each frame sequence, where c _num represents the numth category of target;

Step 1c) randomly select more than half of the frame sequences from the frame sequence set V to form a training sample set Among them, S/2＜N＜S, for each frame sequence in the training set Each target frame image to be segmented Scale the image into a size of p×p×h, convert the RGB image into a Lab image, and extract half of the frame sequence from the remaining frame sequence to form a verification sample set Among them, J≤S/4, and the other half constitutes the test sample set T≤S/4. Similarly, convert the original image from RGB format to Lab format;

Set the crop box size to x×y for the frame sequence in the training set For each frame image with segmentation Crop to get the cropped frame image For the cropped frame image Normalization is performed, and the normalized frame images constitute the preprocessed training frame sequence in, is the mth frame sequence in the training sample set;

In this embodiment, x=256, y=256, p=256, h=3;

Step 2) Construct the image reconstruction neural network model R:

Step 2a) Construct the structure of the image reconstruction neural network model R:

Constructing an image reconstruction neural network model composed of a feature extraction network, wherein the feature extraction network adopts a residual network including a plurality of convolutional layers, a plurality of pooling layers, a plurality of residual unit modules and a single fully connected layer connected in sequence;

The feature extraction network contains 17 convolutional layers and 1 fully connected layer. The 18-layer structure is divided into 5 blocks, namely conv_1, conv_2, conv_3, conv_4, and conv_5. Conv_1 is a convolutional layer with a convolution kernel size of 7×7 and 64 channels. Conv_2 contains two convolutional layers with a convolution kernel size of 3×3 and 64 channels. The convolution kernel moving distance is 1. Conv_3 contains two convolutional layers with a convolution kernel size of 3×3 and 128 channels. The convolution kernel moving distance of the first convolution layer is set to 2, and the convolution kernel moving distance of the second convolution layer is 1. Conv_4 contains two convolutional layers with a convolution kernel size of 3×3 and 256 channels. The convolution kernel moving distance is 1. Conv_5 contains two convolutional layers with a convolution kernel size of 3×3 and 512 channels. The convolution kernel moving distance is 1.

Step 2b) Define the loss function of the image reconstruction neural network model:

_Lmix ＝ _αLcls +(1-α) _Lreg

Among them, L _cls represents the cross entropy loss function of the quantitative image reconstruction task, For the training sample set Select E cluster centroids {μ ₁ ,μ ₂ ,...,μ _E }, where E≤50, calculate the category of the sample based on the distance between the training sample and the cluster centroid, and correct the location of the cluster centroid so that the labels of the same target in the same frame are the same, and the labels of different targets are different, where Indicates the category to which the i-th pixel of a given frame image I _t belongs, represents the prediction result using the K-means algorithm, L _reg represents the regression loss function of the RGB image reconstruction task, in, in is the real target frame pixel, To reconstruct the target frame pixels, α represents the weight coefficient, 0.1≤α≤0.9;

In this embodiment, K=16, α=0.6;

Step 3) Iteratively train the image reconstruction neural network model:

Step 3a) The network hyperparameter of the feature extraction network is θ, the number of initialization iterations is n, the maximum number of iterations is N, N ≥ 150000, and n = 1;

In this embodiment, N=300000 is designed to make the model training more sufficient;

Step 3b) Use the target frame image in the training sample set V _train as the input of the image reconstruction neural network model R for forward propagation:

Step 3b1) For each target frame I _t to be segmented, select the q frames preceding it as reference frames {I' ₀ , I ₁ ', ..., I _q '}, where 2≤q≤5, the target frame I _t and its corresponding reference frame set are used as inputs of the feature extraction network Φ(.;θ), the feature extraction network performs feature extraction on I _t and each of its reference frame images, and obtains target frame image features f _t =Φ(I _t ;θ), reference frame image features f′ ₀ =Φ(I′ ₀ ;θ), ..., f′ _q =Φ(I′ _q ;θ), the target frames {I _t |0≤t≤N} in the training sample set are used as inputs of the K-means algorithm, and the quantized image reconstruction loss value L _cls is obtained, the reconstructed target frame I _t and the real target frame I _t are used as inputs of the RGB image reconstruction task, and the RGB image reconstruction loss value L _reg is obtained;

Step 3c) Use the loss function L _mix to calculate the loss value of the image reconstruction neural network through the cross entropy loss L _cls and the regression loss L _reg Back propagation is used to calculate the network parameter gradient g(θ), and then the network parameter θ is updated by the gradient descent method. The update formula is:

Among them, θ′ represents the result after θ ⁿ is updated, γ represents the learning rate, 1e-6≤γ≤1e-3, represents the loss function value of the image reconstruction neural network after the nth iteration, Represents partial derivative calculation.

In this embodiment, the initial learning rate γ=0.001, when the iteration reaches the 150,000th time, the learning rate γ=0.0005, when the iteration reaches the 200,000th time, the learning rate γ=0.00025, when the iteration reaches the 250,000th time, the learning rate γ=0.000125, the optimizer function uses the Adam optimizer, and the learning rate is decayed when the network iterates to a certain number of times in order to prevent the loss function from falling into a local minimum;

Step 3d) Determine whether n=N is established. If so, obtain the trained image reconstruction neural network R. Otherwise, set n=n+1 and execute step (3b);

Step 4) Build the side output edge detection network model Q:

Step 4a) Construct the structure of the side output edge detection network model Q:

Construct an edge detection network model Q including a side output edge detection layer SODL and a side output edge fusion layer SOFL connected in sequence, wherein the side output edge detection layer SODL includes a deconvolution layer and a convolution layer with a convolution kernel size of 1×1 and an output channel number of 1, and the side output edge fusion layer SOFL is a convolution layer with a convolution kernel size of 1×1 and a channel number of 1;

Step 4b) Define the loss function of the side output edge detection network model:

L _edge = L _side + L _fuse

Among them, L _side represents the side output edge detection loss function, Among them, β _i represents the weight coefficient of the i-th side output edge detection network, Represents the loss function of the prediction result of the i-th side output edge detection network:

in, Among them, e represents the true value of the target edge of the input image, |e ^- | represents the number of pixels that are edges in the true value of the target edge of the image, |e ⁺ | represents the number of pixels that are not edges in the true value of the target edge of the image, ω _i represents the parameters of the convolution layer, and L _fuse represents the edge fusion loss function:

Step 5) Iteratively train the side output edge detection network model Q:

Step 5a) Initialize the number of iterations to i, the maximum number of iterations to I, I≥150000, let i=1;

In this embodiment, in order to make the model training more sufficient, I=300000;

Step 5b) The feature map set output by each structural layer of the feature extraction network in the image reconstruction network model Perform forward propagation as input to the side output edge detection network:

Step 5b1) The side output edge detection layer obtains the rough edge of the target from the feature map set, thereby obtaining the rough edge corresponding to each feature map

Step 5b2) The rough edge set output by the side output edge detection layer SODL is used as the input of the side output edge fusion layer SOFL, and the rough edges are weighted and fused to obtain the final predicted edge. in, represents the feature map formed by merging coarse edges, ω _fuse represents the parameters of the side output edge fusion layer;

Step 5c) Use the loss function L _edge to calculate the loss value of the edge detection network through the side output edge detection loss L _side and the side output edge fusion loss L _fuse Back propagation is used to calculate the network parameter gradient g(ω), and then the network parameter ω is updated by the gradient descent method. The update formula is:

Among them, ω′ represents the result after ω ⁱ is updated, β represents the learning rate, 1e-6≤β≤1e-3, represents the loss function value of the output edge detection neural network after the i-th iteration, Represents partial derivative calculation.

In this embodiment, the initial learning rate β=0.001, when the iteration reaches the 150,000th time, the learning rate β=0.0005, when the iteration reaches the 200,000th time, the learning rate β=0.00025, when the iteration reaches the 250,000th time, the learning rate β=0.000125, the optimizer function uses the Adam optimizer, and the learning rate is decayed when the network iterates to a certain number of times in order to prevent the loss function from falling into a local minimum;

Step 5d) Determine whether i=I is established. If so, obtain the trained side output edge detection network model Q. Otherwise, set i=i+1 and execute step (5b);

Step 6) Construct edge correction network model Z:

Step 6a) Construct the structure of the edge correction network model Z:

Constructing an edge correction network model Z including a sequentially connected atrous spatial convolutional pooling pyramid model _Fγ and a softmax activation function output layer, wherein the atrous spatial convolutional pooling pyramid model _Fγ is composed of a plurality of sequentially connected convolutional layers and pooling layers;

The dilated spatial convolutional pooling pyramid model _Fγ includes a convolutional layer, a pooling pyramid, and a pooling block, wherein the convolution kernel size of the convolutional layer is 1×1, the pooling pyramid includes three parallel connected convolutional layers, whose convolution kernel size is 3×3, and the pooling block includes a 1×1 pooling layer, a convolution layer, and an upsampling operation layer, whose convolution kernel size is 1×1. The feature maps output by the convolutional layer, the pooling pyramid, and the pooling block modules are concat-operated, and then passed through a 1×1 convolutional layer to obtain the output of the dilated spatial convolutional pooling pyramid model _Fγ .

Step 6b) Define the loss function of the edge correction network model Z:

in, The rough segmentation result of the target frame output by the edge detection layer, is the prediction result of the dilated spatial convolutional pooling pyramid model F _γ , in, Represents the image edge obtained by the Canny algorithm, and M represents the mask The number of categories of pixels in , Indicates mask The total number of pixels in

Step 7) Iteratively train the edge correction network model Z:

Step 7a) Initialize the number of iterations to h, the maximum number of iterations to H, H ≥ 150000, and set h = 1;

In this embodiment, H=300000 is designed to make the model training more sufficient;

Step 7b) The target frame coarse segmentation result output by the image reconstruction network model and the edge detection result output by the edge detection network model are used as the input of the edge correction network model Z for forward propagation:

Step 7b1) The edge correction network first roughly segments the target frame And edge detection results Merge in the channel dimension to obtain a feature map of size H×W×(K+1)

Step 7b2) Feature Map As the input of the dilated spatial convolutional pooling pyramid model F _γ , the prediction result of the expanded receptive field is obtained;

Step 7b3) The predicted result of the expanded receptive field is used as the input of the output layer of the softmax activation function. The segmentation label of the pixel is determined according to the probability of each category at each pixel position in the feature map, thereby obtaining a more accurate target segmentation mask of the target frame after edge fusion correction. Among them, O _t represents the predicted segmentation label of the target frame I _t ;

Step 7c) Use the loss function L _corr to calculate the loss value of the edge correction network Back propagation is used to calculate the network parameter gradient g(c), and then the network parameter c is updated by the gradient descent method. The update formula is:

Among them, c′ represents the result after ^ch is updated, α represents the learning rate, 1e-6≤α≤1e-3, represents the loss function value of the edge-corrected neural network after the hth iteration, Represents partial derivative calculation.

In this embodiment, the initial learning rate α=0.001, when the iteration reaches the 150,000th time, the learning rate α=0.0005, when the iteration reaches the 200,000th time, the learning rate α=0.00025, when the iteration reaches the 250,000th time, the learning rate α=0.000125, the optimizer function uses the Adam optimizer, and the learning rate is decayed when the network iterates to a certain number of times in order to prevent the loss function from falling into a local minimum;

Step 7d) Determine whether h=H holds. If so, obtain the trained edge correction network model Z. Otherwise, set h=h+1 and execute step (7b);

Step 8) Obtain the self-supervised video target segmentation results:

The test set The frame images in are used as the input of the trained video target segmentation model based on the image target edge correction segmentation result for forward propagation. The video target segmentation model based on the image target edge correction segmentation result is composed of an image reconstruction neural network R, a side output edge detection network Q and an edge fusion network Z. The segmentation labels of all test frame images are obtained, and the segmentation result map is determined according to the test frame image segmentation labels.

The following simulation experiments are used to further illustrate the technical effects of the present invention:

1. Simulation conditions and contents:

4453 video sequences were obtained from the YouTube-VOS dataset for simulation experiments;

The simulation experiment was conducted on a server with an Intel(R) Core(TM) i77800x CPU@3.5GHz 64GB CPU and an NVIDIA GeForce RTX 2080ti GPU. The operating system was UBUNTU 16.04, the deep learning framework was PyTorch, and the programming language was Python 3.6.

The present invention and an existing video target segmentation method are compared and simulated. In order to quantitatively compare the video target segmentation results, the experiment uses two video target segmentation result evaluation indicators, namely, region similarity J and contour similarity F. The higher the two evaluation indicators, the better the change detection result. The simulation results are shown in Table 1.

Table 1

2. Analysis of simulation results:

It can be seen from Table 1 that compared with the existing video segmentation method, the J index and F index of the present invention are significantly improved, indicating that the self-supervised video target segmentation technology constructed by the present invention can effectively solve problems such as target occlusion and tracking drift, thereby improving the video target segmentation accuracy, and therefore has important practical significance and practical value.

The above simulation analysis proves the correctness and effectiveness of the method proposed in the present invention.

Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It is obvious that for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims (6)

1. A video object segmentation method based on self-supervision, characterized in that it includes the following steps: (1) Obtain training sample set, verification sample set and test sample set: Obtain a video sequence from a video target segmentation dataset, perform preprocessing, obtain a frame sequence set V, divide the frame sequences in the set, and obtain a training sample set V _train , a verification sample set V _val , and a test sample set V _test ; (2) Build and train the image reconstruction neural network model R: (2a) constructing an image reconstruction neural network model R consisting of a feature extraction network, wherein the feature extraction network adopts a residual network including multiple convolutional layers, multiple pooling layers, multiple residual unit modules and a single fully connected layer connected in sequence; (2b) Define the loss function of the image reconstruction neural network model R: _Lmix ＝ _αLcls +(1-α) _Lreg in, Represents the cross entropy loss function for the quantitative image reconstruction task, for the training sample set Select E cluster centroids {μ ₁ ,μ ₂ ,...,μ _E }, and E≤50; calculate the category to which the sample belongs based on the distance between the training sample and the cluster centroid, and set the number of target categories contained in the frame sequence set V to be C; modify the position of the cluster centroid so that the labels of the same target in the frame are the same, and the labels of different targets are different, where, Indicates the category to which the i-th pixel of a given frame image I _t belongs, represents the prediction result using the K-means algorithm, L _reg represents the regression loss function of the RGB image reconstruction task, in, in is the real target frame pixel, To reconstruct the target frame pixels, α represents the weight coefficient, 0.1≤α≤0.9; (2c) setting feature extraction network parameters and a maximum number of iterations N, iteratively training the image reconstruction neural network model R according to the loss function of the image reconstruction neural network model R and using the target frame images in the training sample set V _train to obtain a trained image reconstruction neural network model R; (3) Build and train the side output edge detection network model Q: (3a) constructing an edge detection network model Q including a side output edge detection layer SODL and a side output edge fusion layer SOFL connected in sequence, wherein the side output edge detection layer SODL includes a deconvolution layer and a convolution layer with a convolution kernel size of 1×1 and an output channel number of 1, and the side output edge fusion layer SOFL is a convolution layer with a convolution kernel size of 1×1 and a channel number of 1; (3b) Define the loss function of the side output edge detection network model Q: L _edge = L _side + L _fuse Among them, L _side represents the side output edge detection loss function, Among them, β _i represents the weight coefficient of the i-th side output edge detection network, Represents the loss function of the prediction result of the i-th side output edge detection network: in, Among them, e represents the true value of the target edge of the input image, |e ^- | represents the number of pixels that are edges in the true value of the target edge of the image, |e ⁺ | represents the number of pixels that are not edges in the true value of the target edge of the image, ω _i represents the parameters of the convolution layer, and L _fuse represents the edge fusion loss function: (3c) setting a maximum number of iterations I, iteratively training the side output edge detection network model Q according to the loss function of the side output edge detection network model Q and using the feature map set output by each structural layer of the feature extraction network in the image reconstruction neural network model R, and obtaining a trained side output edge detection network model Q; (4) Build and train the edge correction network model Z: (4a) sequentially connecting the dilated spatial convolutional pooling pyramid model _Fγ and the softmax activation function output layer, wherein the dilated spatial convolutional pooling pyramid model _Fγ is composed of a plurality of sequentially connected convolutional layers and pooling layers, and obtaining an edge correction network model Z; (4b) Define the loss function of the edge correction network model Z: in, The rough segmentation result of the target frame output by the edge detection layer, is the prediction result of the dilated spatial convolutional pooling pyramid model F _γ , in, Represents the image edge obtained by the Canny algorithm, and M represents the mask The number of categories of pixels in , Indicates mask The total number of pixels in (4c) setting a maximum number of iterations H, iteratively training the edge correction network model Z according to the loss function of the edge correction network model Z and using the output results of the image reconstruction network model R and the edge detection network model Q to obtain a trained edge correction network model Z; (5) The video target segmentation model based on the image target edge correction segmentation result is obtained by combining the trained image reconstruction neural network R, the side output edge detection network Q and the edge correction network model Z; (6) Obtain self-supervised video target segmentation results: The test set The frame images in are used as the input of the video target segmentation model for forward propagation to obtain the predicted segmentation labels of all test frame images, and the final segmentation result image is obtained based on the predicted segmentation labels of the test frame images. 2. The method according to claim 1, characterized in that: in step (1), the training sample set V _train , the verification sample set V _val and the test sample set V _test are obtained according to the following steps: (1a) Obtain S multi-category video sequences from the video object segmentation dataset and obtain a frame sequence set after preprocessing S≥3000; represents the kth frame sequence consisting of preprocessed image frames, represents the nth image frame in the kth frame sequence, M ≥ 30; (1b) Randomly select more than half of the frame sequences from the frame sequence set V to form a training sample set Where S/2＜N＜S, for each frame sequence in the training sample set Each target frame image to be segmented Scale the image blocks to p×p×h size and convert the image format from RGB to Lab; extract half of the frame sequences from the remaining frame sequences to form a verification sample set Where J≤S/4; the other half constitutes the test sample set T≤S/4, and convert the image format from RGB to Lab. 3. The method according to claim 1, characterized in that: in step (2c), the image reconstruction neural network model R is iteratively trained as follows: (2c1) Let the network hyperparameter of the feature extraction network be θ, the maximum number of iterations be N ≥ 150000, n represents the current number of iterations; let n = 1, the number of initial iterations; (2c2) The target frame images in the training sample set V _train are used as the input of the image reconstruction neural network model R for forward propagation: For each target frame I _t to be segmented, select the q frames in front of it as reference frames {I' ₀ , I' ₁ , ..., I' _q }, where 2≤q≤5, the target frame I _t and its corresponding reference frame set are used as inputs of the feature extraction network Φ(.; θ), the feature extraction network extracts features from I _t and each of its reference frame images, and obtains the target frame image feature f _t =Φ(I _t ; θ), the reference frame image feature f′ ₀ =Φ(I′ ₀ ; θ), ..., f′ _q =Φ(I′ _q ; θ), the target frame {I _t |0≤t≤N} in the training sample set is used as the input of the K-means algorithm, and the quantized image reconstruction loss value L _cls is obtained. The reconstructed target frame I _t and the real target frame I _t are used as the input of the RGB image reconstruction task, and the RGB image reconstruction loss value L _reg is obtained; (2c3) Using the loss function L _mix , the loss value of the image reconstruction neural network is calculated through the cross entropy loss L _cls and the regression loss L _reg Back propagation is used to calculate the network parameter gradient g(θ), and then the network parameter θ is updated by the gradient descent method; (2c4) Determine whether n=N. If so, obtain the trained image reconstruction neural network R; otherwise, set n=n+1 and return to step (2c2). 4. The method according to claim 1, characterized in that: in step (3c), the side output edge detection network model Q is iteratively trained as follows: (3c1) Assume that the maximum number of iterations I≥150000, the current number of iterations is i; let i=1, and initialize the number of iterations; (3c2) The feature map set output by each structural layer of the feature extraction network in the image reconstruction network model Perform forward propagation as input to the side output edge detection network: (3c3) The output edge detection layer obtains the rough edge of the target from the feature map set, thereby obtaining the rough edge corresponding to each feature map (3c4) The rough edge set output by the side output edge detection layer SODL is used as the input of the side output edge fusion layer SOFL, and the rough edges are weighted and fused to obtain the final predicted edge. in, represents the feature map formed by merging coarse edges, ω _fuse represents the parameters of the side output edge fusion layer; (3c5) Using the loss function L _edge , the loss value of the edge detection network is calculated by the side output edge detection loss L _side and the side output edge fusion loss L _fuse Back propagation is used to calculate the network parameter gradient g(ω), and then the network parameter ω is updated by the gradient descent method; (3c6) Determine whether i=I holds. If so, obtain the trained side output edge detection network model Q; otherwise, set i=i+1 and return to execute step (3c2). 5. The method according to claim 1, characterized in that: in step (4c), the edge correction network model Z is iteratively trained as follows: (4c1) Assume that the maximum number of iterations is H ≥ 150000, and the current number of iterations is h; let h = 1, and initialize the number of iterations; (4c2) The target frame rough segmentation result output by the image reconstruction network model R and the edge detection result output by the edge detection network model Q are used as the input of the edge correction network model Z for forward propagation: (4c2.1) The edge correction network first roughly segments the target frame And edge detection results Merge in the channel dimension to obtain a feature map of size H×W×(K+1) (4c2.2) The feature map As the input of the dilated spatial convolutional pooling pyramid model F _γ , the prediction result of the expanded receptive field is obtained; (4c2.3) The predicted result of the expanded receptive field is used as the input of the output layer of the softmax activation function. The segmentation label of the pixel is determined according to the probability that each pixel in the feature map belongs to each category, so as to obtain a more accurate target segmentation mask after edge fusion correction of the target frame target segmentation mask. Where O _t represents the predicted segmentation label of the target frame I _t ; (4c3) Use the loss function L _corr to calculate the loss value of the edge correction network Back propagation is used to calculate the network parameter gradient g(c), and then the network parameter c is updated by the gradient descent method; (4c4) Determine whether h=H holds. If so, obtain the trained edge correction network model Z; otherwise, set h=h+1 and return to execute step (4c2). 6. The method according to claim 1 is characterized in that: the video target segmentation model in step (5) specifically inputs the intermediate feature map extracted by the image reconstruction neural network R into the side output edge detection network Q to obtain the target edge prediction map, and the target segmentation mask prediction map output by the image reconstruction neural network R and the target edge prediction map output by the side output edge detection network Q are used as the input of the edge correction network model Z to obtain a trained video target segmentation model based on the image target edge correction segmentation result.