CN112508959A - Video object segmentation method and device, electronic equipment and storage medium - Google Patents

Abstract

The present disclosure relates to a video target segmentation method, device, electronic device and storage medium, and relates to the field of computer vision. The method includes: determining a first target feature of a first target frame, a first reference feature of a first reference frame, and a reference segmentation map of the first reference frame; Perform weighting to obtain the second target feature; weight the first reference feature based on the second non-local attention of the first reference frame to obtain the second reference feature; determine the first offset based on the second target feature and the second reference feature information; perform offset processing on the second reference feature based on the first offset information, obtain the second reference feature after offset, and determine the first local attention between the second reference feature after offset and the second target feature; Referring to the segmentation map and the first local attention input target segmentation model, the first target segmentation map of the first target frame is obtained; this scheme improves the accuracy of video target segmentation.

Description

Video object segmentation method, device, electronic device and storage medium

technical field

The present disclosure relates to the field of computer vision, and in particular, to a video object segmentation method, apparatus, electronic device, and storage medium.

Background technique

In the process of video processing, for example, video editing, video monitoring, etc., it is sometimes necessary to perform target segmentation on the video, that is, to obtain a segmentation map of the target foreground in each frame of the video. Generally, the target segmentation is performed on the video through the target segmentation model.

In the related art, the process of using the target segmentation model to perform target segmentation on the video is as follows: obtaining the target frame, the reference frame and the segmentation map of the reference frame from the video to be divided; for each first pixel in the target frame, from the In the reference frame, determine the second pixel with the greatest similarity with the first pixel, and based on the first pixel, the second pixel and the reference segmentation map, obtain the target segmentation model through the target segmentation model. When determining the second pixel point, in order to reduce the amount of calculation, a series of pixels are sampled at equal intervals within a certain range of neighborhoods with the pixel position of the first pixel point as the midpoint in the reference frame. point, take these pixels as the second pixel.

In the above method, when the time interval between the reference frame and the target frame is too large, since the sampling interval is fixed, the sampling interval may not be the offset of the first pixel in the above time interval, that is, The second pixel determined based on the sampling interval may not be the pixel with the highest similarity with the first pixel, resulting in segmentation of the target frame based on the first pixel, the second pixel and the reference segmentation map The graph is inaccurate, which in turn leads to lower accuracy of video object segmentation.

SUMMARY OF THE INVENTION

The present disclosure provides a video object segmentation method, device, electronic device and storage medium, which can improve the accuracy of video object segmentation. The technical solutions of the present disclosure are as follows:

According to a first aspect of the embodiments of the present disclosure, a video object segmentation method is provided, including:

Determine the first target feature of the first target frame, the first reference feature of the first reference frame, and the reference segmentation map of the first reference frame, where the first target frame is the video to be segmented without target segmentation. frame, the first reference frame is a video frame in which target segmentation has been performed in the video, and the interval between the first reference frame and the first target frame is greater than a first preset duration;

Based on the first non-local attention of the first target frame, the first target feature is weighted to obtain a second target feature, and based on the second non-local attention of the first reference frame, the The first reference feature is weighted to obtain the second reference feature;

Determine first offset information based on the second target feature and the second reference feature, where the first offset information is used to represent the offset between the first pixel point corresponding to the position and the second pixel point, The first pixel is a pixel in the first target frame, and the second pixel is a pixel in the first reference frame;

Based on the first offset information, an offset process is performed on the second reference feature to obtain a second reference feature after offset, and the difference between the second reference feature after offset and the second target feature is determined. the first partial attention in between;

The reference segmentation map and the first local attention are input into a target segmentation model to obtain a first target segmentation map of the first target frame.

In some embodiments, the determining the first offset information based on the second target feature and the second reference feature includes:

Splicing the second target feature and the second reference feature to obtain the first feature;

Inputting the first feature into an offset prediction model to obtain the first offset information.

In some embodiments, the inputting the reference segmentation map and the first local attention into a target segmentation model to obtain a first target segmentation map of the first target frame, including:

Through the target segmentation model, sliding window extraction is performed on each third pixel point in the reference segmentation map to obtain multiple first neighborhood maps corresponding to each third pixel point, and the multiple The first neighborhood map is spliced to obtain the reference segmentation map after sliding window extraction;

Perform dimension transformation on the reference segmentation map after the sliding window extraction and the first local attention, respectively, to obtain the dimension-transformed reference segmentation map and the dimension-transformed first local attention, and the dimension-transformed reference The segmentation map has the same dimension as the first local attention after the dimension transformation;

A first target segmentation map of the first target frame is determined based on the dimension-transformed reference segmentation map and the dimension-transformed first local attention.

In some embodiments, determining the first target segmentation map of the first target frame based on the dimension-transformed reference segmentation map and the dimension-transformed first local attention includes:

For each fourth pixel in the dimension-transformed reference segmentation map, a plurality of similarities are determined from the dimension-transformed first local attention, and the plurality of similarities are the fourth pixel similarity between points and other pixels;

Based on the multiple degrees of similarity, weighted summation is performed on the pixel values of the fourth pixel point to obtain a weighted and summed pixel value;

Modifying the pixel value of the fourth pixel point to the weighted and summed pixel value to obtain a first target segmentation map of the first target frame.

In some embodiments, the first target feature is weighted based on the first non-local attention of the first target frame to obtain the second target feature, including:

normalizing the first non-local attention to obtain a first weight;

Perform matrix multiplication processing on the first weight and the first target feature by using the target segmentation model to obtain a third target feature;

The third target feature is superimposed on the first target feature to obtain the second target feature.

In some embodiments, the method further includes:

In the case that the interval between the first reference frame and the first target frame is not greater than the first preset duration, the second interval between the first reference feature and the first target feature is The local attention and the reference segmentation map are input into the target segmentation model to obtain the first target segmentation map of the first target frame.

In some embodiments, the target segmentation model is trained by the following methods:

Obtain the sample target frame and the first sample reference frame without target segmentation from the sample video, input the sample target frame and the first sample reference frame into the initial target segmentation model, and obtain the first sample target frame. A sample target feature and a first sample reference feature of the first sample reference frame, and the interval between the sample target frame and the first sample reference frame is greater than the second preset duration and less than the third preset time. set duration;

The first sample target feature is weighted based on the third non-local attention of the sample target frame to obtain the second sample target feature, and the fourth non-local attention based on the first sample reference frame , weighting the first sample reference feature to obtain a second sample reference feature;

Based on the second sample target feature and the second sample reference feature, second offset information is determined, where the second offset information is used to indicate the difference between the first sample pixel point corresponding to the position and the second sample pixel point The offset between, the first sample pixel point is the pixel point in the sample target frame, and the second sample pixel point is the pixel point in the first sample reference frame;

Based on the second offset information, performing an offset process on the second sample reference feature to obtain a shifted second sample reference feature;

Based on the first sample reference frame and the third local attention, the initial target segmentation model is trained to obtain the target segmentation model, and the third local attention is the shifted second sample Local attention between the reference feature and the second sample target feature.

In some embodiments, the initial target segmentation model is trained based on the first sample reference frame and the third local attention to obtain the target segmentation model, including:

Through the initial target segmentation model, sliding window extraction is performed on each second sample pixel in the first sample reference frame to obtain a plurality of second neighborhoods corresponding to each second sample pixel splicing the plurality of second neighborhood graphs to obtain a second sample reference frame;

Perform dimension transformation on the second sample reference frame and the third local attention respectively to obtain the second sample reference frame after dimension transformation and the third local attention after dimension transformation, and the second sample reference frame after dimension transformation and the third local attention after dimension transformation are obtained. The sample reference frame is the same as the dimension of the third local attention after the dimension transformation;

Based on the dimension-transformed second sample reference frame and the dimension-transformed third local attention, the initial target segmentation model is trained to obtain the target segmentation model.

In some embodiments, the initial target segmentation model is trained based on the dimension-transformed second sample reference frame and the dimension-transformed third local attention to obtain the target segmentation model, include:

For each third sample pixel in the dimension-transformed second sample reference frame, a plurality of similarities are determined from the dimension-transformed third local attention, and the plurality of similarities are the The similarity between the third sample pixel point and other pixel points;

Based on the multiple degrees of similarity, weighted summation is performed on the pixel values of the third sample pixel points to obtain a weighted and summed pixel value;

Modifying the pixel value of the third sample pixel point to the pixel value after the weighted summation to obtain the second target segmentation map of the sample target frame;

The sample target frame is input into the initial target segmentation model, and based on the output result of the initial target segmentation model and the loss value between the second target segmentation map, the model parameters of the initial target segmentation model are analyzed. Iteratively update to obtain the target segmentation model.

According to a second aspect of the embodiments of the present disclosure, there is provided an apparatus for training a target segmentation model, including:

a first determining unit configured to perform determining a first target feature of a first target frame, a first reference feature of a first reference frame, and a reference segmentation map of the first reference frame, where the first target frame is to be segmented The video frame that has not been subjected to target segmentation in the video, the first reference frame is the video frame that has been subjected to target segmentation in the video, and the interval between the first reference frame and the first target frame preset duration;

A first weighting unit configured to perform a first non-local attention based on the first target frame, weight the first target feature to obtain a second target feature, and perform a first non-local attention based on the first reference frame. 2. Non-local attention, weighting the first reference feature to obtain a second reference feature;

a second determining unit, configured to determine first offset information based on the second target feature and the second reference feature, where the first offset information is used to indicate that the first pixel point corresponding to the position and the first pixel point The offset between two pixel points, the first pixel point is a pixel point in the first target frame, and the second pixel point is a pixel point in the first reference frame;

a first offset unit, configured to perform offset processing on the second reference feature based on the first offset information, to obtain a shifted second reference feature, and to determine the shifted second reference feature a first local attention between the reference feature and the second target feature;

A first target segmentation unit, configured to input the reference segmentation map and the first local attention into a target segmentation model to obtain a first target segmentation map of the first target frame.

In some embodiments, the second determination unit is configured to perform splicing of the second target feature and the second reference feature to obtain a first feature; input the first feature into an offset prediction model , to obtain the first offset information.

In some embodiments, the first target segmentation unit includes:

The first sliding window extraction subunit is configured to perform sliding window extraction on each third pixel point in the reference segmentation map through the target segmentation model, to obtain a a plurality of first neighborhood graphs, splicing the plurality of first neighborhood graphs to obtain a reference segmentation graph after sliding window extraction;

The first dimension transformation subunit is configured to perform dimension transformation on the reference segmentation map after the sliding window extraction and the first local attention respectively, so as to obtain the reference segmentation map after dimension transformation and the first dimension transformation map. Local attention, the dimension of the reference segmentation map after the dimension transformation is the same as the dimension of the first local attention after the dimension transformation;

A determining subunit is configured to perform determining a first target segmentation map of the first target frame based on the dimension-transformed reference segmentation map and the dimension-transformed first local attention.

In some embodiments, the determining subunit is configured to perform, for each fourth pixel point in the dimension-transformed reference segmentation map, determining the number of pixels from the dimension-transformed first local attention. similarity degrees, the multiple degrees of similarity are the degrees of similarity between the fourth pixel point and other pixel points; based on the multiple degrees of similarity, the pixel values of the fourth pixel point are weighted and summed, Obtaining the weighted and summed pixel value; modifying the pixel value of the fourth pixel point to the weighted and summed pixel value to obtain the first target segmentation map of the first target frame.

In some embodiments, the first weighting unit is configured to perform normalization processing on the first non-local attention to obtain a first weight; the first weight and the sum of the first weights are calculated by the target segmentation model. The first target feature is subjected to matrix multiplication processing to obtain a third target feature; the third target feature is superimposed with the first target feature to obtain the second target feature.

In some embodiments, the apparatus further includes:

The second target segmentation unit is configured to perform, under the condition that the interval between the first reference frame and the first target frame is not greater than the first preset duration, combine the first reference feature with the first target frame. The second local attention between the first target features and the reference segmentation map is input to the target segmentation model to obtain a first target segmentation map of the first target frame.

In some embodiments, the apparatus further includes:

The acquisition unit is configured to perform the acquisition of the sample target frame and the first sample reference frame without target segmentation from the sample video, and input the sample target frame and the first sample reference frame into the initial target segmentation model to obtain The first sample target feature of the sample target frame and the first sample reference feature of the first sample reference frame, the interval between the sample target frame and the first sample reference frame is greater than the second The preset duration is less than the third preset duration;

A second weighting unit configured to perform a third non-local attention based on the sample target frame, weight the first sample target feature to obtain a second sample target feature, and based on the first sample The fourth non-local attention of the reference frame weights the first sample reference feature to obtain the second sample reference feature;

a third determining unit, configured to determine second offset information based on the second sample target feature and the second sample reference feature, where the second offset information is used to represent the first sample corresponding to the position The offset between the pixel point and the second sample pixel point, the first sample pixel point is the pixel point in the sample target frame, and the second sample pixel point is the first sample reference frame. pixel point;

a second offset unit, configured to perform offset processing on the second sample reference feature based on the second offset information, to obtain a shifted second sample reference feature;

a training unit, configured to perform training on the initial target segmentation model based on the first sample reference frame and a third local attention, to obtain the target segmentation model, and the third local attention is the Local attention between the offset second sample reference feature and the second sample target feature.

In some embodiments, the training unit includes:

The second sliding window extraction subunit is configured to perform sliding window extraction on each second sample pixel point in the first sample reference frame through the initial target segmentation model, to obtain the same value as each second sample pixel point in the first sample reference frame. Multiple second neighborhood maps corresponding to the two-sample pixel points, splicing the multiple second neighborhood maps to obtain a second sample reference frame;

A second dimensional transformation subunit configured to perform dimensional transformation on the second sample reference frame and the third local attention, respectively, to obtain a dimensionally transformed second sample reference frame and a dimensionally transformed third local Attention, the dimension of the second sample reference frame after the dimension transformation is the same as the dimension of the third local attention after the dimension transformation;

The training subunit is configured to perform training on the initial target segmentation model based on the dimension-transformed second sample reference frame and the dimension-transformed third local attention to obtain the target segmentation model.

In some embodiments, the training subunit is configured to perform, for each third sample pixel in the dimensionally transformed second sample reference frame, a third local attention transformed from the dimension Determine a plurality of similarities in Carry out weighted summation to obtain the pixel value after the weighted summation; modify the pixel value of the third sample pixel point to the pixel value after the weighted summation to obtain the second target segmentation map of the sample target frame; The sample target frame is input into the initial target segmentation model, and based on the output result of the initial target segmentation model and the loss value between the second target segmentation map, the model parameters of the initial target segmentation model are analyzed. Iteratively update to obtain the target segmentation model.

According to a third aspect of embodiments of the present disclosure, there is provided an electronic device comprising a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions , so as to implement the video object segmentation method in the above embodiment.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, storing instructions in the storage medium, and when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device enables the electronic device to Execute the video object segmentation method in the above embodiment.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device can execute the video object segmentation method in the above embodiment .

The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

In the embodiment of the present disclosure, since the feature of the video frame can reflect the semantic information of the video frame, based on the target feature of the first target frame and the reference feature of the first reference frame, the determined first offset information has been fused with Semantic information of two video frames, so that the first offset information can accurately represent the offset of each pixel corresponding to the position from the first reference frame to the first target frame, so as to predict based on the first offset information The target segmentation map of the first target frame is more accurate, thereby improving the accuracy of video target segmentation.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure and do not unduly limit the present disclosure.

Fig. 1 is a flow chart of a method for segmenting a video object according to an exemplary embodiment.

Fig. 2 is a flowchart showing a training method of a target segmentation model according to an exemplary embodiment.

Fig. 3 is a schematic diagram showing a training method of a target segmentation model according to an exemplary embodiment.

Fig. 4 is a flow chart of a method for segmenting a video object according to an exemplary embodiment.

Fig. 5 is a block diagram of a video object segmentation apparatus according to an exemplary embodiment.

Fig. 6 is a block diagram of a server according to an exemplary embodiment.

Fig. 7 is a block diagram of a terminal according to an exemplary embodiment.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

The user information involved in the present disclosure may be information authorized by the user or fully authorized by all parties.

An embodiment of the present disclosure provides a video object segmentation method, which is implemented by an electronic device. Among them, the target segmentation model used in this method can be applied in scenarios such as video surveillance, transportation, and video editing.

For example, in a video editing scenario, a technician needs to edit objects such as characters or objects in the video, but the video is generally a dynamic video, that is, the object in the video is moving. Therefore, it is necessary to separate the target from the video first, and then edit the target; for example, if the target is actor c in the video, the electronic device obtains the video collected by the video capture device, divides the video into frames, and obtains A plurality of video frames, from each video frame, an image including actor c is segmented, and then the image is edited.

Fig. 1 is a flowchart of a method for segmenting a video object according to an exemplary embodiment. As shown in Fig. 1 , the method for segmenting a video object is used in an electronic device, and includes the following steps.

In step S11, the first target feature of the first target frame, the first reference feature of the first reference frame, and the reference segmentation map of the first reference frame are determined, and the first target frame is an untargeted video in the video to be segmented A segmented video frame, the first reference frame is a video frame in which target segmentation has been performed in the video, and the interval between the first reference frame and the first target frame is greater than the first preset duration;

In step S12, the first target feature is weighted based on the first non-local attention of the first target frame to obtain the second target feature, and based on the second non-local attention of the first reference frame, the The first reference feature is weighted to obtain a second reference feature;

In step S13, based on the second target feature and the second reference feature, determine first offset information, where the first offset information is used to indicate the offset between the first pixel point corresponding to the position and the second pixel point moving, the first pixel is a pixel in the first target frame, and the second pixel is a pixel in the first reference frame;

In step S14, based on the first offset information, an offset process is performed on the second reference feature to obtain a second reference feature after offset, and the second reference feature after offset and the second target feature are determined the first local attention between;

In step S15, the reference segmentation map and the first local attention are input into a target segmentation model to obtain a first target segmentation map of the first target frame.

In some embodiments, determining the first offset information based on the second target feature and the second reference feature includes:

splicing the second target feature and the second reference feature to obtain the first feature;

The first feature is input into an offset prediction model to obtain the first offset information.

In some embodiments, the reference segmentation map and the first local attention are input into a target segmentation model to obtain a first target segmentation map of the first target frame, including:

Through the target segmentation model, sliding window extraction is performed on each third pixel in the reference segmentation map to obtain a plurality of first neighborhood maps corresponding to each third pixel, and the plurality of first neighborhood maps are obtained. The domain map is spliced to obtain the reference segmentation map after sliding window extraction;

Dimensionally transform the reference segmentation map extracted by the sliding window and the first local attention, respectively, to obtain the dimension-transformed reference segmentation map and the dimension-transformed first local attention. The dimension-transformed reference segmentation map is the same as The dimension of the first local attention after the dimension transformation is the same;

Based on the dimension-transformed reference segmentation map and the dimension-transformed first local attention, a first target segmentation map of the first target frame is determined.

In some embodiments, determining the first target segmentation map of the first target frame based on the dimension-transformed reference segmentation map and the dimension-transformed first local attention includes:

For each fourth pixel in the dimension-transformed reference segmentation map, determine a plurality of similarities from the dimension-transformed first local attention, where the plurality of similarities are the fourth pixel and other pixels similarity between points;

Based on the plurality of similarities, weighted summation is performed on the pixel values of the fourth pixel point to obtain a weighted and summed pixel value;

The pixel value of the fourth pixel is modified to the weighted and summed pixel value to obtain a first target segmentation map of the first target frame.

In some embodiments, the first target feature is weighted based on the first non-local attention of the first target frame to obtain the second target feature, including:

Normalize the first non-local attention to obtain a first weight;

Perform matrix multiplication processing on the first weight and the first target feature through the target segmentation model to obtain a third target feature;

The third target feature is superimposed on the first target feature to obtain the second target feature.

In some embodiments, the method further includes:

Under the condition that the interval between the first reference frame and the first target frame is not greater than the first preset duration, the second local attention between the first reference feature and the first target feature and the Referring to the segmentation map, input the target segmentation model to obtain a first target segmentation map of the first target frame.

In some embodiments, the target segmentation model is trained by the following methods:

Obtain the sample target frame and the first sample reference frame without target segmentation from the sample video, input the sample target frame and the first sample reference frame into the initial target segmentation model, and obtain the first sample of the sample target frame The target feature and the first sample reference feature of the first sample reference frame, and the interval between the sample target frame and the first sample reference frame is greater than the second preset duration and less than the third preset duration;

Based on the third non-local attention of the sample target frame, the first sample target feature is weighted to obtain the second sample target feature, and based on the fourth non-local attention of the first sample reference frame, the The first sample reference feature is weighted to obtain the second sample reference feature;

Based on the second sample target feature and the second sample reference feature, second offset information is determined, where the second offset information is used to indicate the offset between the first sample pixel point corresponding to the position and the second sample pixel point moving, the first sample pixel is a pixel in the sample target frame, and the second sample pixel is a pixel in the first sample reference frame;

Based on the second offset information, performing offset processing on the second sample reference feature to obtain the offset second sample reference feature;

Based on the first sample reference frame and the third local attention, the initial target segmentation model is trained to obtain the target segmentation model, and the third local attention is the offset of the second sample reference feature and the third local attention. Local attention between two-sample target features.

In some embodiments, the initial target segmentation model is trained based on the first sample reference frame and the third local attention to obtain the target segmentation model, including:

Through the initial target segmentation model, sliding window extraction is performed on each second sample pixel in the first sample reference frame, and a plurality of second neighborhood maps corresponding to each second sample pixel are obtained. The plurality of second neighborhood graphs are spliced to obtain a second sample reference frame;

Perform dimension transformation on the second sample reference frame and the third local attention respectively to obtain the second sample reference frame after dimension transformation and the third local attention after dimension transformation, the second sample reference frame after dimension transformation It is the same as the dimension of the third local attention after the dimension transformation;

Based on the dimensionally transformed second sample reference frame and the dimensionally transformed third local attention, the initial target segmentation model is trained to obtain the target segmentation model.

In some embodiments, the initial target segmentation model is trained based on the dimension-transformed second sample reference frame and the dimension-transformed third local attention to obtain the target segmentation model, including:

For each third sample pixel in the dimension-transformed second sample reference frame, a plurality of similarities are determined from the dimension-transformed third local attention, and the plurality of similarities are the third sample pixels similarity between points and other pixels;

Based on the multiple degrees of similarity, weighted summation is performed on the pixel values of the third sample pixel point to obtain a weighted and summed pixel value;

Modify the pixel value of the third sample pixel point to the pixel value after the weighted summation to obtain the second target segmentation map of the sample target frame;

Input the sample target frame into the initial target segmentation model, based on the output result of the initial target segmentation model and the loss value between the second target segmentation map, iteratively update the model parameters of the initial target segmentation model to obtain the target segmentation model.

In the embodiment of the present disclosure, since the feature of the video frame can reflect the semantic information of the video frame, based on the target feature of the first target frame and the reference feature of the first reference frame, the determined first offset information has been fused with Semantic information of two video frames, so that the first offset information can accurately represent the offset of each pixel corresponding to the position from the first reference frame to the first target frame, so as to predict based on the first offset information The target segmentation map of the first target frame is more accurate, thereby improving the accuracy of video target segmentation.

FIG. 2 is a flowchart showing a training method of a target segmentation model according to an exemplary embodiment. As shown in FIG. 2 , in the embodiment of the present disclosure, the training of a target segmentation model is taken as an example for description. The training method used in electronic equipment includes the following steps:

In step 201, a sample target frame and a first sample reference frame without target segmentation are obtained from the sample video, and the sample target frame and the first sample reference frame are input into an initial target segmentation model to obtain the sample target frame The first sample target feature and the first sample reference feature of the first sample reference frame, the interval between the sample target frame and the first sample reference frame is greater than the second preset duration and less than the third preset duration.

Among them, the sample video is the video in the training data set. The training dataset includes multiple sample videos. Before training the initial target segmentation model, the electronic device obtains a plurality of sample videos in advance, and forms a training data set from the plurality of sample videos. In this step, the electronic device acquires the training data set, and selects sample videos from the training data set.

Wherein, the sample target frame is a video frame without target segmentation, the first sample reference frame is a video frame of a known target segmentation map, and in the sample video, the first sample reference frame is located in the sample target frame before. The interval between the sample target frame and the first sample reference frame is greater than the second preset duration and smaller than the third preset duration. The second preset duration is a value less than 15, and the third preset duration is a value greater than 15. For example, if the interval is Δt, the second preset duration is 5, and the third preset duration is 20, then Δt is 5- A value between 20.

The first sample target feature is a high-dimensional feature of the sample target frame, and the first sample reference feature is a high-dimensional feature of the first sample reference frame. The initial target segmentation model is a neural network model, and the neural network model is a Residual Network (Residual Network, ResNet), for example, ResNet-18. In this step, the electronic device uses the initial target segmentation model as a feature extractor, and based on the sample target frame, obtains the first sample target feature through the following formula 1, and based on the first sample reference frame, through the following formula 2, Get the first sample reference feature:

Formula 1: f _t =Φ(I _t ; θ _Φ )

Formula 2: f _r =Φ(I _r ; θ _Φ )

Among them, f _t is the first sample target feature, I _t is the sample target frame, Φ is the initial target segmentation model, θ _Φ is the learnable parameter of Φ, _{fr is the first sample reference feature, I r is} the first sample Sample reference frame, I _t ∈ R ^C×H×W , I _r ∈ R ^C×H×W , where H, W and C are the height, width and channel number of the video frame, respectively, and R is the set of real numbers.

In step 202, the first sample target feature is weighted based on the third non-local attention of the sample target frame to obtain the second sample target feature.

Among them, the third non-local attention is the similarity matrix. Non-local attention can represent the mutual response of two pixels separated by any distance in an image, and this response is used to represent the similarity of the two pixels.

This step can be achieved through the following steps (1) to (2), including:

(1) Determine the third non-local attention of the sample target frame.

In some embodiments, the electronic device determines the similarity between each first sample pixel point in the sample target frame based on the first sample target feature, and the implementation manner of obtaining the third non-local attention includes the following: Steps A1-A3:

A1: Perform dimension reduction processing on the first sample target feature to obtain a dimension-reduced first sample target feature, where the dimension-reduced first sample target feature includes the feature of each first sample pixel point.

The electronic device may reduce the dimension of the first sample target feature to any dimension. For example, the electronic device reduces the number of channels of the target feature of the first sample and the reference feature of the first sample to 1/8 of the number of channels respectively, that is, from f _t ∈ R ^C×H×W to f _t ∈ R ^{( C/8)×H×W} , dimensionality reduction from fr ∈R ^C×H×W to _{fr r ∈R} ^(C/8)×H×W .

A2: For each first sample pixel, based on the features of the first sample pixel and the features of other first sample pixels, determine the similarity between the first sample pixel and the other pixels degree to obtain multiple similarities.

In some embodiments, for each first sample pixel point, based on the characteristics of the first sample pixel point and the characteristics of all other first sample pixel points, the first sample pixel point and the other The similarity between the first sample pixels is obtained to obtain a plurality of similarities.

In some embodiments, for each first sample pixel, based on the characteristics of the first sample pixel and the characteristics of some other first sample pixels, it is determined that the first sample pixel is different from the other The similarity between the first sample pixels is obtained to obtain a plurality of similarities. Wherein, the other first sample pixel points in the part are a plurality of first sample pixel points obtained after removing edge pixel points in the sample target frame. The embodiment of the present disclosure does not specifically limit the selection of edge pixels.

It should be noted that, for a first sample pixel, multiple similarities will be obtained, and the multiple similarities can be stored in the form of a matrix, then the multiple similarities obtained by the first sample pixel form a first local matrix. Since the first sample target feature includes features of multiple first sample pixel points, multiple first local matrices will be obtained in this step.

A3: The multiple similarities compose the third non-local attention.

For example, according to the coordinates of the first sample pixel point a in the sample target frame, the plurality of first local matrices obtained in step A2 are formed into the third non-local attention.

Steps A2-A3 are implemented by formula three:

Formula 3: _Anlocal (i,j)=g(f _t (i); θ _g ) ^T h(f _t (j); θ _h )

N＝H×W，f_t(i),f_t(j)为第一样本目标特征中两个不同坐标的像素值，g,h分别表示一个1×1卷积层，θ_g,θ_h，分别为g,h的可学习参数。Among them, i, j are the coordinate positions, _Anlocal is the third non-local attention,

N=H×W, f _t (i), f _t (j) are the pixel values of two different coordinates in the first sample target feature, g, h respectively represent a 1×1 convolutional layer, θ _g , θ _h , are the learnable parameters of g and h, respectively.

(2) Based on the third non-local attention, weight the first sample target feature to obtain the second sample target feature.

In some embodiments, the electronic device weights the first sample target feature based on the third non-local attention of the sample target frame, and the implementation manner of obtaining the second sample target feature includes the following steps B1-B3:

B1: Normalize the third non-local attention to obtain the second weight.

The second weight is used to determine the offset of the first sample pixel point in the sample target frame in the subsequent process, that is, to determine the movement of the target in the sample target frame.

B2: Perform matrix multiplication processing on the second weight and the first sample target feature through the initial target segmentation model to obtain a third sample target feature.

In step B2, by performing matrix multiplication processing on the first weight and the first sample target feature, the residual of the first sample target feature, that is, the third sample target feature is obtained.

B3: Superimpose the third sample target feature and the first sample target feature to obtain the second sample target feature.

In step B3, by superimposing the residual obtained in step B2 into the first sample target feature, the second sample target feature is obtained, so that the initial target segmentation model is trained based on the residual and the new target feature, thereby improving the The learning efficiency of the model.

Steps B1-B3 are realized by formula four:

Formula 4: f _r =softmax(A _nlocal , dim=0)·m(f _r ; θ _m )+f _r

where m is a convolutional layer of the initial target segmentation model, θ _m is the learnable parameter of m, and softmax is the normalization function.

In step 203, based on the fourth non-local attention of the first sample reference frame, the first sample reference feature is weighted to obtain the second sample reference feature.

Among them, the fourth non-local attention is the similarity matrix. This step can be achieved through the following steps (1) to (2), including:

(1) Determine the third non-local attention of the sample target frame.

In some embodiments, the electronic device determines the similarity between each second sample pixel point of the first sample reference frame based on the first sample reference feature, and obtains the fourth non-local attention by the following step C1 -C3 to achieve:

C1: Perform dimension reduction processing on the first sample reference feature to obtain a dimension-reduced first sample reference feature, where the dimension-reduced first sample reference feature includes a feature of each second sample pixel point.

This step is similar to step A1 and will not be repeated here.

C2: For each second sample pixel point, based on the characteristics of the second sample pixel point and the characteristics of other second sample pixel points, determine the similarity between the second sample pixel point and the other pixel points, and obtain the most a similarity.

This step is similar to step A2 and will not be repeated here.

C3: The plurality of similarities are composed of the fourth non-local attention.

This step is similar to step A3 and will not be repeated here.

In the embodiment of the present disclosure, by performing dimension reduction processing on the first sample target feature and the first sample reference feature, the amount of calculation in obtaining non-local attention is reduced, and the The non-local attention of a sample target feature and the dimension-reduced first sample reference feature makes the pixels in the frame stimulate each other, promotes the flow of semantic information between the pixels in the frame, and speeds up the initial target segmentation model. training speed.

(2) Based on the fourth non-local attention, weight the first sample reference feature to obtain the second sample reference feature.

In some embodiments, the implementation of this step includes the following steps D1-D3:

D1: Normalize the fourth non-local attention to obtain the third weight.

This step is similar to step B1 and will not be repeated here.

D2: Perform matrix multiplication processing on the third weight and the first sample reference feature by using the initial target segmentation model to obtain the third sample reference feature.

This step is similar to step B2 and will not be repeated here.

D3: The electronic device superimposes the third sample reference feature and the first sample reference feature to obtain the second sample reference feature.

This step is similar to step B3 and will not be repeated here.

In the embodiment of the present disclosure, the non-local attention of the first sample target feature is weighted into the first sample target feature in the form of residual, and the non-local attention of the first sample reference feature is weighted as residual The difference form is weighted into the first sample reference feature, so that the non-local attention can be embedded in the neural network as a module, and the learning efficiency of the neural network is improved.

In step 204, based on the second sample target feature and the second sample reference feature, determine second offset information, where the second offset information is used to represent the first sample pixel point and the second sample pixel corresponding to the position The offset between points, the first sample pixel point is a pixel point in the sample target frame, and the second sample pixel point is a pixel point in the first sample reference frame.

Among them, in the sample video, the sample target frame is partially similar to the image content in the first sample reference frame, for example, the foreground object in the sample target frame may be the foreground object in the first sample reference frame. The result obtained by the rigid transformation, such as the background image in the sample target frame, may be the result of translation of the background image in the first sample reference frame. In this step, the second offset information determined by the electronic device is an offset map, and the offset map represents the semantic and timing correlation between the sample target frame and the first sample reference frame, that is, the offset map not only represents the sample target frame The difference in image between the frame and the first sample reference frame, also represents the high-level difference between the sample target frame and the first sample reference frame.

In some embodiments, step 204 includes the following steps A1-A2:

A1: Splicing the second sample target feature and the second sample reference feature to obtain a second feature.

In step A1, since the second sample target feature and the second sample reference feature are spliced, high-level information fusion of the sample target frame and the first sample reference frame is realized.

A2: Input the second feature into an offset prediction model to obtain the second offset information.

Wherein, the offset prediction model is used to output the second offset information based on the input second feature. The offset prediction model is a three-layer neural network. The present disclosure does not specifically limit the network type of the offset prediction model.

Steps A1-A2 are implemented by formula five:

Formula five:

为偏移预测模型，

为

的可学习参数，concat为拼接函数。Among them, O is the second offset information,

is the migration prediction model,

for

The learnable parameters of , concat is the concatenation function.

In the embodiment of the present disclosure, since the second sample target feature and the second sample reference feature are spliced, the high-level information of the sample target frame and the first sample reference frame is fused, and then the second feature obtained by splicing is input to the offset The prediction model is used to predict the semantic and temporal correlation offset of the two. Since the value of this offset is a real value, the sampling position is not fixed, and it is easier to hit the target pixel. Moreover, since the neural network is used for positioning, the amount of computation is small, thereby reducing the memory occupation of the graphics processor.

In step 205, based on the second offset information, an offset process is performed on the second sample reference feature to obtain a shifted second sample reference feature.

This step is realized by formula six:

Formula six:

为偏移后的第二样本参考特征，p为第二样本参考特征中的一个像素点。in,

is the offset second sample reference feature, and p is a pixel in the second sample reference feature.

中的p处的像素值为f_r中的p+O(p)处的像素值进行双线性插值后得到的像素值。The offset processing operation in this step is a warp (warping) operation, that is,

The pixel value at p in _fr is the pixel value obtained by performing bilinear interpolation on the pixel value at p+O(p) in fr.

In the embodiment of the present disclosure, the second sample reference feature after offset is obtained by performing offset processing on the second sample reference feature based on the second offset information, and then the offset second sample reference feature and the second sample reference feature after offset are obtained. The sample target features are semantically and temporally aligned, thereby providing a basis for the subsequent reconstruction of the sample target frame.

In step 206, using the initial target segmentation model, perform sliding window extraction on each second sample pixel in the first sample reference frame to obtain a plurality of second sample pixels corresponding to each second sample pixel Neighborhood graph, splicing the plurality of second neighborhood graphs to obtain a second sample reference frame.

其中，本公开对P的数值的选取不做具体限定，例如，P可以为3、9。Referring to FIG. 3, the electronic device performs sliding window extraction on each second sample pixel point in the first sample reference frame to obtain a plurality of P×P neighborhood blocks, and then splices the plurality of neighborhood blocks, get the second sample reference frame

Wherein, the present disclosure does not specifically limit the selection of the value of P, for example, P may be 3 or 9.

In step 207, a third local attention is determined based on the offset second sample reference feature and the second sample target feature.

In some embodiments, this step includes: the electronic device determines a second sample pixel point having the same coordinate position as the first sample pixel point in the shifted second sample reference feature, and selects the second sample pixel point with the same coordinate position as the first sample pixel point. The pixel point is the first partial image with the center point, and the similarity between the first sample pixel point and each second sample pixel point in the first partial image is determined, and the third partial attention is obtained.

For example, the first partial image has a size of P×P. The way in which the electronic determination of this third partial attention is realized is realized by Equation 7:

Formula seven:

P＝2M+1，P为第一局部图像的边长，M为第二样本像素点与第一局部图像的边之间的距离，i和k为第一样本像素点的坐标值，j和l为第二样本像素点的坐标值。Among them, A _local is the third local attention,

P=2M+1, P is the side length of the first partial image, M is the distance between the second sample pixel point and the edge of the first partial image, i and k are the coordinate values of the first sample pixel point, j And l is the coordinate value of the second sample pixel point.

In the embodiment of the present disclosure, since the similarity between each first sample pixel point in the second sample target feature and part of the second sample pixel point in the offset second sample reference feature is determined, and then capture The response of semantic information and timing information between frames is promoted, and the flow of semantic and timing information of pixels between frames is promoted.

In step 208, dimension transformation is performed on the second sample reference frame and the third local attention respectively to obtain the second sample reference frame after dimension transformation and the third local attention after dimension transformation. The second sample reference frame has the same dimension as the third local attention transformed by this dimension.

In this step, the electronic device transforms the dimension of the third local attention to obtain the third local attention after the dimension transformation. For example, the size of the third local attention is P×P×H×W, that is, the size of P ² ×H×W, and the size of the second sample reference frame obtained by sliding window extraction is C×P×P×H× W, the size of the second sample reference frame after dimension transformation is C×P ² ×H×W, therefore, the size of the third local attention after dimension transformation is 1×P ² ×H×W.

In the embodiment of the present disclosure, since the third local attention and the second sample reference frame are dimensionally transformed, the third local attention and the second sample reference frame with the same dimensions are obtained, thereby reducing the number of target frames of subsequent reconstruction samples It can improve the learning efficiency of the neural network in the training phase.

In step 209, the initial target segmentation model is trained based on the dimension-transformed second sample reference frame and the dimension-transformed third local attention to obtain the target segmentation model.

In some embodiments, step 209 is implemented by the following steps A1-A3:

A1: For each third sample pixel in the second sample reference frame transformed by the dimension, determine a plurality of similarities from the third local attention transformed by the dimension, and the multiple similarities are the third Similarity between sample pixels and other pixels.

Among them, the size of the third local attention after dimension transformation is 1×P ² ×H×W. It can be seen that each third sample pixel is the same as the second local attention in the third local attention after dimension transformation. Corresponding to the matrix, the second local matrix is a matrix of size P×P, and the matrix includes a plurality of similarities.

A2: Based on the multiple degrees of similarity, perform weighted summation on the pixel values of the third sample pixel point to obtain a weighted and summed pixel value.

Based on the plurality of similarities, weighted summation is performed on the pixel values of the third sample pixel point.

Wherein, for each third sample pixel in the dimension-transformed second sample reference frame, the electronic device determines the product of the third sample pixel and each similarity in the second local matrix to obtain a plurality of products value, and then add the multiple product values, that is, perform weighted summation on the pixel values of the third sample pixel point to obtain a new pixel value.

A3: Modify the pixel value of the third sample pixel point to the weighted and summed pixel value to obtain a second target segmentation map of the sample target frame.

Wherein, based on the new pixel value, the electronic device obtains the second target segmentation map through the initial target segmentation model, that is, reconstructs the sample target frame.

Steps A1-A3 are implemented by formula eight:

Formula eight:

为第二目标分割图，sum为求和函数，A_r为维度变换后的第三局部注意力。in,

is the second target segmentation map, sum is the summation function, and A _r is the third local attention after dimension transformation.

A4: Input the sample target frame into the initial target segmentation model, based on the loss value between the output result of the initial target segmentation model and the second target segmentation map, iteratively update the model parameters of the initial target segmentation model, Get the target segmentation model.

In the embodiment of the present disclosure, based on the third local attention after dimension transformation, the sample target frame and the sample reference frame are combined to capture the response of the semantic and timing information between the frames, and the sample reference obtained through the sliding window extraction operation. frame, and determine the second target segmentation map of the sample target frame, so that the accuracy of the target segmentation is high, thereby realizing the reconstruction of the sample target frame.

Wherein, the electronic device determines the loss function based on the second target segmentation map of the sample target frame. This step is implemented by Equation Nine:

Formula nine:

为损失函数，l_i为中间变量。Among them, N=H×W is the total number of pixels in the second target segmentation map,

is the loss function, and _li is the intermediate variable.

In the embodiment of the present disclosure, the electronic device performs steps 201-209 based on the sample video. In response to the end of the execution of step 209, the electronic device obtains other videos from the training data set, and continues to execute steps 201-209.

Among them, since the training data set includes multiple videos, each time the initial target segmentation model is trained, that is, sample videos are obtained from the multiple videos, the model will update the loss function once, and then based on the output results of the initial target segmentation model and the loss value between the second target segmentation map, and based on the loss value, back-propagate the initial target segmentation model and update the parameters of the initial target segmentation model.

Among them, in the training process of the initial target segmentation model, step A4 needs to be performed multiple times, that is, iterative update is performed. In this process, the training accuracy of the initial target segmentation model is improved by back-propagating and updating parameters, thereby improving the prediction accuracy of the target segmentation model.

In the embodiment of the present disclosure, the sliding window extraction is performed on the first sample reference frame to obtain the second sample reference frame, and the third local attention and the second sample reference frame are used to train the initial target segmentation model, so that the The training accuracy of the model is high.

In the embodiment of the present disclosure, the initial target segmentation model is trained based on the sample videos in the training data set, so that the initial target segmentation model can learn the semantic information relationship between the sample target frame and the sample reference frame during the training process , thereby improving the model prediction accuracy of the initial target segmentation model.

FIG. 4 is a flow chart of a method for segmenting a video object according to an exemplary embodiment. As shown in FIG. 4 , in the embodiment of the present disclosure, based on the object segmentation model, the objects in the video to be segmented are not segmented. The video frame of the video frame is used for target segmentation as an example to illustrate. The video target segmentation method is used in electronic equipment and includes the following steps:

In step 401, a first target feature of a first target frame, a first reference feature of a first reference frame, and a reference segmentation map of the first reference frame are determined, and the first target frame is an untargeted video in the video to be segmented A segmented video frame, the first reference frame is a video frame in which target segmentation has been performed in the video, and the interval between the first reference frame and the first target frame is greater than a first preset duration.

For example, the electronic device performs target segmentation on the first video frame in the video to be divided in advance to obtain a reference segmentation map, and then the electronic device uses the first video frame as the first reference frame, and the time is located in the first reference frame. The video frame after the frame is subjected to object segmentation. The reference segmentation map includes at least one target segmentation map.

For example, for the current frame It, that is, the first target frame, the electronic device extracts {I ₀ , M ₀ }, {I ₅ , M ₅ } (if t>5), {I _{t -5} , M from the memory bank _t-5 } (if t>5 and t≠10), {I _t-3 ,M _t-3 } (if t>3 and t≠8) and {I _t-1 ,M _t-1 } (if t>1 and t≠6) as the first reference frame. The electronic device uses the first reference frame whose interval from the first target frame is greater than the first preset duration as the long-term memory reference frame, and the remaining first reference frames as the short-term memory reference frame, and the total number of the first reference frames is k ; For example, the first preset duration is Δt=15. The content stored in the memory library is the video frame whose time is located before the first target frame and the target segmentation map of the video frame in the video to be divided.

The first target feature is a high-dimensional feature of the first target frame, and the first reference feature is a high-dimensional feature of the first reference frame. In this step, the electronic device determines the first target feature of the first target frame, the implementation manner of the first reference feature of the first reference frame, and the electronic device determines the first sample target feature of the sample target frame in step 201 and The implementation manner of the first sample reference feature of the first sample reference frame is similar, and details are not described herein again.

In step 402, the first target feature is weighted based on the first non-local attention of the first target frame to obtain a second target feature.

Among them, the first non-local attention is the similarity matrix. This step can be achieved through the following steps (1) to (2), including:

(1) Determine the first non-local attention of the first target frame.

The implementation of this step includes the following steps A1-A3:

A1: Normalize the first non-local attention to obtain the first weight.

This step is similar to step A1 in step 202, and is not repeated here.

A2: Perform matrix multiplication processing on the first weight and the first target feature through the target segmentation model to obtain a third target feature.

This step is similar to step A2 in step 202, and is not repeated here.

A3: Superimpose the third target feature and the first target feature to obtain the second target feature.

This step is similar to step A3 in step 202, and will not be repeated here.

(2) Based on the first non-local attention, weight the first sample target feature to obtain the second sample target feature.

This step is similar to step (2) in step 202 and will not be repeated here.

In step 403, the first reference feature is weighted based on the second non-local attention of the first reference frame to obtain a second reference feature.

This step is similar to step 203 and will not be repeated here.

In the embodiment of the present disclosure, by performing dimension reduction processing on the first sample target feature and the first sample reference feature, the amount of calculation in obtaining non-local attention is reduced, and the The non-local attention of a sample target feature and the dimension-reduced first sample reference feature makes the pixels in the frame stimulate each other, promotes the flow of semantic information between the pixels in the frame, and speeds up the initial target segmentation model. training speed.

In step 404, based on the second target feature and the second reference feature, determine first offset information, where the first offset information is used to indicate the offset between the first pixel point corresponding to the position and the second pixel point moving, the first pixel is a pixel in the first target frame, and the second pixel is a pixel in the first reference frame.

In some embodiments, step 404 includes the following steps A1-A2:

A1: Splicing the second target feature and the second reference feature to obtain the first feature.

This step is similar to step A1 in step 204 and will not be repeated here.

A2: Input the first feature into an offset prediction model to obtain the first offset information.

This step is similar to step A2 in step 204 and will not be repeated here.

In the embodiment of the present disclosure, since the second target feature and the second reference feature are spliced, the high-level information of the first target frame and the first reference frame is fused, and then the spliced first feature is input into the offset prediction model, The model is used to predict the semantic and temporal correlation offset of the two. Since the value of this offset is a real value, the sampling position is not fixed, and it is easier to hit the target pixel. Moreover, since the neural network is used for positioning, the amount of computation is small, thereby reducing the memory occupation of the graphics processor.

In step 405, an offset process is performed on the second reference feature based on the first offset information to obtain a second reference feature after offset, and the second reference feature after offset and the second target feature are determined between the first local attention.

The offset processing operation in this step is a warp operation. Wherein, based on the first offset information, the electronic device performs offset processing on the second reference feature to obtain the shifted second reference feature. The implementation manner is similar to the implementation manner in which the electronic device determines the third local attention in step 207, and details are not repeated here.

It should be noted that, when the interval between the first reference frame and the first target frame is greater than the first preset duration, the electronic device performs the operations of steps 402-405; In the case where the interval between the first target frames is not greater than the first preset duration, the second local attention between the first reference feature and the first target feature and the reference segmentation map are input into the target segmentation model to obtain the first target segmentation map of the first target frame.

In the embodiment of the present disclosure, when the sample target frame is the second video frame in the video to be segmented, the first reference frame obtained by the model is the first video frame, that is, the number of the first reference frame is 1, that is, there is only one short-term memory reference frame, and based on the target segmentation model, only two local attentions need to be determined to determine the target segmentation map of the first target frame, thereby reducing the time required for model prediction, and also The efficiency of object segmentation is improved.

In step 406, using the target segmentation model, perform sliding window extraction on each third pixel point in the reference segmentation map to obtain a plurality of first neighborhood maps corresponding to each third pixel point. A plurality of first neighborhood maps are spliced to obtain a reference segmentation map after sliding window extraction.

This step is similar to step 206 and will not be repeated here.

In step 407, dimension transformation is performed on the reference segmentation map extracted by the sliding window and the first local attention, respectively, to obtain the dimension-transformed reference segmentation map and the dimension-transformed first local attention. The reference segmentation map of is the same as the dimension of the first local attention after this dimension transformation.

Since the number of the first reference frame is at least one, the number of the reference segmentation map and the first local attention after the sliding window extraction is at least one, then the reference segmentation map after the sliding window extraction and the first local attention are obtained. Before the local attention is dimensionally transformed, the electronic device needs to splicing at least one reference segmentation map extracted by the sliding window, and splicing at least one first local attention; correspondingly, this step consists of formula ten and formula ten. One to achieve:

Formula 10: A＝concat(A ₁ ,A ₂ ,...,A _k )

Formula eleven:

Among them, A∈R ^{k×P×P×H×W} ,

In this step, the electronic device performs dimension transformation on the reference segmentation map extracted by the sliding window and the first local attention, respectively, to obtain the reference segmentation map after dimension transformation and the implementation manner and steps of the first local attention after dimension transformation. 206 is similar and will not be repeated here.

For example, the first local attention is the size of k×P×P×H×W, the reference segmentation map after sliding window extraction is the size of C×(k×P ² )×H×W, and the first image after dimension transformation The local attention is of size (k×P ² )×H×W.

In step 408, a first target segmentation map of the first target frame is determined based on the dimension-transformed reference segmentation map and the dimension-transformed first local attention.

In some embodiments, step 408 is implemented by the following steps A1-A3:

A1: For each fourth pixel in the dimension-transformed reference segmentation map, determine multiple similarities from the dimension-transformed first local attention, where the multiple similarities are the difference between the fourth pixel and the similarity between other pixels.

This step is similar to step A1 in step 209 and will not be repeated here.

A2: Based on the multiple degrees of similarity, weighted summation is performed on the pixel values of the fourth pixel point to obtain a weighted and summed pixel value.

This step is similar to step A2 in step 209 and will not be repeated here.

A3: Modify the pixel value of the fourth pixel point to the weighted and summed pixel value to obtain a first target segmentation map of the first target frame.

This step is similar to step A3 in step 209 and will not be repeated here.

Steps A1-A3 are implemented by Equation 12:

Formula twelve:

为第一目标分割图。in,

Segment the map for the first target.

In the embodiment of the present disclosure, based on the first local attention after dimension transformation, the first target frame is combined with the first reference frame, the response of semantic and timing information between frames is captured, and the result obtained through the sliding window extraction operation With reference to the segmentation map, a first target segmentation map of the first target frame is determined, so that the accuracy of the target segmentation is high.

In the embodiment of the present disclosure, since the feature of the video frame can reflect the semantic information of the video frame, based on the target feature of the first target frame and the reference feature of the first reference frame, the determined first offset information has been fused with Semantic information of two video frames, so that the first offset information can accurately represent the offset of each pixel corresponding to the position from the first reference frame to the first target frame, so as to predict based on the first offset information The target segmentation map of the first target frame is more accurate, thereby improving the accuracy of video target segmentation.

Fig. 5 is a block diagram of a video object segmentation apparatus according to an exemplary embodiment. 5 , the apparatus 50 includes: a first determination unit 501 , a first weighting unit 502 , a second determination unit 503 , a first offset unit 504 and a first target segmentation unit 505 .

The first determining unit 501 is configured to perform determining a first target feature of a first target frame, a first reference feature of a first reference frame, and a reference segmentation map of the first reference frame, where the first target frame is to be segmented The video frame without target segmentation in the video, the first reference frame is the video frame in which the target segmentation has been performed, and the interval between the first reference frame and the first target frame is greater than the first preset duration;

The first weighting unit 502 is configured to perform a first non-local attention based on the first target frame, weight the first target feature to obtain a second target feature, and a second non-local attention based on the first reference frame. Local attention, weighting the first reference feature to obtain the second reference feature;

The second determining unit 503 is configured to determine first offset information based on the second target feature and the second reference feature, where the first offset information is used to represent the first pixel point and the second pixel corresponding to the position The offset between points, the first pixel is a pixel in the first target frame, and the second pixel is a pixel in the first reference frame;

The first offset unit 504 is configured to perform offset processing on the second reference feature based on the first offset information to obtain a second reference feature after offset, and determine the second reference feature after offset the first local attention with the second target feature;

The first object segmentation unit 505 is configured to input the reference segmentation map and the first local attention into an object segmentation model to obtain a first object segmentation map of the first target frame.

In some embodiments, the second determining unit 503 is configured to perform splicing of the second target feature and the second reference feature to obtain a first feature; input the first feature into an offset prediction model to obtain the first offset information.

In some embodiments, the first target segmentation unit 505 includes:

The first sliding window extraction subunit is configured to perform sliding window extraction for each third pixel point in the reference segmentation map through the target segmentation model, to obtain a plurality of first pixel points corresponding to each third pixel point. a neighborhood graph, splicing the plurality of first neighborhood graphs to obtain a reference segmentation graph after sliding window extraction;

The first dimension transformation subunit is configured to perform dimension transformation on the reference segmentation map after the sliding window extraction and the first local attention, respectively, to obtain the dimension-transformed reference segmentation map and the dimension-transformed first local attention force, the reference segmentation map after the dimension transformation is the same as the dimension of the first local attention after the dimension transformation;

The determining subunit is configured to perform determining a first target segmentation map of the first target frame based on the dimension-transformed reference segmentation map and the dimension-transformed first local attention.

In some embodiments, the determining subunit is configured to perform, for each fourth pixel point in the dimension-transformed reference segmentation map, determining a plurality of similarities from the dimension-transformed first local attention , the multiple degrees of similarity are the degrees of similarity between the fourth pixel point and other pixel points; based on the multiple degrees of similarity, the pixel values of the fourth pixel point are weighted and summed to obtain the weighted and summed pixels value; modify the pixel value of the fourth pixel point to the weighted and summed pixel value to obtain the first target segmentation map of the first target frame.

In some embodiments, the first weighting unit 502 is configured to perform a normalization process on the first non-local attention to obtain a first weight; the first weight and the first weight are obtained through the target segmentation model The target feature is subjected to matrix multiplication processing to obtain a third target feature; the third target feature is superimposed with the first target feature to obtain the second target feature.

In some embodiments, the apparatus further includes:

The second target segmentation unit is configured to perform, under the condition that the interval between the first reference frame and the first target frame is not greater than the first preset duration, the first reference feature and the first target feature Between the second local attention and the reference segmentation map, input the target segmentation model to obtain the first target segmentation map of the first target frame.

In some embodiments, the apparatus further includes:

The acquisition unit is configured to perform acquiring the sample target frame and the first sample reference frame without target segmentation from the sample video, input the sample target frame and the first sample reference frame into the initial target segmentation model, and obtain the sample The first sample target feature of the target frame and the first sample reference feature of the first sample reference frame, and the interval between the sample target frame and the first sample reference frame is greater than the second preset duration and less than the third preset duration;

a second weighting unit configured to perform a third non-local attention based on the sample target frame, weight the first sample target feature to obtain a second sample target feature, and perform a third sample target feature based on the first sample reference frame The fourth non-local attention weights the first sample reference feature to obtain the second sample reference feature;

a third determining unit, configured to determine second offset information based on the second sample target feature and the second sample reference feature, where the second offset information is used to indicate that the first sample pixel point corresponding to the position and the The offset between the second sample pixel points, the first sample pixel point is the pixel point in the sample target frame, and the second sample pixel point is the pixel point in the first sample reference frame;

a second offset unit, configured to perform offset processing on the second sample reference feature based on the second offset information to obtain the offset second sample reference feature;

A training unit, configured to perform training on the initial target segmentation model based on the first sample reference frame and a third local attention, to obtain the target segmentation model, where the third local attention is the offset Local attention between the two-sample reference feature and the second-sample target feature.

In some embodiments, the training unit includes:

The second sliding window extraction subunit is configured to perform sliding window extraction on each second sample pixel point in the first sample reference frame through the initial target segmentation model, to obtain the same pixel as each second sample pixel. a plurality of second neighborhood graphs corresponding to the points, and splicing the plurality of second neighborhood graphs to obtain a second sample reference frame;

The second dimension transformation subunit is configured to perform dimension transformation on the second sample reference frame and the third local attention respectively to obtain the second sample reference frame after dimension transformation and the third local attention after dimension transformation , the dimension of the second sample reference frame after the dimension transformation is the same as the dimension of the third local attention after the dimension transformation;

The training subunit is configured to perform training on the initial target segmentation model based on the dimension-transformed second sample reference frame and the dimension-transformed third local attention to obtain the target segmentation model.

In some embodiments, the training sub-unit is configured to perform, for each third sample pixel in the dimensionally transformed second sample reference frame, determining how much from the dimensionally transformed third local attention similarity degrees, the multiple degrees of similarity are the degrees of similarity between the third sample pixel point and other pixel points; based on the multiple degrees of similarity, the pixel values of the third sample pixel point are weighted and summed to obtain the weighted The pixel value after the summation; modify the pixel value of the third sample pixel point to the pixel value after the weighted summation to obtain the second target segmentation map of the sample target frame; input the sample target frame into the initial target segmentation In the model, based on the loss value between the output result of the initial target segmentation model and the second target segmentation map, the model parameters of the initial target segmentation model are iteratively updated to obtain the target segmentation model.

In the embodiment of the present disclosure, since the feature of the video frame can reflect the semantic information of the video frame, based on the target feature of the first target frame and the reference feature of the first reference frame, the determined first offset information has been fused with Semantic information of two video frames, so that the first offset information can accurately represent the offset of each pixel corresponding to the position from the first reference frame to the first target frame, so as to predict based on the first offset information The target segmentation map of the first target frame is more accurate, thereby improving the accuracy of video target segmentation.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

In an exemplary embodiment, an electronic device is provided that includes a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the above-described embodiments The training method of the medium target segmentation model.

In some embodiments, the electronic device may be provided as a server. FIG. 6 is a block diagram of a server 600 according to an exemplary embodiment. The server 600 may vary greatly due to different configurations or performances, and may include One or more processors (Central Processing Units, CPUs) 601 and one or more memories 602, wherein the memory 602 is used to store executable instructions, and the processor 601 is configured to execute the above executable instructions to implement the above The method embodiment provides a training method for a target segmentation model. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input/output interface for input and output, and the server may also include other components for implementing device functions, which will not be described here.

In some embodiments, the electronic device may be provided as a terminal, and FIG. 7 is a block diagram of a terminal 700 according to an exemplary embodiment. The terminal 700 may be: a smart phone, a tablet computer, a notebook computer, or a desktop computer. Terminal 700 may also be called user equipment, portable terminal, laptop terminal, desktop terminal, and the like by other names.

Generally, the terminal 700 includes: a processor 701 and a memory 702 .

In some embodiments, processor 701 includes one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. In some embodiments, the processor 701 adopts at least one of DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, programmable logic array) A hardware form is implemented. In some embodiments, the processor 701 also includes a main processor and a co-processor, and the main processor is a processor for processing data in a wake-up state, also referred to as a CPU (Central Processing Unit, central processing unit). ; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 701 is integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor 701 further includes an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is used to process computing operations related to machine learning.

In some embodiments, memory 702 includes one or more computer-readable storage media that are non-transitory. In some embodiments, memory 702 also includes high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, a non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction for execution by the processor 701 to achieve the video object provided by the method embodiments of the present disclosure segmentation method.

In some embodiments, the terminal 700 may optionally further include: a peripheral device interface 703 and at least one peripheral device. In some embodiments, the processor 701, the memory 702 and the peripheral device interface 703 are connected through a bus or signal line. In some embodiments, various peripheral devices are connected to the peripheral device interface 703 through a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704 , a display screen 705 , a camera assembly 706 , an audio circuit 707 , a positioning assembly 708 and a power supply 709 .

The peripheral device interface 703 may be used to connect at least one peripheral device related to I/O (Input/Output) to the processor 701 and the memory 702 . In some embodiments, processor 701, memory 702, and peripherals interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one of processor 701, memory 702, and peripherals interface 703 or The two are implemented on a separate chip or circuit board, which is not limited in this embodiment.

The radio frequency circuit 704 is used for receiving and transmitting RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals. The radio frequency circuit 704 communicates with the communication network and other communication devices via electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, radio frequency circuitry 704 includes: an antenna system, an RF transceiver, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, subscriber identity module cards, and the like. In some embodiments, radio frequency circuitry 704 communicates with other terminals via at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: World Wide Web, Metropolitan Area Network, Intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), wireless local area network and/or WiFi (Wireless Fidelity, Wireless Fidelity) network. In some embodiments, the radio frequency circuit 704 further includes a circuit related to NFC (Near Field Communication, short-range wireless communication), which is not limited in the present disclosure.

The display screen 705 is used to display a UI (User Interface). In some embodiments, the UI includes graphics, text, icons, video, and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to acquire touch signals on or above the surface of the display screen 705 . In some embodiments, the touch signal is input to the processor 701 as a control signal for processing. At this time, the display screen 705 is also used to provide virtual buttons and/or virtual keyboards, also called soft buttons and/or soft keyboards. In some embodiments, there is one display screen 705, which is arranged on the front panel of the terminal 700; in other embodiments, there are at least two display screens 705, which are respectively arranged on different surfaces of the terminal 700 or in a folded design; In some embodiments, the display screen 705 is a flexible display screen disposed on a curved or folded surface of the terminal 700 . Even, the display screen 705 is also set as a non-rectangular irregular figure, that is, a special-shaped screen. In some embodiments, the display screen 705 is made of materials such as LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light emitting diode).

The camera assembly 706 is used to capture images or video. In some embodiments, camera assembly 706 includes a front-facing camera and a rear-facing camera. Usually, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth-of-field camera to realize the background blur function, the main camera It is integrated with the wide-angle camera to achieve panoramic shooting and VR (Virtual Reality, virtual reality) shooting functions or other integrated shooting functions. In some embodiments, camera assembly 706 also includes a flash. In some embodiments, the flash is a single color temperature flash, and in some embodiments, the flash is a dual color temperature flash. Dual color temperature flash refers to the combination of warm light flash and cold light flash, which is used for light compensation under different color temperatures.

In some embodiments, the audio circuit 707 includes a microphone and a speaker. The microphone is used to collect the sound waves of the user and the environment, convert the sound waves into electrical signals, and input them to the processor 701 for processing, or to the radio frequency circuit 704 to realize voice communication. For the purpose of stereo collection or noise reduction, in some embodiments, there are multiple microphones, which are respectively disposed in different parts of the terminal 700 . In some embodiments, the microphones are array microphones or omnidirectional acquisition microphones. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. In some embodiments, the loudspeaker is a conventional thin-film loudspeaker, and in some embodiments, the loudspeaker is a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for distance measurement and other purposes. In some embodiments, the audio circuit 707 also includes a headphone jack.

The positioning component 708 is used to locate the current geographic location of the terminal 700 to implement navigation or LBS (Location Based Service, location-based service). In some embodiments, the positioning component 707 is a positioning component based on the GPS (Global Positioning System) of the United States, the Beidou system of China or the Galileo system of Russia.

The power supply 709 is used to power various components in the terminal 700 . In some embodiments, the power source 709 is alternating current, direct current, a disposable battery, or a rechargeable battery. When the power source 709 includes a rechargeable battery, the rechargeable battery is a wired rechargeable battery or a wireless rechargeable battery. Wired rechargeable batteries are batteries that are charged through wired lines, and wireless rechargeable batteries are batteries that are charged through wireless coils. The rechargeable battery is also used to support fast charging technology.

In some embodiments, the terminal 700 also includes one or more sensors 710 . The one or more sensors 710 include, but are not limited to, an acceleration sensor 711 , a gyro sensor 712 , a pressure sensor 713 , a fingerprint sensor 714 , an optical sensor 715 and a proximity sensor 716 .

In some embodiments, the acceleration sensor 711 detects the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 700 . For example, the acceleration sensor 711 is used to detect the components of the gravitational acceleration on the three coordinate axes. In some embodiments, the processor 701 controls the display screen 705 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 711 . In some embodiments, the acceleration sensor 711 is also used for game or user movement data collection.

In some embodiments, the gyroscope sensor 712 detects the body direction and rotation angle of the terminal 700 , and the gyroscope sensor 712 cooperates with the acceleration sensor 711 to collect 3D actions of the user on the terminal 700 . The processor 701 can implement the following functions according to the data collected by the gyro sensor 712 : motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

In some embodiments, the pressure sensor 713 is disposed on the side frame of the terminal 700 and/or the lower layer of the display screen 705 . When the pressure sensor 713 is disposed on the side frame of the terminal 700 , it can detect the user's holding signal of the terminal 700 , and the processor 701 performs left and right hand identification or shortcut operations according to the holding signal collected by the pressure sensor 713 . When the pressure sensor 713 is disposed on the lower layer of the display screen 705 , the processor 701 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 705 . The operability controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

The fingerprint sensor 714 is used to collect the user's fingerprint, and the processor 701 identifies the user's identity according to the fingerprint collected by the fingerprint sensor 714 , or the fingerprint sensor 714 identifies the user's identity according to the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 701 authorizes the user to perform related sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. In some embodiments, the fingerprint sensor 714 is disposed on the front, back, or side of the terminal 700 . When the terminal 700 is provided with a physical button or a manufacturer's logo, the fingerprint sensor 714 is integrated with the physical button or the manufacturer's logo.

Optical sensor 715 is used to collect ambient light intensity. In one embodiment, the processor 701 controls the display brightness of the display screen 705 according to the ambient light intensity collected by the optical sensor 715 . Specifically, when the ambient light intensity is high, the display brightness of the display screen 705 is increased; when the ambient light intensity is low, the display brightness of the display screen 705 is decreased. In another embodiment, the processor 701 also dynamically adjusts the shooting parameters of the camera assembly 706 according to the ambient light intensity collected by the optical sensor 715 .

A proximity sensor 716 , also called a distance sensor, is usually provided on the front panel of the terminal 700 . The proximity sensor 716 is used to collect the distance between the user and the front of the terminal 700 . In one embodiment, when the proximity sensor 716 detects that the distance between the user and the front of the terminal 700 is gradually decreasing, the processor 701 controls the display screen 705 to switch from the bright screen state to the off screen state; when the proximity sensor 716 detects When the distance between the user and the front of the terminal 700 gradually increases, the processor 701 controls the display screen 705 to switch from the closed screen state to the bright screen state.

Those skilled in the art can understand that the structure shown in FIG. 7 does not constitute a limitation on the terminal 700, and can include more or less components than the one shown, or combine some components, or adopt different component arrangements.

In an exemplary embodiment, a storage medium including instructions, such as a memory 602 or 702 including instructions, is also provided, and the above-mentioned instructions can be executed by the processor 601 or 701 to complete the above-mentioned video object segmentation method. Alternatively, the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage equipment, etc.

In an exemplary embodiment, a computer program product is also provided, when the instructions in the computer program product are executed by the processor of the electronic device, the electronic device can execute the video object segmentation method in the above embodiment.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. A method for segmenting video objects, comprising:

determining a first target feature of a first target frame, a first reference feature of a first reference frame and a reference segmentation map of the first reference frame, wherein the first target frame is a video frame which is not subjected to target segmentation in a video to be segmented, the first reference frame is a video frame which is subjected to target segmentation in the video, and the interval between the first reference frame and the first target frame is greater than a first preset duration;

weighting the first target feature based on the first non-local attention of the first target frame to obtain a second target feature, and weighting the first reference feature based on the second non-local attention of the first reference frame to obtain a second reference feature;

determining first offset information based on the second target feature and the second reference feature, wherein the first offset information is used for representing offset between a first pixel point and a second pixel point corresponding to positions, the first pixel point is a pixel point in the first target frame, and the second pixel point is a pixel point in the first reference frame;

performing offset processing on the second reference feature based on the first offset information to obtain an offset second reference feature, and determining a first local attention between the offset second reference feature and the second target feature;

inputting the reference segmentation map and the first local attention into a target segmentation model to obtain a first target segmentation map of the first target frame.

2. The method of claim 1, wherein determining first offset information based on the second target feature and the second reference feature comprises:

splicing the second target characteristic and the second reference characteristic to obtain a first characteristic;

and inputting the first characteristic into an offset prediction model to obtain the first offset information.

3. The method of claim 1, wherein said inputting the reference segmentation map and the first local attention into a target segmentation model to obtain a first target segmentation map for the first target frame comprises:

performing sliding window extraction on each third pixel point in the reference segmentation graph through the target segmentation model to obtain a plurality of first neighborhood graphs corresponding to each third pixel point, and splicing the plurality of first neighborhood graphs to obtain a reference segmentation graph after the sliding window extraction;

performing dimension transformation on the reference segmentation graph after the sliding window extraction and the first local attention respectively to obtain a reference segmentation graph after the dimension transformation and the first local attention after the dimension transformation, wherein the dimension of the reference segmentation graph after the dimension transformation is the same as that of the first local attention after the dimension transformation;

determining a first target segmentation map of the first target frame based on the dimension-transformed reference segmentation map and the dimension-transformed first local attention.

4. The method of claim 3, wherein determining the first target segmentation map for the first target frame based on the dimension-transformed reference segmentation map and the dimension-transformed first local attention comprises:

for each fourth pixel point in the reference segmentation map after the dimension transformation, determining a plurality of similarity degrees from the first local attention after the dimension transformation, wherein the similarity degrees are the similarity degrees between the fourth pixel point and other pixel points;

based on the similarity, carrying out weighted summation on the pixel value of the fourth pixel point to obtain a weighted-summation pixel value;

and modifying the pixel value of the fourth pixel point into the weighted and summed pixel value to obtain a first target segmentation graph of the first target frame.

5. The method of claim 1, wherein weighting the first target feature based on the first non-local attention of the first target frame to obtain a second target feature comprises:

normalizing the first non-local attention to obtain a first weight;

performing matrix multiplication processing on the first weight and the first target characteristic through the target segmentation model to obtain a third target characteristic;

and superposing the third target feature and the first target feature to obtain the second target feature.

6. The method of claim 1, further comprising:

and under the condition that the interval between the first reference frame and the first target frame is not more than the first preset duration, inputting the second local attention between the first reference feature and the first target feature and the reference segmentation map into the target segmentation model to obtain a first target segmentation map of the first target frame.

7. The method of claim 1, wherein the object segmentation model is trained by:

obtaining a sample target frame and a first sample reference frame which are not subjected to target segmentation from a sample video, inputting the sample target frame and the first sample reference frame into an initial target segmentation model, obtaining a first sample target feature of the sample target frame and a first sample reference feature of the first sample reference frame, wherein the interval between the sample target frame and the first sample reference frame is longer than a second preset duration and shorter than a third preset duration;

weighting the first sample target feature based on a third non-local attention of the sample target frame to obtain a second sample target feature, and weighting the first sample reference feature based on a fourth non-local attention of the first sample reference frame to obtain a second sample reference feature;

determining second offset information based on the second sample target feature and the second sample reference feature, wherein the second offset information is used for representing offset between a first sample pixel point and a second sample pixel point corresponding to the second sample target feature, the first sample pixel point is a pixel point in the sample target frame, and the second sample pixel point is a pixel point in the first sample reference frame;

performing offset processing on the second sample reference feature based on the second offset information to obtain an offset second sample reference feature;

training the initial target segmentation model based on the first sample reference frame and a third local attention to obtain the target segmentation model, wherein the third local attention is a local attention between the shifted second sample reference feature and the second sample target feature.

8. A video object segmentation apparatus, comprising:

a first determining unit, configured to perform determination of a first target feature of a first target frame, a first reference feature of a first reference frame and a reference segmentation map of the first reference frame, where the first target frame is a video frame of a video to be segmented, which is not subjected to target segmentation, the first reference frame is a video frame of the video, which is subjected to target segmentation, and an interval between the first reference frame and the first target frame is greater than a first preset duration;

a first weighting unit configured to perform weighting of the first target feature based on a first non-local attention of the first target frame to obtain a second target feature, and weighting of the first reference feature based on a second non-local attention of the first reference frame to obtain a second reference feature;

a second determining unit configured to perform determining first offset information based on the second target feature and the second reference feature, where the first offset information is used to represent an offset between a first pixel point and a second pixel point corresponding to a position, where the first pixel point is a pixel point in the first target frame, and the second pixel point is a pixel point in the first reference frame;

a first offset unit configured to perform offset processing on the second reference feature based on the first offset information, obtain an offset second reference feature, and determine a first local attention between the offset second reference feature and the second target feature;

a first target segmentation unit configured to perform inputting the reference segmentation map and the first local attention into a target segmentation model resulting in a first target segmentation map of the first target frame.

9. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the video object segmentation method of any one of claims 1 to 7.

10. A storage medium having instructions which, when executed by a processor of an electronic device/server, enable the electronic device to perform a video object segmentation method as claimed in any one of claims 1 to 7.

Images