CN113869154B - Video actor segmentation method according to language description - Google Patents

Abstract

The invention discloses a video actor segmentation method according to language description. The method utilizes cascaded cross-modal attention modules, uses clip-level visual features to roughly focus on the information words of language queries, and then uses frame-level visual features to Fine-tune the attention of words, fine-tune the weight of the language in the target frame, can identify and segment positive examples in the rich video information, and mine difficult negative examples through comparative learning, can learn to identify the target actor from the video, and can also be used in different Distinguish it in the video, significantly improving the accuracy of intra-frame matching and segmentation.

Description

A Video Actor Segmentation Method Based on Language Description

technical field

The invention relates to the technical field of video recognition, in particular to a method for segmenting video actors according to language description.

Background technique

Video understanding tasks have received a lot of attention in recent years, especially those involving natural language processing. In this field, great achievements have been made in the tasks of language-selective temporal action localization, video caption generation, and segmentation of actors and actions in videos based on sentence descriptions. In real-world scenarios, it is common for a video to contain multiple actors in action. Therefore, selectively fine-grained localization of a specific actor and its actions in space and time through linguistic queries becomes an important task for computers to better understand videos.

A widely used framework in related tasks, such as video/image object grounding, is to generate region proposals in videos/images by some detection methods, and then match text features with proposed visual features to select the best object proposal as a match Object. To improve the performance of matching two heterogeneous features, previous work first utilized bidirectional LSTM and self-attention mechanism to generate linguistic features, then used weighted text features to process visual features, and finally performed text-visual feature matching. But the linguistic attention learned by this self-attention mechanism is actually an average solution of the training data, rather than focusing on the individual solutions of a certain video. In this way, during inference, regardless of the input video, the linguistic features of interest are identified, and since video is a high-level semantic space containing rich content, it is difficult to grasp the most discriminative features of video. Video thus determines the key in language queries, capturing informative words and learning discriminative language representations for visual perception are crucial for language-guided video actor-action segmentation tasks.

How to design a visual perception language encoder and generate discriminative language, so as to segment the actors and their actions in the video, it is necessary to further optimize the segmentation method and improve the accuracy of intra-frame matching and segmentation.

SUMMARY OF THE INVENTION

In order to overcome the above problems, the present invention provides a method for segmenting video actors and their actions based on language descriptions, wherein the cooperative optimization network using cascaded cross-modal attention mechanisms significantly improves the accuracy of matching and segmentation . Using visual features from two perspectives to focus on language from coarse to fine, to generate language features with discriminative visual perception. In addition, it is equipped with a comparative learning method to design a mining strategy for difficult negative examples, which is beneficial for the network to identify positive examples from negative examples. example, and further improve the performance, thereby completing the present invention.

The purpose of the first aspect of the present invention is to provide a video actor segmentation method according to language description, the method utilizes cascaded cross-modal attention modules to generate discriminative sentence query features, and improves the accuracy of matching and segmentation. sex.

The cascaded cross-modal attention module includes a clip-level feature attention unit and a frame-level feature attention unit.

The clip-level feature attention unit takes as input the sentence embedding s and the clip-level feature _vc of the target frame j.

The clip-level feature attention unit uses the clip-level feature _vc to roughly weight the language features, and obtain:

F ₁ =Att ₁ ·ψ( _vc )+φ(s)

和ψ(·)处理后得到

和ψ(v_c)；将组合单词嵌入e_t形成句子嵌入s，接着将其放入一个卷积层φ(·)中以生成句子特征φ(s)。Among them, T is the matrix transpose; σ _softmax is the softmax activation function; Att ₁ is the attention map of the clip feature _vc and sentence embedding s; F ₁ is the _coarsely weighted sentence feature;

and ψ( ) after processing to get

and ψ(v _c ); combine word embeddings e _t to form sentence embeddings s, which are then put into a convolutional layer φ( ) to generate sentence features φ(s).

For video V, the clip-level features _vc are encoded by:

表示L₂范数，θ_avg为均值池化操作，I3D(·)为双流I3D编码器，优选地，使用I3D网络的Mixed-4f层的输出作为I3D编码器。in,

denotes the L2 norm, _θavg is the mean pooling operation, _I3D ( ) is the dual-stream I3D encoder, preferably, the output of the Mixed-4f layer of the I3D network is used as the I3D encoder.

The frame-level feature attention unit processes the roughly weighted sentence features F ₁ and frame-level features v _f to obtain fine-tuned sentence features F ₂ :

F ₂ =Att ₂ ·ψ′(v _f )+F ₁

为v_f经过一个线性层

得到的特征；ψ′(v_f)为v_f经过一个线性层ψ′(·)得到的特征。Among them, Att ₂ is the attention map of the frame-level feature v _f and the coarsely weighted sentence feature F ₁ ; F ₂ represents the fine-tuned sentence feature, and each column of which represents a word vector;

go through a linear layer for v _f

The obtained feature; ψ′(v _f ) is the feature obtained by v _f through a linear layer ψ′(·).

The frame-level features v _f are extracted by using the ResNet-101 network. Preferably, before extracting the frame-level features, pre-training is performed on the COCO dataset, and fine-tuning is performed on the A2D dataset training segmentation;

The frame-level feature is a linear weighted combination of the warped feature from frame j to frame i and the original feature v _i , as shown in the following formula:

Among them, vi is the ResNet-101 coding feature for the target frame _i , β is the weight coefficient, i is the target frame, j is the reference frame, 2K is the number of frames of the reference frame (that is, take K frames in the front of the target frame, and then Take K frames to compensate the target frame features), v _j→i is the distortion feature from the reference frame j to the target frame i;

The _vj→i is:

为双线性扭曲方程。Among them, v _j is the ResNet-101 encoding feature of reference frame j; OF _j→i is the optical flow between reference frame j and target frame i;

is the bilinear distortion equation.

The weighted word feature h _t goes through a fully connected layer to obtain the sentence query feature q:

m _t =FC(h _t )

α _t =σ _softmax (m _t )

Among them, h _t is the t-th column of the F ₂ feature, representing the vector of the t-th word; FC(h _t ) is a fully connected layer; m _t is the intermediate vector of h _t through a fully-connected layer; α _t is the th Weighting coefficients for t words.

A second aspect of the present invention also provides a computer-readable storage medium, which stores a training program for segmentation of video actors according to language description, and when the program is executed by a processor, causes the processor to execute the video according to language description. The steps of the actor segmentation method.

The language-described video actor segmentation method described in the present invention can be implemented by means of software plus a necessary general hardware platform, and the software is stored in a computer-readable storage medium (including ROM/RAM, magnetic disk, optical disk) , including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, a network device, etc.) execute the method of the present invention.

A third aspect of the present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a training program for segmentation of video actors according to language description, and when the program is executed by the processor, the processor executes the Describe the steps of a video actor segmentation method according to language description.

The beneficial effects of the present invention include:

(1) The method provided in the present invention uses clip-level visual features to roughly pay attention to the information words of language queries, and then uses frame-level visual features to fine-tune the attention of words, fine-tune the weight of the language in the target frame, and significantly improve the intra-frame Matching and segmentation accuracy.

(2) The method provided in the present invention can identify and segment positive examples in rich video information, and mine difficult negative examples through comparative learning, can learn to identify the target actor from the video, and can also distinguish it in different videos.

(3) Using the visual feature encoder in the present invention to extract clip-level features, frame-level features, and visual features of positive and negative examples to obtain sentence query features q, and distinguish positive and negative examples, positive examples, and difficult negative examples through comparative learning , which strengthens the discriminative ability of cascaded cross-modal attention modules.

Description of drawings

1 shows a schematic diagram of the network structure of a video actor segmentation method described by language according to the present invention;

FIG. 2 shows an effect diagram of a method for segmentation of video actors according to language description applied to an A2D data set in Embodiment 1 of the present invention.

Detailed ways

The present invention will be further described in detail below through the accompanying drawings and embodiments. The features and advantages of the present invention will become more apparent from these descriptions. Therein, although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The present invention provides a video actor segmentation method according to language description, which utilizes cascaded cross-modal attention modules to generate discriminative sentence query features and improves the accuracy of matching and segmentation.

The cascaded cross-modal attention module includes a clip-level feature attention unit and a frame-level feature attention unit.

In the present invention, since the video clip feature is a global feature containing all information, first, the clip-level visual feature attention unit is used to roughly focus on the informative words of the language query, such as words reflecting motion and temporal changes. Then, frame-level feature attention units are used to fine-tune word attention, fine-tuning the weights of sentences in specific frames, e.g., to reflect appearance properties, significantly improve the accuracy of intra-frame matching and segmentation. In the method, a clip-level feature attention unit and a frame-level feature attention unit are used to learn a discriminative language representation based on a given video.

In the cascaded cross-modal attention module, the clip-level feature attention unit takes the sentence embedding s and the clip-level feature _vc of the target frame i as input, and obtains the coarsely weighted sentence feature F ₁ .

In the present invention, the clip-level feature _vc is extracted by using a dual-stream I3D encoder. Preferably, ImageNet and Kinetics are used to pre-train the dual-stream I3D encoder before extracting the clip-level feature.

The dual-stream I3D encoder is specifically described in the document "Carreira J, Zisserman A. Quo vadis, actionrecognition a new model and the kinetics dataset[C]//proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6299-6308." described in.

The ImageNet is specifically described in the document "Russakovsky O, Deng J, Su H, et al. Imagenet largescale Visual recognition challenge [J]. International journal of computervision, 2015, 115(3): 211-252.".

The Kinetics are specifically described in the document "Kay W, Carreira J, Simonyan K, et al. The kineticshuman action video dataset[J]. arXiV preprint arXiv: 1705.06950, 2017.".

For video V, the clip-level features _vc are encoded by:

为L₂范数(即指向量各元素的平方和再求平方根)，θ_avg为均值池化操作。I3D(·)为双流13D编码器，优选地，使用I3D网络的Mixed-4f层的输出作为I3D编码器。in,

is the L ₂ norm (that is, the squared sum of each element of the pointing vector and then the square root), and θ _avg is the mean pooling operation. I3D(·) is a dual-stream 13D encoder, preferably, the output of the Mixed-4f layer of the I3D network is used as the I3D encoder.

The Mixed-4f layer of the I3D network is specifically described in the document "Carreira J, Zisserman A. Quo vadis, action recognition a new model and the kinetics dataset [C]//proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6299 -6308."

和ψ(·))以分别生成两个新的视觉特征

和ψ(v_j)。同时，通过组合单词嵌入e_t形成句子嵌入s，e_t为第t个单词的嵌入特征。接着将其放入一个卷积层φ(·)中以生成句子特征φ(s)。其中，t∈[1，n]，且t为整数，n为单词总数；e_t具体如文献“MIKOLOV T，SUTSKEVER I，CHEN K，etal.Distributed representations of words and phrases and theircompositionality[C]//Advances in neural information processing systems，2013：3111-3119.”中所述。In the present invention, the visual feature vj of the target frame _i is first sent into two convolutional layers (

and ψ( )) to generate two new visual features respectively

and ψ(v _j ). At the same time, sentence embedding s is formed by combining word embeddings _t , which is the embedding feature of the _t -th word. It is then put into a convolutional layer φ( ) to generate sentence features φ(s). Among them, t∈[1,n], and t is an integer, and n is the total number of words; e _{t is} specifically as in the document "MIKOLOV T, SUTSKEVER I, CHEN K, etal. Distributed representations of words and phrases and their compositionality[C]// Advances in neural information processing systems, 2013: 3111-3119.".

乘以φ(s)以生成逐词空间位置的注意图：said

Multiply by φ(s) to generate an attention map of word-wise spatial locations:

Among them, T is the matrix transpose, σ _softmax is the softmax function, and Att is the attention map that measures the spatial position of each word.

The Att is multiplied by ψ(v _j ), plus φ(s) to obtain visually weighted sentence features:

F=Att·ψ(v _j )+φ(s)

where F is the visually weighted sentence feature.

In the present invention, the clip-level feature attention unit uses the clip-level feature _vc to roughly weight the language features, and obtain respectively:

F ₁ =Att ₁ ·ψ( _vc )+φ(s)

where T is the matrix transpose, σ _softmax is the softmax function, Att ₁ is the attention map of the clip feature _vc and sentence embedding s, and F ₁ is the coarsely weighted sentence feature.

In the present invention, the frame-level feature attention unit processes the roughly weighted sentence feature F ₁ and the frame-level feature v _f to obtain a fine-tuned sentence feature F ₂ .

The frame-level features v _f are extracted by using the ResNet-101 network. Preferably, before extracting the frame-level features, the ResNet-101 network is pre-trained on the COCO dataset, and fine-tuned on the A2D dataset training segmentation.

The ResNet-101 network is specifically described in the document "He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778." Described in; the COCO dataset is specifically described in the document "Lin T Y, Maire M, Belongie S, et al. Microsoft coco: Common objects incontext [C]//European conference on computer vision. Springer, Cham, 2014: 740- 755.”; the A2D dataset is specifically described in the document “Xu C, Hsieh S H, Xiong C, et al. Can humans fly action understanding with multiple classes of actors[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015: 2264-2273.".

In the present invention, in order to enhance the feature map, the optical flow between 2K frames and reference frames j near the target frame i is calculated, that is, K frames are taken in the forward direction of the target frame i, and K frames are taken in the backward direction, as reference frames, and based on the light The stream warps the ResNet-101 encoded features of the nearby 2K frames onto the target frame, where K is the number of frames, according to the following formula:

为双线性扭曲方程；v_j→i为从参考帧j到目标帧i的扭曲特征。上述操作具体如文献“ZHU X，WANG Y，DAI J，et al.Flow-guided feature aggregation for Video objectdetection[C]//Proceedings of the IEEE International Conference on ComputerVision.2017：408-417.”中所述。Among them, v _j is the ResNet-101 encoding feature of reference frame j; OF _j→i is the optical flow between reference frame j and target frame i;

is the bilinear warping equation; v _j→i is the warping feature from the reference frame j to the target frame i. The above operation is specifically described in the document "ZHU X, WANG Y, DAI J, et al.Flow-guided feature aggregation for Video objectdetection[C]//Proceedings of the IEEE International Conference on ComputerVision.2017:408-417." .

The frame-level feature is a linear weighted combination of the warped feature from frame j to frame i and the original feature v _i , as shown in the following formula:

Among them, vi is the ResNet-101 encoding feature of the target frame _i , β is the weight coefficient, i is the target frame, j is the reference frame, 2K is the number of reference frames, that is, take K frames in the forward direction of the target frame, and backward Take K frames to compensate the target frame features. In this way, the frame-level features _vf are enhanced by nearby frame-level features.

In the present invention, the enhanced frame-level features _vf are directly put into the Region Proposal Network (RPN) to generate proposals. Since the target frame has been compensated by its nearby frames, moving objects can be detected more easily. Preferably, the Regional Proposal Network (RPN) is as detailed in the document "He K, Gkioxari G, Dollár P, et al. Mask r-cnn [C]//Proceedings of the IEEE international conference on computer vision. 2017: 2961-2969 .", preferably, the Region Proposal Network (RPN) is pre-trained on the COCO dataset and fine-tuned on the A2D dataset training branch cut.

The frame-level feature attention unit processes the roughly weighted sentence features F ₁ and frame-level features v _f to obtain fine-tuned sentence features F ₂ :

F ₂ =Att ₂ ·ψ′(v _f )+F ₁

为v_f经过一个线性层

得到的特征；ψ′(v_f)为v_f经过一个线性层ψ′(·)得到的特征。Among them, Att ₂ is the attention map of the frame-level feature v _f and the coarsely weighted sentence feature F ₁ ; F ₂ represents the fine-tuned sentence feature;

go through a linear layer for v _f

The obtained feature; ψ′(v _f ) is the feature obtained by v _f through a linear layer ψ′(·).

The weighted word feature h _t goes through a fully connected layer to obtain the sentence query feature q:

m _t =FC(h _t )

α _t =σ _softmax (m _t )

Among them, h _t is the t-th column of the F ₂ feature, representing the vector of the t-th word; FC(h _t ) is a fully connected layer, which is as detailed in the document "Krizhevsky A, Sutskever I, Hinton G E. ImageNet Classification with Described in DeepConvolutional Neural Networks[C]//Advances in Neural Information ProcessingSystems.2012."; m _t is the intermediate vector of h _t after a fully connected layer; α _t is the weighting coefficient of the t-th word; e _t is the th-th word Embedding features of t words, as in the document "MIKOLOV T, SUTSKEVER I, CHEN K, etal. Distributed representations of words and phrases and their compositionality[C]//Advances in neural information processing systems, 2013: 3111-3119." said.

In a preferred embodiment of the present invention, during training, the method for segmenting video actors based on language description further includes using sentence query feature q to perform comparative learning to distinguish positive examples from difficult negative examples. The positive example is the region proposal with the highest IoU value on the target frame, and the true value IoU (intersection-union ratio) is greater than 0.5. The hard negative example is the region proposal on the target frame with an IoU less than 0.5 with the ground truth. The IoU is specifically described in the document "He K, Gkioxari G, Dollár P, etal.Mask r-cnn[C]//Proceedings of the IEEE international conference oncomputer vision.2017:2961-2969."

Noting that only using hard negative examples in the same target frame is not enough to learn the ability of sentence query feature q to match the desired result from the video region proposal, a hard negative example mining strategy is designed in the present invention, which is divided into two parts. The first part, mining hard negative examples from the video containing the target frame: from other key frames in the video, find the region proposals that are different from the ground truth of the current frame (that is, the region proposals with IoU less than 0.5) as hard negative examples. The second part, mining hard negative examples from different videos: when the hard negative examples are insufficient, continuously find videos that contain the same actor-action label, and find the key frames of these videos that are different from the actor-action label The region proposed as a hard negative example. During testing, from the region proposals on the target frame, finding the region proposal with the highest similarity score with the obtained sentence query feature q is the matching result.

In the present invention, it is necessary to segment the actors and their actions that can match the sentence query feature q from the video containing rich information. Therefore, in the method, the sentence query feature q is used for comparative learning, and the difficult negative examples are mined to obtain Strengthen the matching ability between q and region proposals.

In the present invention, after the sentence embedding s is processed by the clip-level feature attention unit and the frame-level feature attention unit, the fine _- tuned sentence feature F2 is obtained, and then the sentence query feature q is obtained by weighting with the word embedding. The r ⁺ is the visual feature of the positive example, and r _l ^- is the visual feature of the negative example, wherein, l=1, 2,...,L, where L is the number of negative examples, and L is greater than or equal to 1 Integer, preferably an integer of 5-50, more preferably 15-35, such as 25. The loss function of the sentence query feature q is:

where T is the matrix transpose.

Among them, the loss function Loss utilizes (L+1) tuples to optimize the identification of positive examples from L negative examples, reducing the possibility of the network converging to a local optimum.

The visual feature r ⁺ of the positive example or the visual feature _r l- ^of the negative example is extracted by using the RPN network, that is, the region proposal network, to generate region proposals, and its backbone network is the ResNet-101 network. Preferably, before extracting features, the ResNet-101 network and the region proposal network (RPN) are pre-trained on the COCO dataset and fine-tuned on the A2D dataset training branch. The region proposal has object feature o and location feature l.

The target feature o is the feature after passing through the ROI-Aligned module of the RPN network, preferably, the dimension of the sentence query feature q is consistent, such as a 1024-dimensional vector. The ROI-Aligned module is specifically described in "Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick. Mask r-cnn[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2961-2969.".

The location feature l is:

where (x _tl , y _tl ) and (x _br , y _br ) are the coordinates of the upper-left and lower-right corners of the region proposal, respectively, and w, h, W, and H are the width, height, and target of the region proposal, respectively The width and height of the frame.

The visual feature r of the positive example ⁺ or the visual feature r _l of the negative example ^- by concatenating the location feature l to the target feature o, and then passing the concatenated feature to a fully connected layer to obtain a positive image with the same dimension as the text representation The visual feature r ⁺ of the example or the visual feature r _l ^- of the negative example, as shown in the following formula:

r=σ _tanh (W([o;l]))

Among them, r is the visual feature r ⁺ of the positive example or the visual feature r _l ^- of the negative example, which is the C-dimensional target-level feature, [;] represents the connection operation, W is the parameter matrix to be learned, and σ _tanh is the tanh activation function.

The tanh activation function is specifically described in the document "LeCun Y, Bottou L, Bengio Y, et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE, 1998, 86(11): 2278-2324." described in.

In the present invention, after comparing and learning the sentence query feature q and positive/difficult negative examples, after obtaining a trained network, select the region proposal with the highest similarity with the text feature, and input the visual feature r ⁺ of the selected region proposal into the in the split branch to generate the resulting mask. The segmentation branch in the present invention adopts the segmentation branch of mask r-cnn, which is specifically as in the document "He K, Gkioxari G, Dollár P, et al.Mask r-cnn [C]//Proceedings of the IEEE international conference on computer Vision.2017:2961-2969.", the generated mask contains the entire entity and has a nice edge. A schematic diagram of the network structure of the method for segmenting video actors and their actions according to language descriptions in the present invention is shown in FIG. 1 .

A second aspect of the present invention also provides a computer-readable storage medium, which stores a training program for segmentation of video actors according to language description, and when the program is executed by a processor, causes the processor to execute the video according to language description. The steps of the actor segmentation method.

The language-described video actor segmentation method described in the present invention can be implemented by means of software plus a necessary general hardware platform, and the software is stored in a computer-readable storage medium (including ROM/RAM, magnetic disk, optical disk) , including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, a network device, etc.) execute the method of the present invention.

A third aspect of the present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a training program for segmentation of video actors according to language description, and when the program is executed by the processor, the processor executes the Describe the steps of a video actor segmentation method according to language description.

The video actor segmentation method according to the language description provided in the present invention firstly uses the clip-level visual features to roughly pay attention to the informative words of the language query, and then uses the frame-level visual features to fine-tune the attention of the words, which significantly improves the intra-frame matching and segmentation accuracy. Moreover, in this segmentation method, positive examples can be identified in videos that contain rich content, through the hard negative example mining strategy and the introduction of contrastive learning, it can not only learn to identify target actors from videos, but also in different videos. differentiate it. In this way, it has stronger discriminative ability for cross-model retrieval. Furthermore, a dedicated segmentation network is employed to segment the regions obtained in the matching, which makes the generated masks with better completeness and better edges.

Example

The present invention is further described below through specific examples, but these examples are only exemplary and do not constitute any limitation to the protection scope of the present invention.

Example 1

(1) Use the I3D network to train on the dataset ImageNet and the dataset Kinetics, and use the output of the Mixed-4f layer of the obtained I3D network as the I3D encoder.

The dataset ImageNet is specifically described in the document "Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, AdityaKhosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International Journal of computer vision 115, 3(2015), 211–252”.

The data set Kinetics is specifically described in the document "Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action videodataset. arXiv preprint arXiv:1705.06950 (2017)”.

Given a video V (specifically as shown in the video frame in Figure 1 or Figure 2), the clip feature v _c is encoded by the following formula:

表示L₂范数，θ_avg表示均值池化操作；v_c表示到得的剪辑特征，其为32×32×832的矩阵。in,

represents the L ₂ norm, θ _avg represents the mean pooling operation; _vc represents the resulting clipping feature, which is a 32×32×832 matrix.

(2) The ResNet-101 network is fine-tuned on the A2D dataset training split after pre-training on the COCO dataset. The ResNet-101 network is specifically described in the document "He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778." Described in; the COCO dataset is specifically described in the document "Lin T Y, Maire M, Belongie S, et al. Microsoft coco: Common objects incontext [C]//European conference on computer vision. Springer, Cham, 2014: 740- 755.”; the A2D dataset is specifically described in the document “Xu C, Hsieh S H, Xiong C, et al. Can humans fly action understanding with multiple classes of actors[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015: 2264-2273.".

Calculate the optical flow between 14 reference frames j near the target frame i (the forward 7 frames and the backward 7 frames of the target frame i are the reference frame j), and based on the optical flow, the ResNet-101 of the above 14 nearby reference frames j is calculated. The encoded features are warped onto the target frame by the following formula:

表示双线性扭曲方程，v_j→i表示从帧j到帧i扭曲的特征，具体操作如文献“ZHU X，WANG Y，DAI J，et al.Flow-guided feature aggregation for video object detection[C]//Proceedings of the IEEE International Conference on ComputerVision.2017：408-417.”中所述。where v _j is the ResNet-101 encoding feature for reference frame j, OF _j→i is the optical flow between reference frame j and target frame i,

Represents the bilinear warping equation, v _j→i represents the feature of warping from frame j to frame i, the specific operation is described in the literature "ZHU X, WANG Y, DAI J, et al.Flow-guided feature aggregation for video object detection[C ]//Proceedings of the IEEE International Conference on ComputerVision. 2017:408-417.”.

The frame-level feature v _f is the linear weighted combination of the warped feature from frame j to frame i and the original feature v _i , as shown in the following formula, where v _i is the ResNet-101 encoding feature for the target frame i:

where β is a weight coefficient of 0.1 and K is set to 7. The reference frame j is the first 7 frames and the last 7 frames of the target frame i, and the video sampling frame rate is 24fps. The frame-level feature vf is enhanced by nearby frame-level features by a linearly weighted combination of the warped features from frame j to frame _i and the original feature _vi . The enhanced features are then directly put into a Region Proposal Network (RPN) to generate proposals, which is as detailed in the document "Girshick R. Fast r-cnn[C]//Proceedings of the IEEE international conference on As described in computer vision. 2015: 1440-1448.", the region proposal network (RPN) is pre-trained on the COCO dataset and fine-tuned on the A2D dataset training branch cut. Since the target frame i has been compensated by its nearby frames, moving objects can be detected more easily.

Among them, v _f is a 16×16×1024-dimensional matrix.

知ψ(·)，分别生成两个新的视觉特征

和ψ(v_c)。(3) Use the clip-level feature attention unit to roughly weight the clip-level feature v _c to the language features, and send the clip-level feature v _c of the target frame i to the convolutional layer

Knowing ψ( ), two new visual features are generated respectively

and ψ( _vc ).

为1024维的1×1的卷积。所述卷积层ψ(·)为1024维的1×1的卷积。The convolutional layer

is a 1×1 convolution of 1024 dimensions. The convolution layer ψ(·) is a 1×1 convolution of 1024 dimensions.

和ψ(v_c)维度保持一致。Sentence embeddings s are formed by combining word embeddings e _t , which are then put into a 1024-dimensional fully connected layer to generate sentence features φ(s). where t∈[1,n], and t is an integer, and n is the total number of words. e _t is the embedding feature of the t-th word, which is a 300-dimensional vector obtained by the word2vec model trained on the Google News dataset, as shown in the document "Mikolov T, Sutskever I, Chen K, et al. Distributed representations of words andphrases and their compositionality[C].//In Advances in neural information processing systems 2013.3111-3119.”; the Google News dataset is specifically as described in the document “AS Das, M. Datar, A. Garg, and S. Rajaram, "Google news personalization: scalable online collaborative filtering," in Proceedings of the 16th internationalconference on World Wide Web. ACM, 2007, pp. 271-280." φ( ) is a fully connected layer whose role is to transform the 300-dimensional information to 1024-dimensional, and the visual features

and ψ( _vc ) dimension is consistent.

The clip-level feature attention unit uses the clip-level feature _vc to roughly weight the language features, and obtain:

F ₁ =Att ₁ ·ψ( _vc )+φ(s)

Among them, σ _softmax is the softmax activation function, Att ₁ is the attention map of clip feature v _c and sentence embedding s, and its dimension is 20×1024, where each element (a, b) represents the visual feature of the bth position. The influence of a word, F ₁ is the coarsely weighted sentence feature, and its dimension is 20×1024, which represents the feature of the word after the global feature weighting. The softmax activation function is specifically described in the document "LeCun Y, Bottou L, Bengio Y, et al.Gradient-basedlearning applied to document recognition[J].Proceedings of the IEEE, 1998, 86(11): 2278-2324." said.

Input the above roughly weighted sentence feature F ₁ and frame-level feature v _f to the frame-level feature attention unit to obtain the fine-tuned sentence feature F ₂ :

F ₂ =Att ₂ ·ψ′(v _f )+F ₁

为1024维的1×1的卷积；ψ′(v_f)为1024维的1×1的卷积。Among them, Att ₂ is the attention map of the frame-level feature v _f and the coarsely weighted sentence feature F ₁ , and F ₂ represents the fine-tuned sentence feature.

is a 1×1 convolution of 1024 dimensions; ψ′(v _f ) is a 1×1 convolution of 1024 dimensions.

(4) Embed the weighted word h _t of the fine-tuned sentence feature F ₂ to obtain the sentence query feature q:

m _t =FC(h _t )

α _t =σ _softmax (m _t )

Among them, h _t is the t-th column of the F ₂ feature, representing the vector of the t-th word; FC(h _t ) is a fully connected layer; m _t is the intermediate vector of h _t through a fully-connected layer; α _t is the th Weighting coefficients for t words.

(5) For the video V, the sentence embedding s, the sentence query feature q (as shown in step 4) is obtained through the above steps (1)-(4), of which 1 positive example visual feature is r ⁺ , 25 negative examples The visual features are r _l ⁻ , where l=1, 2, . . . , 25.

The loss function is:

where T is the matrix transpose.

(6) The visual feature r ⁺ of the positive example or the visual feature r _l of the negative example ^— uses the ResNet-101 network (the ResNet-101 network is specifically described in the document “He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C].//In Proceedings of the IEEE conference on computervision and pattern recognition. 2016, 770-778.”) for extraction. Before extracting features, the network is in the COCO dataset (specifically, the COCO dataset is described in the document "Lin T, Maire M, Belongie S, et al. Microsoft coco: Common objects in context[C].//In European conference on computer vision. 2014.740-755.”) and pre-trained on the A2D dataset (the A2D dataset is specifically described in the document “Xu C, Hsieh S, Xiong, C, et al. Can humans fly action understanding with multiple classes of actors [C].//In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.2264-2273.”) fine-tune on training segmentation to generate region proposals. The feature o of the region proposal is the target feature of the region output by the fine-tuned ResNet-01, and the feature l is the position feature.

The target feature o is the feature after passing through the ROI-Aligned module of the RPN network, and is specifically a 1024-dimensional vector (consistent with the dimension of the sentence query feature q).

The location feature l is:

where (x _tl , y _tl ) and (x _br , y _br ) are the coordinates of the upper-left and lower-right corners of the region proposal, respectively, and w, h, W, and H are the width, height, and target of the region proposal, respectively The width and height of the frame.

The visual feature r ⁺ of the positive example or the visual feature r _l ⁻ of the negative example is obtained by concatenating the location feature l to the target feature o, and then passing the concatenated feature to a fully connected layer to obtain a The visual feature r ⁺ of the positive example or the visual feature r _l ^- of the negative example, as shown in the following formula:

r=σ _tanh (W([o;l]))

Among them, r is the visual feature r ⁺ of the positive example or the visual feature r _l ^- of the negative example, which is the C-dimensional target-level feature, [;] represents the connection operation, and σ _tanh is the tanh activation function.

The tanh activation function is specifically described in the document "LeCun Y, Bottou L, Bengio Y, et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE, 1998, 86(11): 2278-2324." described in.

The visual feature r ⁺ of the positive example or the visual feature r _l ^- of the negative example is compared with the sentence query feature q, and the positive example that can match q is the positive example to be segmented, and the region proposal with the highest similarity to the text feature is obtained. , and input the visual feature r ⁺ proposed by the selected region into the segmentation branch of mask r-cnn to generate the resulting mask, and get the final result after segmentation.

Figure 2 is the effect diagram of the method in Embodiment 1 on the A2D data set, in which the text in the first row is the input sentence, the picture sequence in the second row is the input video, and the third row is obtained in this embodiment. The segmentation result, the real segmentation map given by the fourth row dataset. As can be seen from Fig. 2, the target actor can be completely segmented from the video frame by using the video actor segmentation method according to the language description provided in the present invention, and the segmentation result is close to the real segmentation map given by the data set degree is very high.

The present invention has been described in detail above with reference to specific embodiments and/or exemplary examples and accompanying drawings, but these descriptions should not be construed as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, various equivalent replacements, modifications or improvements can be made to the technical solutions of the present invention and the embodiments thereof, which all fall within the scope of the present invention. The scope of protection of the present invention is determined by the appended claims.

Claims (10)

1. a video actor segmentation method according to language description, is characterized in that, described method utilizes cascading cross-modal attention module to carry out, and described cascading cross-modal attention module comprises clip-level feature attention unit and Frame-level feature attention unit; The clip-level feature attention unit uses the clip-level feature vc to roughly weight the language features to obtain:

F ₁ =Att ₁ ·ψ( _vc )+φ(s) Among them, T is the matrix transpose; σ _softmax is the softmax activation function; Att ₁ is the attention map of the clip feature _vc and sentence embedding s; F ₁ is the _coarsely weighted sentence feature;

and ψ( ) after processing to get

and ψ(v _c ); combine word embeddings et _t to form sentence embedding s, which are then put into a convolutional layer φ( ) to generate sentence features φ(s); et is the embedding feature of the t-th word ; The frame-level feature attention unit processes the roughly weighted sentence features F ₁ and frame-level features v _f to obtain fine-tuned sentence features F ₂ :

F ₂ =Att ₂ ·ψ′(v _f )+F ₁ Among them, Att ₂ is the attention map of the frame-level feature v _f and the coarsely weighted sentence feature F ₁ ; F ₂ represents the fine-tuned sentence feature;

go through a linear layer for v _f

The obtained feature; ψ′(v _f ) is the feature obtained by v _f through a linear layer ψ′(·). 2 . The method of claim 1 , wherein the clip-level feature attention unit uses sentence embedding s and clip-level features _vc of target frame i as input. 3 . 3. method according to claim 1, is characterized in that, for video V, clip-level feature v _c is encoded by following formula:

in,

represents the L ₂ norm, θ _avg is the mean pooling operation, and I3D( ) is the dual-stream I3D encoder. 4. The method of claim 1, wherein The frame-level features v _f are extracted using the ResNet-101 network. 5. The method according to claim 4, wherein, before extracting frame-level features, the ResNet-101 network is pre-trained on the COCO data set, and fine-tuned on the A2D data set training segmentation; The frame-level feature is a linear weighted combination of the warped feature from frame j to frame i and the original feature v _i , as shown in the following formula:

Among them, vi is the ResNet-101 encoding feature of the target frame _i , β is the weight coefficient, i is the target frame, j is the reference frame, 2K is the number of frames of the reference frame, K frames are taken in the forward direction of the target frame, and the backward direction is Take K frames to compensate the target frame features, v _j→i is the distortion feature from the reference frame j to the target frame i; The _vj→i is:

Among them, v _j is the ResNet-101 encoding feature of reference frame j; OF _j→i is the optical flow between reference frame j and target frame i;

is the bilinear distortion equation. 6. The method according to claim 1, wherein the weighted word feature _ht passes through a fully connected layer to obtain the sentence query feature q: m _t =FC(h _t ) α _t =σ _softmax (m _t )

Among them, h _t is the t-th column of the F ₂ feature, representing the vector of the t-th word; FC(h _t ) is a fully connected layer; m _t is the vector obtained by h _t after passing through a fully connected layer; α _t is Weighting factor for the t-th word. 7. The method according to one of claims 1 to 6, wherein the method further comprises using sentence query feature q to perform comparative learning to distinguish positive examples from difficult negative examples, wherein the positive examples are on the target frame. , and the region proposal with the true value IoU greater than 0.5 and the largest IoU value, the hard negative example is the region proposal on the target frame, and the true value IoU is less than 0.5. 8. The method according to one of claims 1 to 6, characterized in that, In the method, the loss function of the sentence query feature q is:

Among them, T is the matrix transpose, the r ⁺ is the visual feature of the positive example, r _l ^- is the visual feature of the negative example, where l=1, 2,...,L, where L is the number of negative examples number; The visual feature r ⁺ of the positive example or the visual feature r _l ^- of the negative example is: r=σ _tanh (W([o;l])), r is the visual feature r ⁺ of the positive example or the visual feature r _l ^- of the negative example, the r is the C-dimensional target-level feature, [;] represents the connection operation, W is the parameter matrix to be learned, and σ _tanh is the tanh activation function ; The visual feature r ⁺ of the positive example or the visual feature r _l - of the negative example is extracted by using the RPN network to generate a region proposal, the target feature of which is o, and the position feature corresponding to the region is 1; The target feature o is the feature extracted and generated by the RPN network, The location feature l is:

where (x _tl , y _tl ) and (x _br , y _br ) are the coordinates of the upper-left and lower-right corners of the region proposal, respectively, and w, h, W, H are the width and height of the region proposal, the target The width and height of the frame. 9. A computer-readable storage medium, characterized in that, it stores a training program for segmentation of video actors according to language description, and when the program is executed by a processor, the processor is made to execute one of claims 1 to 8 The steps of the video actor segmentation method according to the language description. 10. A computer device, characterized in that it comprises a memory and a processor, wherein the memory stores a training program segmented according to a video actor described in a language, and when the program is executed by the processor, the processor executes the program as described in the right. The steps of the video actor segmentation method according to the language description described in one of claims 1 to 8.

Images