CN112734803B - Single target tracking method, device, equipment and storage medium based on character description - Google Patents

Abstract

The invention discloses a single target tracking method, a device, equipment and a storage medium based on character description, wherein the single target tracking method comprises the following steps: dividing a video to be tracked into a plurality of video packets according to a set frame number on average; extracting first, second and third character features from the character description; respectively extracting first, second and third visual features from the nth sampling frame of each video packet; updating the first, second and third character features respectively based on the first, second and third visual features of the nth sampling frame of each video packet to obtain updated first, second and third character features; respectively extracting a fourth visual characteristic, a fifth visual characteristic and a sixth visual characteristic from the sample plate image of the target to be tracked; extracting seventh, eighth and ninth visual features from the search area image respectively; respectively fusing the updated first, second and third character feature vectors with the fourth to ninth visual features to obtain fused features; and obtaining a target tracking result of each frame in the current video packet of the video to be tracked according to the fusion characteristics.

Description

Single target tracking method, device, equipment and storage medium based on text description

technical field

The present application relates to the technical fields of machine vision and natural language processing, and in particular to a single target tracking method, device, equipment and storage medium based on text description.

Background technique

The statements in this section merely mention the background art related to this application, and do not necessarily constitute the prior art.

Single object tracking is a classic and long-term research topic in the field of machine vision. Traditional single-target tracking methods usually manually mark the box of the target to be tracked in the frame of the video. In recent years, related topics that combine machine vision and natural language processing technology, such as image/video labeling, visual question answering, etc., have made great progress, and single-target tracking based on text descriptions has also received more and more attention. Given a piece of text annotation, tracking the target marked with text in the video can enable the algorithm to better handle many complex scenarios, such as occlusion, frame offset, target deformation, blurring, etc. Because the semantic information provided by the natural language description can help the target tracking algorithm to alleviate the impact of these complex scenes.

However, the task of single-object tracking based on textual descriptions poses a special problem. Natural language can describe the appearance and movement state of the target in the first frame, or describe the movement process of the target in the entire video. It is not feasible to label each frame of the video. For the commonly used single-target tracking datasets with natural language annotations, text annotations usually describe the overall content of the video, and no datasets annotate all frames. However, the location and appearance of objects are constantly changing in the video, so natural language annotations cannot accurately describe the location or motion of objects in most scenarios. Although past related works have achieved good performance, they only regard text annotation as a global constraint.

Contents of the invention

In order to solve the deficiencies of the prior art, this application provides a single target tracking method, device, equipment and storage medium based on text description;

In the first aspect, the present application provides a single target visual tracking method based on text description;

Single target visual tracking method based on text description, including:

Obtain the sample image of the target to be tracked; obtain the video to be tracked and the text description related to the target to be tracked; divide the video to be tracked into several video packets on average according to the set number of frames;

extracting first, second and third character features from the text description;

The first, second and third visual features are respectively extracted for the nth sampling frame of each video packet; n is a positive integer, and the upper limit of n is a specified value; based on the first, second and third visual features of the nth sampling frame of each video packet The second and third visual features update the first, second, and third text features respectively to obtain the updated first, second, and third text features; for the sample image of the target to be tracked, extract the fourth and third text features respectively. The fifth and sixth visual features; the template image of the target to be tracked refers to the first frame image of the video to be tracked; for the search area image, the seventh, eighth and ninth visual features are respectively extracted; the search area image is Refers to all images in the current video package;

Fusion the updated first, second and third text feature vectors with the fourth, fifth, sixth, seventh, eighth and ninth visual features respectively to obtain six fusion features;

According to the six fusion features, the target tracking result of each frame in the current video packet of the video to be tracked is obtained.

In the second aspect, the present application provides a single-target visual tracking device based on text description;

Single target visual tracking device based on text description, including:

The video packet division module is configured to: obtain a sample image of the target to be tracked; obtain a video to be tracked and a text description related to the target to be tracked; divide the video to be tracked into several video packets on average according to the set number of frames;

A text feature extraction module configured to: extract first, second and third text features from the text description;

A visual feature extraction module, which is configured to: extract the first, second and third visual features for the nth sampling frame of each video packet; n is a positive integer, and the upper limit of n is a specified value; based on each video The first, second and third visual features of the nth sampling frame are respectively updated to the first, second and third text features to obtain the updated first, second and third text features; the target to be tracked The model image of the target, extract the fourth, fifth and sixth visual features respectively; the model image of the target to be tracked refers to the first frame image of the video to be tracked; for the search area image, extract the seventh, eighth and ninth respectively Visual features; the search area image refers to all images in the current video package;

The feature fusion module is configured to: fuse the updated first, second and third character feature vectors with the fourth, fifth, sixth, seventh, eighth and ninth visual features respectively to obtain Six fusion features;

The output module is configured to: obtain the target tracking result of each frame in the current video packet of the video to be tracked according to the six fusion features.

In a third aspect, the present application also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the above one or more The computer program is stored in the memory, and when the electronic device is running, the processor executes one or more computer programs stored in the memory, so that the electronic device executes the method described in the first aspect above.

In a fourth aspect, the present application also provides a computer-readable storage medium for storing computer instructions, and when the computer instructions are executed by a processor, the method described in the first aspect is completed.

In the fifth aspect, the present application also provides a computer program (product), including a computer program, which is used to implement the method of any one of the aforementioned first aspects when running on one or more processors.

Compared with prior art, the beneficial effect of the present application is:

It is proposed to use the depth visual features of the search area generated during the tracking process to update the depth features of the text description, in order to expect that the text features of the depth can change with the changes of the target in the video, and improve the accuracy of the single target tracking algorithm.

Advantages of additional aspects of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The accompanying drawings constituting a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application, and do not constitute improper limitations to the present application.

Fig. 1 is the method flowchart of the first embodiment;

Fig. 2 is the method flowchart of the first embodiment;

Fig. 3(a)-Fig. 3(g) are schematic diagrams showing the effect of the first embodiment.

Detailed ways

It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that the terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive Comprising, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include steps or units not explicitly listed or for these processes, methods, Other steps or units inherent in a product or equipment.

In the case of no conflict, the embodiments and the features in the embodiments of the present invention can be combined with each other.

Embodiment one

This embodiment provides a single target visual tracking method based on text description;

Single target visual tracking method based on text description, including:

S101: Obtain a sample image of the target to be tracked; acquire a video to be tracked and a text description related to the target to be tracked; divide the video to be tracked into several video packets on average according to the set number of frames;

S102: Extracting first, second and third text features from the text description;

S103: extract the first, second and third visual features respectively for the nth sampling frame of each video packet; n is a positive integer, and the upper limit of n is a specified value;

Based on the first, second and third visual features of the nth sampling frame of each video packet, the first, second and third text features are respectively updated to obtain the updated first, second and third text features ;

The template image of the target to be tracked, extracting the fourth, fifth and sixth visual features respectively; the sample image of the target to be tracked refers to the first frame image of the video to be tracked;

For the search area image, extract the seventh, eighth and ninth visual features respectively; the search area image refers to all images in the current video package;

S104: Fusing the updated first, second, and third character feature vectors with the fourth, fifth, sixth, seventh, eighth, and ninth visual features respectively to obtain six fused features;

S105: According to the six fusion features, obtain the target tracking result of each frame in the current video packet of the video to be tracked.

Exemplarily, the video to be tracked is equally divided into several video packets according to the set number of frames; for example, 1000 frames of video to be tracked are divided into 10 video packets on average in units of 100 frames; another example, 100 video frames to be tracked The frame video is divided into 10 video packets on average in units of 10 frames.

As one or more embodiments, the S102: extract the first, second and third text features from the text description; the specific steps include:

Using the BERT method, the first, second and third text features are extracted from the text description.

As one or more embodiments, said S103: extract the first, second and third visual features respectively for the nth sampling frame of each video packet; n is a positive integer, and the upper limit of n is a specified value; specific steps include:

Using RestNet-50, perform visual feature extraction on the nth sampling frame of each video packet;

The convolutional layer Conv2_3 outputs the first visual feature;

The convolutional layer Conv3_4 outputs the second visual feature;

The convolutional layer Conv5_3 outputs the third visual feature.

As one or more embodiments, said S103: update the first, second and third text features based on the first, second and third visual features of the nth sampling frame of each video packet, and obtain the updated The first, second, and third character features after that; the specific steps include:

The first visual feature is processed by Global Average Pooling (GAP) to obtain the first sub-visual feature; the first text feature is used as the initial hidden state of the first LSTM model; at the set time t, the first sub-visual feature is The visual feature is input into the first LSTM model, and the first LSTM model outputs the updated first text feature; in the first LSTM model, the forget gate is used to determine whether the hidden state at the current moment should be discarded; the input gate is used to determine the input whether the value of the visual feature should be written;

The second visual feature is processed by global average pooling to obtain the second sub-visual sub-feature; the second text feature is used as the initial hidden state of the second LSTM model; at the set time t, the second sub-visual feature is input to the second sub-visual feature In the second LSTM model, the second LSTM model outputs the updated second character feature;

The third visual feature is processed by global average pooling to obtain the third sub-visual feature; the third text feature is used as the initial hidden state of the third LSTM model; at the set time t, the third sub-visual feature is input to the third In the LSTM model, the third LSTM model outputs the updated third character feature.

As one or more embodiments, said S103: Extracting the fourth, fifth and sixth visual features respectively from the template image of the target to be tracked; the sample image of the target to be tracked refers to the first frame image of the video to be tracked; To search area image, extract the seventh, the eighth and the ninth visual feature respectively; Described search area image refers to all images in the current video package; Concrete steps include:

Use RestNet-50 to extract visual features from the sample image of the target to be tracked;

The convolutional layer Conv2_3 of RestNet-50 outputs the fourth visual feature;

The convolutional layer Conv3_4 of RestNet-50 outputs the fifth visual feature;

The convolutional layer Conv5_3 of RestNet-50 outputs the sixth visual feature.

Use RestNet-50 to extract visual features from the image of the search area of the target to be tracked;

The convolutional layer Conv2_3 of RestNet-50 outputs the seventh visual feature;

The convolutional layer Conv3_4 of RestNet-50 outputs the eighth visual feature;

The convolutional layer Conv5_3 of RestNet-50 outputs the ninth visual feature.

As one or more embodiments, said S104: perform the updated first, second and third character feature vectors with the fourth, fifth, sixth, seventh, eighth and ninth visual features respectively Fusion, six fusion features are obtained; the specific steps include:

Splicing the updated first character feature vector with the fourth visual feature to obtain the first fusion feature;

Splicing the updated second character feature vector with the fifth visual feature to obtain the second fusion feature;

Splicing the updated third character feature vector with the sixth visual feature to obtain the third fusion feature;

Splicing the updated first character feature vector with the seventh visual feature to obtain the fourth fusion feature;

Splicing the updated second character feature vector with the eighth visual feature to obtain the fifth fusion feature;

The updated third character feature vector is spliced with the ninth visual feature to obtain the sixth fusion feature.

As one or more embodiments, said S105: Obtain the target tracking result of each frame in the current video packet of the video to be tracked according to the six fusion features; the specific steps include:

Input the first fusion feature into the first convolutional neural network CNN, input the output value of the first convolutional neural network and the output value of the fourth convolutional neural network into the first classification network; obtain the first classification result ;

Inputting the fourth fusion feature into the fourth convolutional neural network CNN, inputting the output value of the fourth convolutional neural network and the output value of the first convolutional neural network into the first regression network; obtaining the first regression result ;

The second fusion feature is input in the second convolutional neural network CNN, the output value of the second convolutional neural network and the output value of the fifth convolutional neural network are all input in the second classification network; obtain the second classification result ;

The fifth fusion feature is input into the fifth convolutional neural network CNN, the output value of the fifth convolutional neural network and the output value of the second convolutional neural network are all input into the second regression network; the second regression result is obtained ;

The third fusion feature is input into the third convolutional neural network CNN, the output value of the third convolutional neural network and the output value of the sixth convolutional neural network are all input into the third classification network; the third classification result is obtained ;

Input the sixth fusion feature into the sixth convolutional neural network CNN, input the output value of the sixth convolutional neural network and the output value of the third convolutional neural network into the third regression network; obtain the third regression result ;

Fusing the first classification result, the second classification result and the third classification result to obtain the final classification result;

Fusing the first regression result, the second regression result and the third regression result to obtain the final regression result;

According to the final classification result and the final regression result, the target tracking result of each frame in the current video packet of the video to be tracked is obtained.

The core of the method proposed in this application is a text feature update module based on Long-Short Term Memory (LSTM). The text feature update module uses the depth features of the initial text description as the initial hidden state, and sets The frame number input depth feature of the current frame is used to update the text feature as a hidden state, so that it is expected that the depth text feature will change accordingly when the target moves or the appearance changes in the video. Then, the updated deep text features are fused with the next deep visual features for a set number of frames. This application uses the SiamRPN method to detect the target of each frame based on the fusion feature.

Tracking algorithms in the past usually use detection or matching methods, and they randomly select some positive and negative samples from the data set during the training process. However, in order to update deep text features, time series issues must be taken into account. Therefore, this application uses a serialized training method to train the feature update module, and divides each video into the same number of segments, and the number of frames in each segment can be different.

The main contributions of this application are as follows: a text feature update module is proposed to reduce the gap between text expression and visual information such as the position and appearance of the target. A serialized training method is proposed to train the text feature update module to achieve the expectation of updating deep text features.

Single-target tracking using manually marked target frames is a long-term challenge in the field of machine vision. Researchers have proposed many single-target tracking algorithms, among which the representative ones are algorithms based on correlation filtering (Correlation Filter, CF) and algorithms based on recurrent neural networks. Network (Recurrent Neural Network, RNN) algorithm. In recent years, matching network-based siamese structures have attracted increasing attention due to their accuracy and efficiency. SiamFC, SiamRPN, SiamRPN++, SiamMask and other algorithms based on Siamese networks.

In recent years, the research on single target tracking algorithm based on text description has received more and more attention, but most of the algorithms regard text description as the global constraint of single target tracking, and ignore the limitations of text description.

Given a video and a text annotation related to the tracked target, the purpose of this application is to track the current target in the video. The main challenge in most scenarios is that text annotations cannot accurately describe the position and appearance changes of tracked objects in different frames. In order to solve this problem, this application proposes a tracking algorithm including two modules: a feature update module and a tracking module, and the details of the two modules will be described next.

Feature update module: The purpose of the feature update module is to reduce the limitation of text description in single target tracking and make the updated deep text features better reflect the state of the tracked target. The feature update module proposed in this application realizes the task of feature update by using a set of LSTM networks.

The feature update module consists of three parallel LSTM units. First, use the BERT (BidirectionalEncoding Representations from Transformers) method to encode text into a 768-dimensional feature vector, then use a fully connected network to fully connect the text feature vector to 512 dimensions, and then use the text features as the initial hidden state of the LSTM unit at the initial moment , at a specific moment t, the way LSTM updates the hidden state is as follows:

f _t ＝σ(ω _f [l _t-1 ,v _t ]+b _f )

i _t =σ(ω _i [l _t-1 ,v _t ]+b _i )

l _t ＝f _t Θl _t-1 +i _t Θtanh(ω _l v _t +b _l )

where _lt and _vt denote the hidden state of LSTM initialized with textual features and the visual features of input LSTM, respectively. f _t and _it denote the forget gate and input gate of the LSTM cell, respectively. The forget gate determines whether the value of the hidden state at the current moment should be discarded, and the input gate determines whether the value of the deep visual feature input at the current moment should be written. ω and b denote the weight and bias parameters of the trainable gate operation. σ and Θ represent sigmoid activation function and Hamilton operation.

At time t, LSTM inputs deep visual features to process the hidden state l _t-1 . By initializing the LSTM hidden state with textual features and updating the hidden state through input and forget gate operations, deep textual features can change as the location and appearance of the tracked object change.

The three parallel LSTM networks proposed in this application input serialized deep visual features to update the hidden state initialized with text features to update the text features and make the text features change with the location and appearance of the tracking target. The visual information of video can efficiently expand and enrich deep text features.

In the structure of the Siamese network, the template image of the target and the search area image of the current frame are extracted through the ResNet50 network and output three visual features of different depths, and then the three visual features of different depths are input to three In a parallel LSTM network, the deep text features updated through the LSTM network will change with the visual features.

Tracking module: The tracking module proposed in this application finds the area with high similarity with the template image in the search area image by inputting the template image containing the target and the search area image as the result of the tracking algorithm. Different from the traditional Siamese network that pre-cuts and fills the search area image, this application does not cut the original image but fills the original image to the size of the standard input. In most scenarios, maintaining the size of the original image can maintain the association between the location information of the target and the text annotation. The sample images used in the training process come from the manual annotation of the data set, and the visual grounding method is used to obtain the sample images in the test phase.

As shown in Figure 1, similar to the update module, the deep visual features of the template image and the search region image are extracted by the same ResNet50 network, and then the depth features of the template image and the search region image are fused with the updated text features. The updated text feature is fully connected to a 256-dimensional feature vector, and then the 1×1×256 one-dimensional feature vector is stacked into a 7×7×256 dimension and a 31×31×256 dimension (the 7×7 dimension and the template image The dimension of the feature is the same, the dimension of 31×31 is the same as the dimension of the image feature of the search area), and then the text feature and visual feature are connected together for fusion. Fusion features utilize visual information to further reduce the ambiguity in language description, which can improve the target perception ability of visual features. Next, the fused features are processed using a Convolutional Neural Network (CNN). Finally, the fused features are input to the candidate region network in the Siamese network structure to detect and track objects. The output of the classification branch and the regression branch of the candidate area network is the foreground and background classification of the detection frame and the regression of the target frame. Similar to traditional Siamese networks, we use binary cross-entropy loss and smooth L1 loss.

The classification loss is as follows:

where _yi denotes the foreground and background prediction of the Nth candidate region.

A _x , A _y , A _w , A _h respectively represent the x, y axis coordinates, width and height of the center point of the candidate box, T _x , _Ty , T _w , T _h represent the coordinates and width of the real tracking target frame High, the standard distance in four dimensions is as follows

Then the regression loss is as follows:

The total loss is L _total = L _cls + λL _reg , where λ denotes a hyperparameter that balances classification and regression losses.

Application details: During the training process, the video is divided into 50 clips and packages containing natural language annotations, and the frame size in each package is resized to 255×255 for updating the module and the search area image of the Siamese network, which is different from the traditional Compared with the image operation of the twin network clipping search area, using the original image better maintains the consistency of the target and natural language annotation in the image. At the same time, the template image containing the tracking target is input as an example into the Siamese network.

Text annotations are encoded into feature vectors by BERT and fully connected networks, and used to initialize the hidden state of the LSTM network. Then, the updating module updates the textual features as the hidden state according to the deep visual features to improve its object perception ability in re-searching image sequences. The updated deep text feature is fused with the template image and the search area image, and the final fused feature is used for the Siamese network to predict the location of the tracking target.

This application uses the modified ResNet50 network and pre-trains on the ImageNet dataset. This model uses the Momentum optimizer, the decay rate is 1×10 ^-4 and the momentum is set to 0.9, the initial learning rate is 5×10 ^-3 and each round of training is reduced by 1×10 ^-4 , and the training batch size is 32. Each video is cut into 50 video clips, that is, each video is updated with 50 deep text features. This model is trained for 5, 10, 15, 20 rounds and tested.

During the application process, this application uses the visual grounding method to generate template images. Visualgrounding predicts a box corresponding to the text content in the image through text annotation. When the frame of the tracking target is available, the template image can be cut out from the first frame of the video through this frame. At the same time, the visual grounding method is also used to restore the tracking results after the tracking algorithm loses the target.

Experimental results: Next, display and analyze the experimental results. First, the data set and evaluation criteria used in the experiment will be shown, as well as some details of the application. Then the comparison results with traditional methods are presented. This application also analyzes the model under different settings and tries to explain the model performance and working principle. The experiments in this application are run on an Inter Xeon CPU E5-2687W v33.10GHz and an NVIDIA Tesla V100 GPU.

The data sets used in the experiment are the LaSOT data set and the Lingual OTB99 data set, because each video in these two data sets has text annotations. The LaSOT dataset is a large-scale benchmark dataset for single-target tracking, containing 1400 video sequences, each video has a natural language annotation and each frame has a target box. The dataset has 1120 videos for training and 280 videos for testing. Because the main purpose of the text annotation in the LaSOT dataset is to assist the tracking process, and the text annotation is not accurate enough to describe the target, so this application modifies some text annotations to reduce the ambiguity of the text. The Lingual OTB99 data set is an extended version of the OTB100 data set, and a text is marked for each video. The dataset contains 51 training videos and 48 testing videos.

Same as the traditional tracking algorithm, this application uses accuracy and success rate as the evaluation criteria of the tracking algorithm. Accuracy represents the percentage of frames whose coincidence degree between the predicted target box and the real target box exceeds a given threshold. The success rate indicates the percentage of the number of frames whose intersection ratio of the predicted target box and the real target box is higher than a certain threshold.

Compared with single-target tracking algorithms with text descriptions, this application compares these algorithms under two initialization conditions, one is to use only the given text annotation for single-target tracking, and the other is to use the target of the first frame at the same time Box and text labels are initialized. As shown in the table, the algorithm of the present application performs better than the traditional algorithm in LaSOT and LingualOTB99 under the two initialization methods.

Part of the tracking results are shown in Figure 2. With the assistance of the tracking module, the performance of this model is better than many algorithms that use text annotation initialization and the target box initialization of the first frame. The following performance is robust and can recover to track the correct target after the target goes out of view and tracks the wrong target. This application also compares this model with other tracking algorithms that are only initialized with the object box of the first frame. As shown in Table 1, the results of this model using only text annotation initialization are competitive with the algorithm using the target box initialization of the first frame. This model performs better than using the target box initialization and text annotation initialization. The initial tracking algorithm is good. Conclusion: In the single-target tracking task, usually a concise text annotation of a video can describe the state of the target in the first frame of the video or the movement of the target in the entire video instead of its exact position and position in each frame. Appearance, since these properties of the object may change continuously from frame to frame. This application proposes a brand-new feature update module for single-target visual tracking algorithms based on text descriptions, using LSTM networks to update deep text features, and using updated deep text features and deep visual features to improve the performance of single-target tracking algorithms. Experimental results show that text description can assist in improving the single-target tracking algorithm and achieve better single-target tracking performance. Fig. 3(a)-Fig. 3(g) are schematic diagrams showing the effect of the first embodiment.

Table 1 Experimental results data comparison table

Embodiment two

This embodiment provides a single target visual tracking device based on text description;

Single target visual tracking device based on text description, including:

The video packet division module is configured to: obtain a sample image of the target to be tracked; obtain a video to be tracked and a text description related to the target to be tracked; divide the video to be tracked into several video packets on average according to the set number of frames;

A text feature extraction module configured to: extract first, second and third text features from the text description;

A visual feature extraction module, which is configured to: extract the first, second and third visual features for the nth sampling frame of each video packet; n is a positive integer, and the upper limit of n is a specified value; based on each video The first, second and third visual features of the nth sampling frame are respectively updated to the first, second and third text features to obtain the updated first, second and third text features; the target to be tracked The model image of the target, extract the fourth, fifth and sixth visual features respectively; the model image of the target to be tracked refers to the first frame image of the video to be tracked; for the search area image, extract the seventh, eighth and ninth respectively Visual features; the search area image refers to all images in the current video package;

The feature fusion module is configured to: fuse the updated first, second and third character feature vectors with the fourth, fifth, sixth, seventh, eighth and ninth visual features respectively to obtain Six fusion features;

The output module is configured to: obtain the target tracking result of each frame in the current video packet of the video to be tracked according to the six fusion features.

It should be noted here that the above-mentioned video packet division module, text feature extraction module, visual feature extraction module, feature fusion module and output module correspond to steps S101 to S105 in Embodiment 1, and the above-mentioned modules and corresponding steps realize The examples and application scenarios are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that, as a part of the system, the above-mentioned modules can be executed in a computer system such as a set of computer-executable instructions.

The description of each embodiment in the foregoing embodiments has its own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

The proposed system can be implemented in other ways. For example, the above-described system embodiments are only illustrative. For example, the division of the above modules is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules can be combined or integrated into another A system, or some feature, can be ignored, or not implemented.

Embodiment three

This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are programmed Stored in the memory, when the electronic device is running, the processor executes one or more computer programs stored in the memory, so that the electronic device executes the method described in Embodiment 1 above.

It should be understood that in this embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application specific integrated circuits ASIC, off-the-shelf programmable gate array FPGA or other programmable logic devices , discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The memory may include read-only memory and random access memory, and provide instructions and data to the processor, and a part of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software module may be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware. To avoid repetition, no detailed description is given here.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in this embodiment can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Embodiment four

This embodiment also provides a computer-readable storage medium for storing computer instructions, and when the computer instructions are executed by a processor, the method described in the first embodiment is completed.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, there may be various modifications and changes in the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (8)

1. The single-target visual tracking method based on text description is characterized in that it comprises: Obtain the sample image of the target to be tracked; obtain the video to be tracked and the text description related to the target to be tracked; divide the video to be tracked into several video packets on average according to the set number of frames; extracting first, second and third character features from the text description; The first, second and third visual features are respectively extracted for the nth sampling frame of each video packet; n is a positive integer, and the upper limit of n is a specified value; based on the first, second and third visual features of the nth sampling frame of each video packet The second and third visual features update the first, second, and third text features respectively to obtain the updated first, second, and third text features; for the sample image of the target to be tracked, extract the fourth and third text features respectively. The fifth and sixth visual features; the template image of the target to be tracked refers to the first frame image of the video to be tracked; for the search area image, the seventh, eighth and ninth visual features are respectively extracted; the search area image is Refers to all images in the current video package; Among them, based on the first, second and third visual features of the nth sampling frame of each video packet, the first, second and third text features are updated respectively to obtain the updated first, second and third Text features; specific steps include: The first visual feature is processed by global average pooling to obtain the first sub-visual feature; the first text feature is used as the initial hidden state of the first LSTM model; at the set time t, the first sub-visual feature is input to the first LSTM In the model, the first LSTM model outputs the updated first text feature; in the first LSTM model, the forget gate is used to determine whether the hidden state at the current moment should be discarded; the input gate is used to determine whether the value of the input visual feature should be discarded. write; The second visual feature is processed by global average pooling to obtain the second sub-visual sub-feature; the second text feature is used as the initial hidden state of the second LSTM model; at the set time t, the second sub-visual feature is input to the second sub-visual feature In the second LSTM model, the second LSTM model outputs the updated second character feature; The third visual feature is processed by global average pooling to obtain the third sub-visual feature; the third text feature is used as the initial hidden state of the third LSTM model; at the set time t, the third sub-visual feature is input to the third In the LSTM model, the third LSTM model outputs the updated third character feature; Fusion the updated first, second and third text feature vectors with the fourth, fifth, sixth, seventh, eighth and ninth visual features respectively to obtain six fusion features; Among them, the updated first, second and third text feature vectors are respectively fused with the fourth, fifth, sixth, seventh, eighth and ninth visual features to obtain six fusion features; specific steps include: Splicing the updated first text feature vector with the fourth visual feature to obtain the first fusion feature; splicing the updated second text feature vector with the fifth visual feature to obtain the second fusion feature; The three text feature vectors are spliced with the sixth visual feature to obtain the third fusion feature; the updated first text feature vector is spliced with the seventh visual feature to obtain the fourth fusion feature; the updated second text feature vector is combined with The eighth visual feature is spliced to obtain the fifth fusion feature; the updated third character feature vector is spliced with the ninth visual feature to obtain the sixth fusion feature; According to the six fusion features, the target tracking result of each frame in the current video packet of the video to be tracked is obtained. 2. the single target visual tracking method based on text description as claimed in claim 1, is characterized in that, extracts first, second and the 3rd text feature to described text description; Concrete steps comprise: Using the BERT method, the first, second and third text features are extracted from the text description. 3. the single target visual tracking method based on text description as claimed in claim 1, is characterized in that, extracts first, second and the 3rd visual feature respectively to the n sampling frame of each video packet; n is positive Integer, the upper limit of n is the specified value; the specific steps include: Using RestNet-50, perform visual feature extraction on the nth sampling frame of each video packet; convolutional layer Conv2_3 outputs the first visual feature; convolutional layer Conv3_4 outputs the second visual feature; convolutional layer Conv5_3 outputs the third visual feature . 4. the single target visual tracking method based on text description as claimed in claim 1, is characterized in that, the template image of target to be tracked, respectively extracts the 4th, the 5th and the 6th visual feature; The image refers to the first frame image of the video to be tracked; the search area image extracts the seventh, eighth and ninth visual features respectively; the search area image refers to all images in the current video package; the specific steps include: Use RestNet-50 to extract visual features from the sample image of the target to be tracked; the convolutional layer Conv2_3 of RestNet-50 outputs the fourth visual feature; the convolutional layer Conv3_4 of RestNet-50 outputs the fifth visual feature; the convolution of RestNet-50 Layer Conv5_3 outputs the sixth visual feature; Using RestNet-50, visual feature extraction is performed on the image of the search area of the target to be tracked; the convolutional layer Conv2_3 of RestNet-50 outputs the seventh visual feature; the convolutional layer Conv3_4 of RestNet-50 outputs the eighth visual feature; the volume of RestNet-50 The multilayer Conv5_3 outputs the ninth visual feature. 5. the single target visual tracking method based on text description as claimed in claim 1, is characterized in that, According to the six fusion features, the target tracking result of each frame in the current video packet of the video to be tracked is obtained; the specific steps include: Input the first fusion feature into the first convolutional neural network CNN, input the output value of the first convolutional neural network and the output value of the fourth convolutional neural network into the first classification network; obtain the first classification result ; Inputting the fourth fusion feature into the fourth convolutional neural network CNN, inputting the output value of the fourth convolutional neural network and the output value of the first convolutional neural network into the first regression network; obtaining the first regression result ; The second fusion feature is input in the second convolutional neural network CNN, the output value of the second convolutional neural network and the output value of the fifth convolutional neural network are all input in the second classification network; obtain the second classification result ; The fifth fusion feature is input into the fifth convolutional neural network CNN, the output value of the fifth convolutional neural network and the output value of the second convolutional neural network are all input into the second regression network; the second regression result is obtained ; The third fusion feature is input into the third convolutional neural network CNN, the output value of the third convolutional neural network and the output value of the sixth convolutional neural network are all input into the third classification network; the third classification result is obtained ; Input the sixth fusion feature into the sixth convolutional neural network CNN, input the output value of the sixth convolutional neural network and the output value of the third convolutional neural network into the third regression network; obtain the third regression result ; Fusing the first classification result, the second classification result and the third classification result to obtain the final classification result; Fusing the first regression result, the second regression result and the third regression result to obtain the final regression result; According to the final classification result and the final regression result, the target tracking result of each frame in the current video packet of the video to be tracked is obtained. 6. A single target visual tracking device based on text description, characterized in that it includes: The video packet division module is configured to: obtain a sample image of the target to be tracked; obtain a video to be tracked and a text description related to the target to be tracked; divide the video to be tracked into several video packets on average according to the set number of frames; A text feature extraction module configured to: extract first, second and third text features from the text description; A visual feature extraction module, which is configured to: extract the first, second and third visual features for the nth sampling frame of each video packet; n is a positive integer, and the upper limit of n is a specified value; based on each video The first, second and third visual features of the nth sampling frame are respectively updated to the first, second and third text features to obtain the updated first, second and third text features; the target to be tracked The model image of the target, extract the fourth, fifth and sixth visual features respectively; the model image of the target to be tracked refers to the first frame image of the video to be tracked; for the search area image, extract the seventh, eighth and ninth respectively Visual features; the search area image refers to all images in the current video package; Among them, based on the first, second and third visual features of the nth sampling frame of each video packet, the first, second and third text features are updated respectively to obtain the updated first, second and third Text features; specific steps include: The first visual feature is processed by global average pooling to obtain the first sub-visual feature; the first text feature is used as the initial hidden state of the first LSTM model; at the set time t, the first sub-visual feature is input to the first LSTM In the model, the first LSTM model outputs the updated first text feature; in the first LSTM model, the forget gate is used to determine whether the hidden state at the current moment should be discarded; the input gate is used to determine whether the value of the input visual feature should be discarded. write; The second visual feature is processed by global average pooling to obtain the second sub-visual sub-feature; the second text feature is used as the initial hidden state of the second LSTM model; at the set time t, the second sub-visual feature is input to the second sub-visual feature In the second LSTM model, the second LSTM model outputs the updated second character feature; The third visual feature is processed by global average pooling to obtain the third sub-visual feature; the third text feature is used as the initial hidden state of the third LSTM model; at the set time t, the third sub-visual feature is input to the third In the LSTM model, the third LSTM model outputs the updated third character feature; The feature fusion module is configured to: fuse the updated first, second and third character feature vectors with the fourth, fifth, sixth, seventh, eighth and ninth visual features respectively to obtain Six fusion features; Among them, the updated first, second and third text feature vectors are respectively fused with the fourth, fifth, sixth, seventh, eighth and ninth visual features to obtain six fusion features; specific steps include: Splicing the updated first text feature vector with the fourth visual feature to obtain the first fusion feature; splicing the updated second text feature vector with the fifth visual feature to obtain the second fusion feature; The three text feature vectors are spliced with the sixth visual feature to obtain the third fusion feature; the updated first text feature vector is spliced with the seventh visual feature to obtain the fourth fusion feature; the updated second text feature vector is combined with The eighth visual feature is spliced to obtain the fifth fusion feature; the updated third character feature vector is spliced with the ninth visual feature to obtain the sixth fusion feature; The output module is configured to: obtain the target tracking result of each frame in the current video packet of the video to be tracked according to the six fusion features. 7. An electronic device, characterized by comprising: one or more processors, one or more memory, and one or more computer programs; wherein, the processor is connected to the memory, and the above-mentioned one or more computer programs are programmed Stored in the memory, when the electronic device is running, the processor executes one or more computer programs stored in the memory, so that the electronic device executes the method described in any one of claims 1-5 above. 8. A computer-readable storage medium, characterized in that it is used to store computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1-5 is completed.

Images