CN114913465A - Action prediction method based on time sequence attention model - Google Patents

Abstract

The invention discloses an action prediction method based on a time series attention model. Based on deep learning, a time series attention model is built. The model uses a self-attention module to perform feature analysis on image frame data from a video, and integrates the time series model, recursively. It integrates the spatiotemporal context information, and conducts inference and fitting in a self-supervised manner, so that long-term future actions can be predicted; the virtual frame structure is introduced to simplify the complex prediction task into an action classification task for virtual frames, so as to maximize the performance The ability of existing models to integrate information and classification can more effectively solve the problems of low detection accuracy and short prediction time of existing algorithms.

Description

An Action Prediction Method Based on Temporal Attention Model

technical field

The invention relates to the field of computer vision image processing, in particular to an action prediction method based on a time series attention model.

Background technique

Human motion prediction is an emerging task in the field of computer vision and artificial intelligence. Its application scenarios include, but are not limited to, pedestrian trajectory prediction in autonomous driving, home assistance robots, and game VR perception. At the same time, for the video detection of home care, the ability to predict the future movements of the human body plays an important role in scenarios such as fall prevention and emergency rescue. Human action prediction requires algorithmic models to go beyond current spatiotemporal visual classification modeling to predict the multimodal distribution of future actions. Unlike the action recognition task, which can avoid temporal reasoning and reasonably integrate complete context information, the action prediction task requires modeling past actions and predicting the next action, which makes the human action prediction task very complicated and difficult, and the long-term spatiotemporal context. Modeling is at the heart of this task.

Typical long-term spatiotemporal context modeling methods include extracting features from sampled frame images or segments, then using models based on clustering, recursion, or attention mechanism to perform feature aggregation, and directly output the classification results of predicted actions. Most of these models aggregate features only over time, and lack consideration for modeling the time-series evolution of video frames, so they are often inaccurate in predictions.

Another scheme is to predict short-term actions (such as the actions that occur in the next frame) based on the time-series LSTM model. This kind of scheme integrates the information of the previous frame and predicts the information of the next frame in a recursive way, and continuously forwards the reasoning to achieve the prediction. However, due to the accumulation of errors generated by the model during the long-term forecasting process, this solution is ineffective in the long-term forecasting scenario, and cannot meet the forecast-centered demand function in the home-based care scenario.

The latest scheme uses the long-term perception ability of the attention mechanism to build a Transformer model to analyze video data and predict the next action. This type of scheme can well integrate context information, synchronously integrate and predict frame information, and use self-supervision. The method can eliminate the errors generated by the model in the long-term forecasting process to the greatest extent, and can better realize the forecasting function. However, the attention model requires large computing power (such as large-scale GPU cluster computing), and the computational complexity increases at a square rate with the amount of data processed, so it is difficult to process a large amount of video data and deploy it into products. Due to this limitation, this scheme is also difficult to make longer-term predictions.

On the other hand, the performance of deep learning models largely depends on the fit of the training data used to specific tasks. Due to the lack of relevant scene data for model training, existing action prediction schemes cannot be directly applied to home care. Come.

In summary, the existing technical solutions for human motion prediction still have shortcomings such as short prediction time, insufficient prediction accuracy, and large amount of calculation, and are limited and special application scenarios, which cannot meet the fall prevention and emergency rescue in home care scenarios. need.

SUMMARY OF THE INVENTION

In order to solve the above problems, the present invention provides an action prediction method based on a time-series attention model, which can analyze video data with high prediction accuracy by using the time-series attention model.

To this end, the technical solution of the present invention is: an action prediction method based on a time series attention model, comprising the following steps:

1) Video data sampling: select the dense video stream with the corresponding action annotation of each frame as the training video, and sample images of a certain number of frames in the video stream data;

2) Image preprocessing: normalize the image sampled in step 1), and then scale, crop, and flip the image;

3) Establishment and training of time series attention model: The time series attention model includes three parts: encoder, decoder and prediction classifier;

① In the encoding stage, the powerful image analysis ability of the transformer model is used, and the pre-trained Vision Transformer (ViT) model is used to encode the frame image; the Vision Transformer model includes PatchEmbedding (PE) module, Self-Attention (SA) module, feedforward network ( FFN) module and residual connection parts;

The Self-Attention module calculates the weights between each block through the attention mechanism and further performs feature fusion; uses multi-layer linear layers to map X to the high-dimensional space, which are expressed as:

Q=Wq*X

K=Wk*X

V=Wv*X

Where X is the input image, Q is the query matrix, K is the keyword matrix, V is the value matrix, and Wq, Wk, and Wv represent the learning parameters corresponding to Q, K, and V, respectively. Each block can be calculated by Q and K. The relationship between the two, that is, the attention map Am, and then the weight of each block can be calculated through the attention maps Am and V;

Am=SoftMax((Q*K)/sqrt(D)

Among them, SoftMax refers to the use of exponential normalization for the calculation results, D represents the number of characteristic channels of Q, K, and V, and sqrt represents the square root operation;

The feature F1 calculated by the Self-Attention module can be expressed as:

F1=Am*V;

②Decoding stage: The decoder includes a Multi-Head Self-Attention module, a virtual frame structure, and a time-series inference structure;

1) Multi-Head Self-Attention module: the input in the decoding process is the high-dimensional feature representation of the encoded frame image; the calculated feature is the spatiotemporal context information between the frame and the frame in the decoding process;

The introduction of the Multi-Head mechanism is as follows:

Q=[Q1,Q2,...,Qh],Qh=Wq_h*X

K=[K1,K2,...,Kh],Kh=Wk_h*X

V=[V1,V2,...,Vh],Vh=Wv_h*X

II) Position coding: introduce frame position coding and attention map coding to enhance frame image features; frame position coding, number the frame images in sequence and encode them into high-dimensional features Pe through the standard embedding layer;

The attention map encoding encodes the attention map Am calculated in step ① through a standard multi-layer perceptron to obtain high-dimensional features Ae,

Then the initial input of the decoder is:

Input=Pe+Fe

Among them, Fe is the final output of the encoding stage;

Set the calculation process of the first layer Transformer as:

TF_1=FFN(MHSA(Input))

MHSA is the calculation process of the Multi-Head Self-Attention module, and FFN is the calculation process of the feedforward network module;

Then the calculation process of the nth layer Transformer is:

TF_n=FFN(MHSA(TF_n-1+Ae))

As mentioned above, Ae encodes the attention map from the n-1th layer Transformer;

III) virtual frame structure, the initialized virtual frame is equal to the real frame image feature, and the corresponding position code is given to it according to the prediction purpose, and then sent to the multi-head attention model for decoding;

Define the virtual frame as Vf, then the initial input of the decoder after introducing the virtual frame structure is:

Input=Pe+Concatenate(Fe,Vf)

Among them, Concatenate() is the standard splicing operation;

IV) Temporal reasoning structure:

Divide the complete T-frame image feature sequence into non-overlapping sequence segments, each segment contains t-frame sequences, and then input them into the multi-head attention model respectively, that is, the length of the input sequence of the multi-head attention model is limited to t; The recursive reasoning method can cyclically decode the complete sequence, and finally obtain the required decoding features;

③ Prediction classifier: The number of channels of the decoded frame image features is mapped to the number of specific action categories by standard MLP, and the maximum value of the channel is taken as the classification result.

Preferably, the video data sampling in step 1) specifically includes the following steps:

a. Obtain the action segmentation block of the video stream: According to the action label of each frame, the complete video stream is divided into different sub-video streams according to a complete action, that is, the video data contains multiple sub-video streams of a complete action, and intercept one of them. sub-video streams and their corresponding actions as the target of the desired prediction;

b. Sampling forward according to the sub-video stream to obtain the observed data, which is used to transmit it to the network for analysis and prediction; set the intercepted sub-video stream from time S to time E, and set the model to be in action Action prediction is performed A seconds before the occurrence, and the data used is the video stream within 0 seconds, then the video stream from the time St=E-O to the time Et=S-A is intercepted as the input data, and T frame images and corresponding action annotations are sampled from them. as model input.

Preferably, during image preprocessing in step 2), the image is normalized according to the mean and standard deviation of the three channels of the RGB of the frame image, that is, the color values in the range of [0,255] are normalized to the range of [0,1] ; When the image is zoomed, the length and width of the frame image are randomly scaled to the range of [248, 280] pixels; when the image is cropped, the frame image is randomly cropped to the size of 224×224 pixels; and the frame image is randomly flipped horizontally.

Preferably, the Patch Embedding module divides an image into 16×16 blocks of the same size, and flattens the pixels in the blocks; It is implemented and includes a layer normalization module LayerNorm, which is expressed as:

PE(X)=LayerNorm(Conv(X))

where Conv represents a two-dimensional convolution with a convolution kernel size and stride of 16.

Preferably, the feedforward network module is composed of a multi-layer perceptron, and the multi-layer perceptron includes two linear layers and a Relu activation function.

Preferably, the calculation process of the Vision Transformer model is:

X=PE(X)

F1=SA(X)

Af=F1+X

Fe=FFN(Af)+Af

Among them: X is the input image, PE is the calculation process of the Patch Embedding module, SA is the calculation process of the Self-Attention module, and FFN is the calculation process of the feedforward network module.

Preferably, the time series reasoning structure also includes a memory module, and the memory unit is set to store K and V in the calculation process of the Self-Attention module, and pass them to the next recursive process, that is, the Self-Attention calculated in the nth recursive calculation Introduced in the process:

K_n=Concatenate(K_n-1,Wk*X)

V_n=Concatenate(V_n-1,Wv*X)

K_n-1 and V_n-1 are K and V from the last recursive calculation process.

Compared with the prior art, the beneficial effects of the present invention are:

1. The self-attention module analyzes the image frame data from the video, and integrates the time series model, recursively integrates the spatiotemporal context information, and conducts inference and fitting through a self-supervised method, so that long-term future actions can be predicted.

2. Introduce the concept of virtual frame. In the process of recursive reasoning, the virtual frame and the real context information are integrated and processed, and then the corresponding action classification is output; the introduction of virtual frame enables the model to focus on information integration and classification. The prediction task is simplified to the action classification task of virtual frames, which can greatly exert the learning ability of the model.

3. Combined with the long-term information capture ability of the attention model, and through recursive reasoning, while integrating context information, the computational complexity of the model increases linearly with the amount of processed data, which can more effectively solve the detection accuracy of existing algorithms. Low, forecast time is short, difficult to deploy and cannot be used for home care scenarios.

4. Collect the action data in the home care scenario, and mark the relevant human body according to the needs of action prediction, so as to fit the time series attention model to the home care scene, and maximize the performance of the algorithm.

Description of drawings

Further detailed description will be given below in conjunction with the accompanying drawings and the embodiments of the present invention

Fig. 1 is the overall structure diagram of the algorithm of the present invention;

Fig. 2 is the sequence reasoning structure diagram of the present invention;

FIG. 3 is a schematic diagram of video sampling according to the present invention.

Detailed ways

See attached image. The video future action prediction method for home care scenarios described in this embodiment is based on a deep neural network, and a time series attention model is designed to analyze video data. The time series attention model uses the self-attention module to analyze the images from the video. Frame data is analyzed for features, and time series models are integrated to recursively integrate spatiotemporal context information, and inference and fitting are performed in a self-supervised manner, so that long-term future actions can be predicted.

Specifically include the following steps:

1. Video data sampling and image preprocessing

1. Video data sampling: The data used for training is a dense video stream with corresponding action annotations for each frame. First, a certain number of frames of images need to be sampled from the video stream data. The specific operations are as follows:

1.1 Obtain the action segmentation block of the video stream. According to the action tag of each frame, the complete video stream is divided into different sub-video streams according to a complete action, and multiple sub-video streams containing a complete action are obtained in the video data, and one sub-video stream and its corresponding action are intercepted as the required predicted target.

1.2 The observed data is obtained by forward sampling according to the sub-video stream, and is used for transmission to the network for analysis and prediction. It is assumed that the intercepted sub-video stream starts at time S and ends at time E. Set the model to perform motion prediction A seconds before the action occurs, the data used is the video stream within 0 seconds, then intercept the video stream from the time St=E-O to the time Et=S-A as the input data, and sample T frames from it Images and corresponding action annotations are used as model inputs.

The sampling method may be random distribution sampling, pre-distributed sampling (selecting the first T frames of the input video stream), and post-distribution sampling (selecting the last T frames of the input video stream). Post-distribution sampling is used as the default sampling method, as this video frame is closest to the desired predicted action.

2. Image preprocessing:

2.1 Image normalization: Normalize the image according to the mean and standard deviation of the three RGB channels of the frame image, that is, normalize the color values in the range of [0, 255] to the range of [0, 1].

2.2 Image data enhancement. The data augmentation of the image is to expand the diversity of the dataset, which can well avoid overfitting of the model and make it have better generalization ability on the test set.

1) Image scaling: Randomly scale the length and width of the frame image to the range of [248, 280] pixel size.

2) Image cropping: randomly crop the frame image to 224×224 pixel size.

3) Random flip: randomly flip the frame image horizontally.

2. Establishment and training of temporal attention model

The role of the temporal attention model is to analyze the observed video data and combine the virtual frame structure for action prediction, including three parts: encoder, decoder and prediction classifier.

1. In the encoding phase, the pre-trained VisionTransformer (ViT) model is used to encode the frame image by using the powerful analytical ability of the transformer model to the image. The ViT model includes Patch Embedding (PE) module, Self-Attention (SA) module, Feed Forward Network (FFN) module and residual connection.

Set X as the input image, the feature extraction process of the Vision Transformer model can be expressed as:

X=PE(X)

Af=SA(X)+X

Fe=FFN(Af)+Af

Among them: X is the input image, PE() is the output of the Patch Embedding module, SA() is the output of the Self-Attention module, FFN() is the output of the feedforward network module, and Fe is the final output of the encoding stage.

The Patch Embedding (PE) module divides an image into 16×16 blocks of the same size, and flattens the pixels in the blocks. And includes a level normalization module LayerNorm, which is expressed as:

PE(X)=LayerNorm(Conv(X))

where Conv represents a two-dimensional convolution with a convolution kernel size and stride of 16.

The Self-Attention (SA) module calculates the weights between each block through the attention mechanism and further performs feature fusion. The specific steps are:

Use a multi-layer Linear Layer to map X to a high-dimensional space, which are expressed as:

Q=Wq*X

K=Wk*X

V=Wv*X

Among them, Wq, Wk, and Wv represent the learning parameters corresponding to Q(query), K(key), and V(value) respectively. The relationship between each block can be calculated through Q and K, that is, the Attention Map , referred to as Am, and then the weight of each block can be calculated through the attention maps Am and V.

Am=SoftMax((Q*K)/sqrt(D)

Among them, SoftMax refers to the use of exponential normalization for the calculation results, D represents the number of characteristic channels of Q, K, and V, and sqrt represents the square root operation.

Weight calculation: The feature F1 calculated by the Self-Attention module can be expressed as:

F1=Am*V;

Then the calculation process of the Self-Attention module can be expressed as F1=SA(X)

The Feed Forward Network (FFN) module is composed of a multi-layer perceptron (MLP). The MLP includes two linear layers and a Relu activation function. It is a basic standard component of existing deep learning networks.

Residual connections. On the basis of the above, add the residual connection to the SA module and the FFN module, that is, to obtain the complete calculation process of ViT:

X=PE(X)

F1=SA(X)

Af=F1+X

Fe=FFN(Af)+Af

Among them: X is the input image, PE() is the calculation process of the Patch Embedding module, SA() is the calculation process of the Self-Attention module, and FFN() is the calculation process of the feedforward network module.

It can be seen from the calculation process that the resolution of the initial image drops to 16×16 when it passes through PE, but its shape remains unchanged when it passes through SA and FFN. In order to strengthen the network learning ability, ViT superimposes SA and FFN in multiple layers to deepen the network. Finally, after a standard pooling layer, the high-dimensional feature representation of the frame image is obtained.

2. Decoding stage. The decoder is the core component of the time series attention model of the present invention, including a Multi-HeadSelf-Attention module, a virtual frame structure and a time series inference structure.

2.1Multi-Head Self-Attention module and position encoding.

The calculation process of the Multi-Head Self-Attention module is similar to the Transformer structure of the encoding process. The key difference is that the input of the encoding process is the patch of the image, while the input of the decoding process is the high-dimensional feature representation of the encoded frame image. . The computed features also change from image information during encoding to frame-to-frame contextual information during decoding.

The introduction of the Multi-Head mechanism is as follows:

Q=[Q1,Q2,...,Qh],Qh=Wq_h*X

K=[K1,K2,...,Kh],Kh=Wk_h*X

V=[V1,V2,...,Vh],Vh=Wv_h*X

The rest of the calculation is consistent with the SA module calculation of the encoding process.

Position coding: Since the attention mechanism is not sensitive to the order structure between frames, in order for the network to learn the position information of the frame image and integrate the spatiotemporal context for action prediction, frame position coding and Attention Map coding are introduced. , to enhance frame image features.

Frame position encoding position encoding: The frame images are numbered in sequence and encoded into high-dimensional features Pe through the standard embedding layer.

Attention Map Attention Map Encoding Attention encoding: The attention map Am calculated by the previous layer is encoded into high-dimensional features through a standard multi-layer perceptron. The high-dimensional feature of the frame image obtained in the encoding step is represented as Ae, then the initial input of the decoder is:

Input=Pe+Fe

Set the calculation process of the first layer Transformer as:

TF_1=FFN(MHSA(Input))

MHSA is the calculation process of the Multi-Head Self-Attention module, which is distinguished from SA() here; the calculation process of the nth layer Transformer is:

TF_n=FFN(MHSA(TF_n-1+Ae))

As mentioned above, Ae encodes the attention map from the n-1th layer Transformer.

2.2 Virtual frame structure. Different from the existing solution, this embodiment introduces the concept of virtual frame, integrates the virtual frame and real context information, and finally outputs the action classification corresponding to the virtual frame to achieve the purpose of prediction. The introduction of virtual frames enables the temporal attention model to focus on information integration and classification, and simplifies complex prediction tasks into action classification tasks for virtual frames. The specific operation is to equate the initialized virtual frame with the real frame image features, and assign its corresponding position code according to the prediction purpose (the action of the frame to be predicted), and then send it to the multi-head attention module for decoding.

Define the virtual frame as Vf, then the initial input of the decoder after introducing the virtual frame structure is:

Input=Pe+Concatenate(Fe,Vf)

Among them, Concatenate is a standard splicing operation.

2.3 Temporal reasoning structure. Since the complexity of the attention model grows at a quadratic rate with the amount of data processed, it is difficult to process large amounts of video data. The present invention designs a time-series reasoning structure, and through a recursive reasoning method, while integrating context information, the computational complexity of the model increases linearly with the amount of processed data.

First, the complete T-frame image feature sequence (including virtual frames) is divided into non-overlapping sequence segments, each segment contains t-frame sequence, and then respectively input into the multi-head attention model, that is, the length of the input sequence of the multi-head attention model is limited to t. Through recursive reasoning, the complete sequence can be cyclically decoded, and finally the required decoding features can be obtained.

It can be seen from the calculation process of the SA module that as t increases, the model complexity increases at a square rate. And when t is fixed, the model complexity grows linearly with T. This greatly expands the overall amount of data that the model can handle.

Since the sequence segments do not overlap, in order not to lose context information during the recursive reasoning process, a memory module is introduced in this embodiment. Specifically, a memory unit is set to store K and V in the SA calculation process, and transmit them to the next In a recursive process, that is, it is introduced in the SA process of the nth recursive calculation:

K_n=Concatenate(K_n-1,Wk*X)

V_n=Concatenate(V_n-1,Wv*X)

Among them: K_n-1, V_n-1 are K and V from the last recursive calculation process, and the rest of the calculation is consistent with the standard SA module calculation.

3. Predictive classifier. The prediction classifier is a standard classifier in the deep learning model, that is, the number of channels of the decoded frame image feature is mapped to the number of specific action categories through standard MLP, and the maximum value of the channel is taken as the classification result.

The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, and all technical solutions under the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims (7)

1. an action prediction method based on time series attention model, is characterized in that: comprise the following steps: 1) Video data sampling: select the dense video stream with the corresponding action annotation of each frame as the training video, and sample images of a certain number of frames in the video stream data; 2) Image preprocessing: normalize the image sampled in step 1), and then scale, crop, and flip the image; 3) Establishment and training of time series attention model: The time series attention model includes three parts: encoder, decoder and prediction classifier; ① In the encoding stage, the powerful image analysis ability of the transformer model is used, and the pre-trained Vision Transformer (ViT) model is used to encode the frame image; the Vision Transformer model includes PatchEmbedding (PE) module, Self-Attention (SA) module, feedforward network ( FFN) module and residual connection parts; The Self-Attention module calculates the weights between each block through the attention mechanism and further performs feature fusion; uses multi-layer linear layers to map X to the high-dimensional space, which are expressed as: Q=Wq*X K=Wk*X V=Wv*X Where X is the input image, Q is the query matrix, K is the keyword matrix, V is the value matrix, and Wq, Wk, and Wv represent the learning parameters corresponding to Q, K, and V, respectively. Each block can be calculated by Q and K. The relationship between the two, that is, the attention map Am, and then the weight of each block can be calculated through the attention maps Am and V; Am=SoftMax((Q*K)/sqrt(D) Among them, SoftMax refers to the use of exponential normalization for the calculation results, D represents the number of characteristic channels of Q, K, and V, and sqrt represents the square root operation; The feature F1 calculated by the Self-Attention module can be expressed as: F1=Am*V; ②Decoding stage: The decoder includes a Multi-Head Self-Attention module, a virtual frame structure, and a time-series inference structure; 1) Multi-Head Self-Attention module: the input in the decoding process is the high-dimensional feature representation of the encoded frame image; the calculated feature is the spatiotemporal context information between the frame and the frame in the decoding process; The introduction of the Multi-Head mechanism is as follows: Q=[Q1,Q2,...,Qh],Qh=Wq_h*X K=[K1,K2,...,Kh],Kh=Wk_h*X V=[V1,V2,...,Vh],Vh=Wv_h*X II) Position coding: introduce frame position coding and attention map coding to enhance frame image features; frame position coding, number the frame images in sequence and encode them into high-dimensional features Pe through the standard embedding layer; The attention map encoding encodes the attention map Am calculated in step ① through a standard multi-layer perceptron to obtain high-dimensional features Ae, Then the initial input of the decoder is: Input=Pe+Fe Among them, Fe is the final output of the encoding stage; Set the calculation process of the first layer Transformer as: TF_1=FFN(MHSA(Input)) MHSA is the calculation process of the Multi-Head Self-Attention module, and FFN is the calculation process of the feedforward network module; Then the calculation process of the nth layer Transformer is: TF_n=FFN(MHSA(TF_n-1+Ae)) As mentioned above, Ae encodes the attention map from the n-1th layer Transformer; III) virtual frame structure, the initialized virtual frame is equal to the real frame image feature, and the corresponding position code is given to it according to the prediction purpose, and then sent to the multi-head attention model for decoding; Define the virtual frame as Vf, then the initial input of the decoder after introducing the virtual frame structure is: Input=Pe+Concatenate(Fe,Vf) Among them, Concatenate() is a standard splicing operation; IV) Temporal reasoning structure: Divide the complete T-frame image feature sequence into non-overlapping sequence segments, each segment contains t-frame sequences, and then input them into the multi-head attention model respectively, that is, the length of the input sequence of the multi-head attention model is limited to t; The recursive reasoning method can cyclically decode the complete sequence, and finally obtain the required decoding features; ③ Prediction classifier: The number of channels of the decoded frame image features is mapped to the number of specific action categories by standard MLP, and the maximum value of the channel is taken as the classification result. 2. a kind of action prediction method based on time series attention model as claimed in claim 1, is characterized in that: the video data sampling in step 1), specifically comprises the following steps: a. Obtain the action segmentation block of the video stream: According to the action label of each frame, the complete video stream is divided into different sub-video streams according to a complete action, that is, the video data contains multiple sub-video streams of a complete action, and intercept one of them. sub-video streams and their corresponding actions as the target of the desired prediction; b. Sampling forward according to the sub-video stream to obtain the observed data, which is used to transmit it to the network for analysis and prediction; set the intercepted sub-video stream from time S to time E, and set the model to be in action Action prediction is performed A seconds before the occurrence, and the data used is the video stream within 0 seconds, then the video stream from the time St=E-O to the time Et=S-A is intercepted as the input data, and T frame images and corresponding action annotations are sampled from them. as model input. 3. a kind of action prediction method based on time series attention model as claimed in claim 1, is characterized in that: during image preprocessing in step 2), according to the respective mean value and standard deviation of three channels of frame image RGB Carry out the normalization operation, that is, normalize the color values in the range of [0, 255] to the range of [0, 1]; when the image is zoomed, the length and width of the frame image are randomly scaled to the range of [248, 280] pixel size; when the image is cropped, the The frame image is randomly cropped to a size of 224×224 pixels; and the frame image is randomly flipped horizontally. 4. A kind of action prediction method based on time series attention model as claimed in claim 1, it is characterized in that: described Patch Embedding module divides an image into 16*16 blocks of the same size, and divides the block into The pixel flattening; that is, it is realized by two-dimensional convolution with a convolution kernel size and stride size of 16, and includes a layer normalization module LayerNorm, which is expressed as: PE(X)=LayerNorm(Conv(X)) where Conv represents a two-dimensional convolution with a convolution kernel size and stride of 16. 5 . The action prediction method based on the time series attention model according to claim 1 , wherein the feedforward network module is composed of a multi-layer perceptron, and the multi-layer perceptron comprises two linear layers and linear layers. 6 . and a Relu activation function. 6. a kind of action prediction method based on time series attention model as claimed in claim 1 is characterized in that: the calculation process of described Vision Transformer model is: X=PE(X) F1=SA(X) Af=F1+X Fe=FFN(Af)+Af Among them: X is the input image, PE is the calculation process of the Patch Embedding module, SA is the calculation process of the Self-Attention module, and FFN is the calculation process of the feedforward network module. 7. a kind of action prediction method based on time series attention model as claimed in claim 1, it is characterized in that: also comprise memory module in time series reasoning structure, set memory unit for storing K and K in Self-Attention module calculation process V, and passed to the next recursive process, that is, introduced in the Self-Attention process of the nth recursive calculation: K_n=Concatenate(K_n-1,Wk*X) V_n=Concatenate(V_n-1,Wv*X) K_n-1 and V_n-1 are K and V from the last recursive calculation process.

Images