CN113255616B - Video behavior identification method based on deep learning - Google Patents

Abstract

The application relates to a video behavior recognition method based on deep learning, wherein a common 2D network is used as a backbone network in a video behavior recognition network, the characteristics of interframe information are extracted by using bilinear operation, and then the intraframe information and the interframe information are fused to obtain high-identification spatiotemporal characteristics for behavior classification. The 2D model has the capability of processing three-dimensional video information by only adding a small number of parameters, and the accuracy rate of behavior identification can be further improved while the calculation load is reduced compared with that of a traditional 3D convolutional network. The method is particularly suitable for occasions with real-time video analysis requirements but limited resources, and has wide application prospects in the fields of intelligent security, automatic driving and the like.

Description

A video action recognition method based on deep learning

technical field

The present application relates to the technical field of video information processing, and in particular, to a video behavior recognition method based on deep learning.

Background technique

In recent years, with the development and popularization of multimedia technology, high-speed Internet technology and large-capacity storage devices, there has been an explosive growth in video image information resources on the Internet. Compared with static pictures, videos contain more information and richer information. Diversity has become an important information carrier in modern society. At present, most of the video content analysis tasks rely on human resources. However, for massive data, manual processing is time-consuming, labor-intensive, expensive, and inevitably there are omissions. Therefore, video intelligent analysis technology is urgently needed. Since the emergence of Alexnet in 2012, deep convolutional neural networks have dominated the field of computer vision, making breakthroughs in many visual tasks including image classification, object detection, etc., and successfully commercialized, changing people's lifestyles. However, compared with the great achievements in image analysis, deep neural networks have shown good potential in the field of video analysis, but they still cannot achieve satisfactory results. The essential reason is the high spatiotemporal complexity of video signals and the accompanying huge The calculation cost and how to design a reasonable and efficient network structure are still under research and exploration.

Video has one more time dimension than image signal. It is generally believed that the motion information between frames plays a decisive role in the task of video action recognition, but how to extract effective inter-frame motion information has not been well resolved. A popular and effective recognition method at present is to use 3D convolution kernels in deep neural networks, which is the result of natural expansion of 2D convolution in the field of image recognition, and the obtained model is also end-to-end trainable. At present, the more advanced video behavior recognition models, such as I3D, use the deep convolutional network constructed by this method for behavior recognition. Through training on large data sets, and then fine-tuning on small data sets, in many Leading results have been achieved on all benchmark sets.

The 3D convolution kernel directly uses the local adjacent data of the front and back frames for fitting to extract spatiotemporal features. Although the effect is good, there are problems of large amount of parameters, complicated calculation, and prone to overfitting. Although there are some simplified technologies, such as P3D, R3D, etc., which use 2D+1D convolution to replace 3D convolution, they have also achieved good results. But in general, there are still deficiencies in inter-frame feature extraction, and the recognition performance needs to be improved.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide a video behavior recognition method based on deep learning for the above technical problems.

A deep learning-based video behavior recognition method, the method comprising:

Acquire video data, and preprocess the video data to obtain training samples.

A video behavior recognition network is constructed; the video behavior recognition network is a convolutional neural network with a two-dimensional convolutional neural network Resnet as a backbone network, and an inter-frame time domain information extraction module is inserted into the backbone network; the two-dimensional volume The product neural network Resnet is used to extract the static features of the target in the video, the inter-frame temporal information extraction module is used to optimize the backbone network, and the bilinear operation is used to extract the inter-frame information features.

The video behavior recognition network is trained by using the training samples, and parameters are optimized to obtain a trained video behavior recognition network model.

The video to be recognized is acquired and preprocessed, and the preprocessed video to be recognized is input into the video behavior recognition network model to obtain a video behavior classification result.

In one embodiment, acquiring video data, and performing preprocessing on the video data to obtain training samples, including:

Get video data.

A number of consecutive frames of images are randomly selected from the video data to form a video block by using a dense sampling method.

The image in the video block is scaled to a size of 120 pixels by 160 pixels, and an image of a size of 112 pixels by 112 pixels is randomly cropped therefrom.

Divide the grayscale of the cropped image by 255 and map it to a numerical range of [0,1].

De-average normalization is performed on the three RGB channels of the cropped image respectively.

The video blocks are randomly flipped in the horizontal direction with a probability of 50% to obtain training samples.

In one embodiment, the training sample is used to train the video behavior recognition network, and parameters are optimized to obtain a trained video behavior recognition network model, including:

Classify the training samples to obtain a training set and a test set.

The training set is input into the video behavior recognition network for network training, and a video behavior prediction classification result is obtained.

According to the video behavior prediction classification result and the test set, the video behavior recognition network is optimized by adopting the momentum stochastic gradient descent method based on cross entropy loss, and a trained video behavior recognition network model is obtained.

In one embodiment, the video behavior recognition network is composed of a first feature extraction sub-module, three second feature extraction sub-modules, a third feature extraction sub-module and a fully connected layer; the The first feature extraction sub-module consists of a convolutional layer and a maximum pooling layer; the second feature extraction sub-module consists of a spatiotemporal feature extraction module and a maximum pooling layer; the third feature extraction sub-module The module consists of one of the spatiotemporal feature extraction modules and a global pooling layer.

The training set is input into the video behavior recognition network for network training, and the video behavior prediction classification result is obtained, including:

The training set is input into the convolutional layer of the first feature extraction sub-module to obtain the first convolutional feature, and the first convolutional feature is input into the maximum pooling layer of the first feature extraction sub-module to maximize the spatial domain. Value pooling to get the first max pooling feature.

The first maximum pooled feature is input into the first spatiotemporal feature extraction module of the second feature extraction sub-module to obtain the first spatiotemporal fusion feature.

Inputting the first spatiotemporal fusion feature into the first maximum pooling layer of the second feature extraction sub-module to obtain a second maximum pooling feature.

The second maximum pooling feature is input into the second second feature extraction sub-module to obtain a third maximum pooling feature.

The third maximum pooling feature is input into the third second feature extraction sub-module to obtain a fourth maximum pooling feature.

Input the fourth maximum pooling feature into the spatiotemporal feature extraction module of the third feature extraction submodule to obtain spatiotemporal fusion features; and input the spatiotemporal fusion features into the third feature extraction submodule. Global pooling layer to get global pooling features.

The global pooling feature is input into the fully connected layer, and softmax is used as the activation function to obtain the video behavior prediction classification result.

In one embodiment, the spatiotemporal feature extraction module is composed of several residual modules and inter-frame time-domain information extraction modules alternately connected in series; the residual module is a basic component unit of a Resnet network; the inter-frame time domain The domain information extraction module includes: an inter-frame time-domain feature extraction unit and a feature fusion unit; the inter-frame time-domain feature extraction unit includes a bilinear operation convolution layer for extracting time-domain features; the feature fusion unit includes a Convolutional layer for feature fusion.

Inputting the first maximum pooling feature into the first spatiotemporal feature extraction module of the second feature extraction submodule to obtain the first spatiotemporal fusion feature, including:

Inputting the first maximum pooled feature into the first residual module in the spatiotemporal feature extraction module of the first and second feature extraction sub-modules to obtain deep spatial domain features.

Inputting the deep spatial domain features into the first inter-frame time domain information extraction module in the spatiotemporal feature extraction modules of the first and second feature extraction submodules to obtain fusion features.

Input the fusion feature into the second residual module and the inter-frame time domain information extraction module of the first second feature extraction sub-module, and repeat this until the feature information is extracted by the first second feature extraction All the residual modules in the sub-modules and the inter-frame time domain information extraction module are up to the first fusion feature.

In one embodiment, the training set is input into the video behavior recognition network for network training to obtain a video behavior prediction classification result, before the step further comprising:

The parameters of the backbone network of the video behavior recognition network are initialized with the parameters pre-trained by the TSN model on the kinetics400 dataset.

The parameters of the inter-frame time domain feature extraction unit in the inter-frame time domain information extraction module are initialized to random numbers, and the parameters of the feature fusion unit in the inter-frame time domain information extraction module are initialized to 0.

The parameters of the fully connected layer are initialized to random numbers.

In one embodiment, preprocessing is performed, and the preprocessed to-be-recognized video is input into the video behavior recognition network model to obtain a video behavior classification result, including:

A video to be identified is acquired, and the to-be-identified video is uniformly sampled to obtain several video sequences of equal length.

Scale the image in the video sequence to 120 pixels × 160 pixels, crop the middle 112 × 112 pixel area, divide the grayscale of the cropped image by 255, and map it to the range of [0,1]. The three channels of RGB are de-averaged and normalized respectively.

The processed video sequence is input into the video behavior recognition network model to obtain a classification prediction score.

The predicted scores are averaged, the obtained average scores are searched, and the category corresponding to the highest average score obtained by the search is used as the video behavior classification result.

In the above-mentioned deep learning-based video behavior recognition method, the video behavior recognition network uses a common 2D network as the backbone network, uses bilinear operations to extract inter-frame information features, and then fuses intra-frame information and inter-frame information to obtain high-resolution images. Spatiotemporal features are used for behavior classification. Only adding a small number of parameters enables the 2D model to have the ability to process 3D video information. Compared with the traditional 3D convolutional network, it can further improve the accuracy of behavior recognition while reducing the computational load. The present invention is particularly suitable for use in situations where real-time video analysis is required but resources are limited, and has broad application prospects in the fields of intelligent security, automatic driving and the like.

Description of drawings

1 is a schematic flowchart of a video behavior recognition method based on deep learning in one embodiment;

2 is a schematic structural diagram of an inter-frame time domain information extraction module in one embodiment;

FIG. 3 is a structural diagram of a video behavior recognition network with Resnet34 as the backbone network in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

In one embodiment, as shown in FIG. 1 , a deep learning-based video behavior recognition method is provided, and the method includes the following steps:

Step 100: Acquire video data, and preprocess the video data to obtain training samples.

The training sample is a sample in the image format after sampling the video data and then performing image processing.

Step 102: Build a video behavior recognition network.

The video behavior recognition network is a convolutional neural network with two-dimensional convolutional neural network Resnet as the backbone network, and the inter-frame time domain information extraction module is inserted into the backbone network.

A two-dimensional convolutional neural network Resnet is used to extract static features of objects in videos.

The inter-frame time domain information extraction module is used to optimize the backbone network, and uses bilinear operations to extract inter-frame information features.

The inter-frame temporal feature extraction module includes a bilinear operation convolution layer for temporal feature extraction and a convolution layer for feature fusion.

Step 104: Use the training samples to train the video behavior recognition network, and optimize the parameters to obtain a trained video behavior recognition network model.

Step 106: Acquire the video to be recognized, and perform preprocessing, and input the preprocessed video to be recognized into a video behavior recognition network model to obtain a video behavior classification result.

In the above video behavior recognition method based on deep learning, the video behavior recognition network uses a common 2D network as the backbone network, uses bilinear operations to extract inter-frame information features, and then fuses intra-frame information and inter-frame information to obtain high recognition. The spatiotemporal features are used for behavior classification. Only adding a small number of parameters enables the 2D model to have the ability to process 3D video information. Compared with the traditional 3D convolutional network, it can further improve the accuracy of behavior recognition while reducing the computational load. The present invention is particularly suitable for use in situations where real-time video analysis is required but resources are limited, and has broad application prospects in the fields of intelligent security, automatic driving and the like.

In one embodiment, step 100 further includes: acquiring video data; randomly extracting several consecutive frames of images from the video data by using a dense sampling method to form a video block; scaling the images in the video block to a size of 120 pixels×160 pixels, and Randomly crop an image with a size of 112 pixels × 112 pixels; divide the grayscale of the cropped image by 255, and map it to the numerical range of [0, 1]; de-mean and normalize the three RGB channels of the cropped image respectively operation; randomly flip the video blocks in the horizontal direction with a probability of 50% to obtain training samples.

In one embodiment, step 104 further includes: classifying the training samples to obtain a training set and a test set; inputting the training set into a video behavior recognition network for network training to obtain a video behavior prediction classification result; The classification results and the test set are used to optimize the parameters of the video action recognition network using the stochastic gradient descent method with momentum based on the cross entropy loss, and the trained video action recognition network model is obtained.

In one of the embodiments, the video behavior recognition network consists of a first feature extraction sub-module, three second feature extraction sub-modules, a third feature extraction sub-module and a fully connected layer; the first feature extraction sub-module The submodule consists of a convolutional layer and a max pooling layer; the second feature extraction submodule consists of a spatiotemporal feature extraction module and a max pooling layer; the third feature extraction submodule consists of a spatiotemporal feature extraction module And the global pooling layer composition. Step 104 further includes: inputting the training set into the convolutional layer of the first feature extraction submodule to obtain the first convolutional feature, and inputting the first convolutional feature into the maximum pooling layer of the first feature extraction submodule for spatial domain analysis. Maximum pooling to obtain the first maximum pooling feature; input the first maximum pooling feature into the spatiotemporal feature extraction module of the first second feature extraction sub-module to obtain the first spatiotemporal fusion feature; The spatiotemporal fusion feature is input into the maximum pooling layer of the first second feature extraction sub-module to obtain the second maximum pooling feature; the second maximum pooling feature is input into the second second feature extraction sub-module , obtain the third maximum pooling feature; input the third maximum pooling feature into the third second feature extraction sub-module to obtain the fourth maximum pooling feature; input the fourth maximum pooling feature into In the spatio-temporal feature extraction module of the third feature extraction sub-module, the spatio-temporal fusion features are obtained; the spatio-temporal fusion features are input into the global pooling layer of the third feature extraction sub-module to obtain the global pooling features; the global pooling features are input into The fully connected layer uses softmax as the activation function to obtain the video behavior prediction classification results.

The residual module is the basic unit in the Resnet series of convolutional neural networks.

In one embodiment, the spatiotemporal feature extraction module is composed of several residual modules and inter-frame time-domain information extraction modules alternately connected in series; the residual module is a basic component unit of the Resnet network; the inter-frame time-domain information extraction module includes: The inter-frame time-domain feature extraction unit and the feature fusion unit; the inter-frame time-domain feature extraction unit includes a bilinear operation convolution layer for extracting time-domain features; the feature fusion unit includes a convolution layer for feature fusion. Step 104 further includes: inputting the first maximum pooling feature into the first residual module in the spatiotemporal feature extraction module of the first second feature extraction sub-module to obtain deep spatial features; The first inter-frame time domain information extraction module in the spatiotemporal feature extraction module of the second feature extraction sub-module to obtain the fusion feature; input the fusion feature to the second residual module of the first second feature extraction sub-module and the inter-frame time domain information extraction module, and so on, until the feature information passes through all the residual modules and the inter-frame time domain information extraction module in the first and second feature extraction sub-modules to obtain the first fusion feature.

The inter-frame temporal feature extraction unit adopts bilinear operation to extract inter-frame information features.

In another embodiment, the design idea of the inter-frame time domain information extraction module is as follows:

The inter-frame time-domain information extraction module includes two parts: an inter-frame time-domain feature extraction unit that uses bilinear operations to extract inter-frame features, and a feature fusion unit that fuses inter-frame features and intra-frame features.

The traditional 3D decomposition method extracts inter-frame information features through 1D convolution in the time domain. Although the calculation is simple, it is essentially linear fitting, with limited modeling ability and weak feature extraction performance. The present invention uses bilinear operation to extract the time domain information features at the corresponding positions of the front and rear frames. The bilinear operation is essentially a second-order fitting, and has been widely used in fine-grained image recognition, which can better capture the front and rear. Changes between frame images. The calculation formula of bilinear operation is as follows:

（1）

(1)

表示输出特征向量Y的第k维分量，

表示前后帧对应位置点的特征向量，

表示二维卷积提取的空域特征的维度，即特征向量

的维度，

为其第i，j维分量。假设输出特征向量Y的维度也为

，则

就是双线性拟合参数，显然其参数数量远多于普通一维卷积。为了简化计算，可以对参数

进行分解：

，p决定了分解的复杂程度，p是模型的超参数，则公式（1）可以展开如下：in

represents the k -th dimension component of the output feature vector Y,

The feature vectors representing the corresponding position points of the previous and subsequent frames,

Represents the dimension of the spatial feature extracted by the two-dimensional convolution, that is, the feature vector

dimension,

is its i-th and j-dimensional components. Suppose the dimension of the output feature vector Y is also

,but

It is the bilinear fitting parameter, obviously the number of parameters is much more than that of ordinary one-dimensional convolution. In order to simplify the calculation, the parameters can be

Break it down:

, p determines the complexity of the decomposition, p is the hyperparameter of the model, then formula (1) can be expanded as follows:

（2）

(2)

，则双线性操作的参数量减少为

。Formula (2) is a conventional 1D time-domain convolution inside the parentheses. The quadratic term is introduced through the square operation, and the outside of the parentheses is also a linear calculation, which can be implemented by a 1×1×1 convolution, so that it can be used with a square A two-layer convolution of terms is computed to approximate the bilinear operation, and the hyperparameter p is the number of output channels of the first layer of convolution. Considering the higher correlation of features of the same channel between adjacent frames, grouped convolutions are used to replace regular convolutions, while the amount of parameters can be further reduced. Set the number of groups to 4, the time domain receptive field size of the first layer of convolution is 3, and the number of output channels of the first layer of convolution is

, the parameters of the bilinear operation are reduced to

.

The extracted inter-frame features need to be fused with the original spatial features to obtain the spatiotemporal features of the current layer. In order to reduce the impact on the original network output, refer to the weighted fusion method using the NonLocal network. The implementation formula is as follows:

（3）

(3)

Among them, Z is the fusion feature, X is the spatial domain feature, Y is the inter-frame time domain feature, and W is the weighting coefficient. When W is initialized to 0, the output fusion feature is equal to the input airspace feature, and becomes the identity output, which is equivalent to having no impact on the original network structure and can better utilize the pre-trained model parameters of the backbone network.

A schematic diagram of the structure of the inter-frame time domain information extraction module is shown in Figure 2. Input the spatial feature into the convolution kernel as 3×1×1 convolution layer (the first layer of convolution), get the convolution feature, input the convolution feature into the square layer to introduce the quadratic term, and input the result of the square layer into The convolution kernel is a 1×1×1 convolution layer (the second layer of convolution), the output is the inter-frame time domain feature, and the inter-frame time domain feature is input to the convolution kernel as a 1×1×1 convolution layer. , and the obtained convolution output is added and fused with the input spatial domain features, and the fused features are output.

In one embodiment, before step 104, it further includes: using the parameters pre-trained by the TSN model on the kinetics400 data set to initialize the backbone network parameters of the video behavior recognition network; The parameters of the feature extraction unit are initialized to random numbers, and the parameters of the feature fusion unit in the inter-frame time domain information extraction module are initialized to 0; the parameters of the fully connected layer are initialized to random numbers.

参数，就是图2中的前两层卷积层的参数。双线性操作是区别传统线性卷积的，本质上是向量二次项的线性组合，传统线性卷积是向量一次项的线性组合。The parameters of the convolutional layer of the bilinear operation refer to the formula (2) in

The parameters are the parameters of the first two convolutional layers in Figure 2. Bilinear operation is different from traditional linear convolution, which is essentially a linear combination of vector quadratic terms, while traditional linear convolution is a linear combination of vector linear terms.

In one embodiment, step 106 further includes: acquiring the video to be recognized, performing uniform sampling on the video to be recognized, and obtaining several video sequences of equal length; scaling the images in the video sequence to 120 pixels×160 pixels, and cropping the middle 112×112 pixel area, divide the grayscale of the cropped image by 255, map it to the numerical range of [0, 1], and perform de-average normalization on the three RGB channels of the cropped image respectively; The latter video sequence is input into the video behavior recognition network model, and the classification prediction score is obtained; the prediction score is averaged, and the obtained average score is searched, and the category corresponding to the highest average score obtained by the search is used as the video behavior classification result.

In a specific embodiment, the ucf101 data set is used as a training sample, and Resnet34 is used as the 2D backbone network to illustrate the step of classifying the behavior categories in the data set by the video behavior recognition model, including the following steps:

Step 1: Get the data.

Download and prepare the ucf101 dataset, decompress the video data frame by frame into image format and store it for network training and testing.

ucf101 contains a total of 101 behavior categories and a total of 13k videos. The first officially provided method is used to divide the training set and the test set. There are 9537 videos in the training set and 3743 videos in the test set.

Randomly extract 16 consecutive frames of images from the video to form a video block, and preprocess the obtained video block: ① Scale the original image to 120×160 size, and then randomly crop the 112×112 size image from it; ② Convert the image to grayscale Divide by 255, and map to the numerical range of [0,1]; ③ Perform de-average normalization on the three RGB channels of the cropped image respectively, using the normalization coefficients on the imagenet dataset, three RGB channels The mean coefficient and variance coefficient are [0.485, 0.456, 0.406], [0.229, 0.224, 0.225] respectively; ④ The video blocks are randomly flipped in the horizontal direction with a probability of 50% to expand the original data. After the above steps, the final input of the network is obtained, and its dimension size is 16 (time dimension) × 112 (space dimension) × 112 (space dimension) × 3 (channel dimension).

Step 2: Build a video action recognition network.

Using Resnet34 as the backbone network, Resnet34 contains 4 residual module groups in total, each residual module group contains several residual modules, and an inter-frame information extraction module is added after each residual module. Except for the last residual module group, each residual module group uses spatial maximum pooling to reduce the spatial size of the feature map, and the temporal dimension is not pooled. After the last module, global pooling is used to obtain the final 512-dimensional feature vector input to the fully-connected layer, the output dimension of the fully-connected layer is changed to 101-dimensional, and softmax is used as the activation function. The output of the network forward operation is the probability that the input sample is recognized by the model as a different class. The structure diagram of the video action recognition network with Resnet34 as the backbone network is shown in Figure 3.

The Resnet34 backbone network is initialized with the parameters pre-trained by the TSN model on the kinetics400 dataset; the inter-frame temporal feature extraction unit in the inter-frame information extraction module is randomly initialized, and the fused convolutional layer is initialized with all 0s; the final full connection Layers are initialized randomly.

Step 3: Get network parameters.

During network training, the stochastic gradient descent method with momentum is used to train the network parameters, and the standard cross-entropy loss function is used to optimize the network parameters. The training batch size is 128, the initial learning rate is 0.001, and the momentum is 0.9. At the 10th round, the learning rate is reduced by 10 times, and a total of 20 epochs are trained to obtain a trained video action recognition network.

Step 4: Use the trained video action recognition network to classify and identify video actions.

Through the learning and training of steps 2 to 3, the optimal network model parameters are obtained, and the network is used to predict the behavior categories contained in the video in the test set. During prediction, the test video is evenly divided into several segments at intervals of 16 frames, and the frames in the video segment are scaled, center cropped, grayscale remapped, and de-averaged normalized, and each processed video segment is processed. It is sent to the network to calculate the classification score, and then the scores of all segments are accumulated, and the category with the highest score is selected as the final predicted category.

It should be understood that although the various steps in the flowchart of FIG. 1 are shown in sequence according to the arrows, these steps are not necessarily executed in the sequence shown by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIG. 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed and completed at the same time, but may be executed at different times. The execution of these sub-steps or stages The sequence also need not be sequential, but may be performed alternately or alternately with other steps or sub-steps of other steps or at least a portion of a phase.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (5)

1. a video behavior recognition method based on deep learning, is characterized in that, described method comprises: Obtaining video data, and preprocessing the video data to obtain training samples; A video behavior recognition network is constructed; the video behavior recognition network is a convolutional neural network with a two-dimensional convolutional neural network Resnet as a backbone network, and an inter-frame time domain information extraction module is inserted into the backbone network; the two-dimensional volume The product neural network Resnet is used to extract the static features of the target in the video, the inter-frame time domain information extraction module is used to optimize the backbone network, and the bilinear operation is used to extract the inter-frame information features; The video behavior recognition network is trained by using the training samples, and parameters are optimized to obtain a trained video behavior recognition network model; Obtaining the video to be identified, and performing preprocessing, and inputting the preprocessed to-be-identified video into the video behavior recognition network model to obtain a video behavior classification result; Wherein: step: use the training samples to train the video behavior recognition network, and optimize parameters to obtain a trained video behavior recognition network model, including: classifying the training samples to obtain a training set and a test set; Inputting the training set into the video behavior recognition network for network training to obtain a video behavior prediction classification result; According to the video behavior prediction classification result and the test set, adopting the momentum stochastic gradient descent method based on cross entropy loss to optimize the parameters of the video behavior recognition network to obtain a trained video behavior recognition network model; Wherein: the video behavior recognition network consists of a first feature extraction sub-module, three second feature extraction sub-modules, a third feature extraction sub-module and a fully connected layer; the first feature extraction sub-module The module consists of a convolution layer and a maximum pooling layer; the second feature extraction sub-module consists of a spatiotemporal feature extraction module and a maximum pooling layer; the third feature extraction sub-module consists of a It consists of the above-mentioned spatiotemporal feature extraction module and global pooling layer; Step: Input the training set into the video behavior recognition network for network training, and obtain the video behavior prediction classification result, including: The training set is input into the convolutional layer of the first feature extraction sub-module to obtain the first convolutional feature, and the first convolutional feature is input into the maximum pooling layer of the first feature extraction sub-module to maximize the spatial domain. Value pooling to obtain the first maximum pooling feature; Inputting the first maximum pooling feature into the first spatiotemporal feature extraction module of the second feature extraction submodule to obtain a first spatiotemporal fusion feature; Inputting the first spatiotemporal fusion feature into the first maximum pooling layer of the second feature extraction sub-module to obtain the second maximum pooling feature; Inputting the second maximum pooling feature into the second second feature extraction sub-module to obtain a third maximum pooling feature; Inputting the third maximum pooling feature into the third second feature extraction sub-module to obtain a fourth maximum pooling feature; Input the fourth maximum pooling feature into the spatiotemporal feature extraction module of the third feature extraction submodule to obtain spatiotemporal fusion features; and input the spatiotemporal fusion features into the third feature extraction submodule. Global pooling layer to obtain global pooling features; The global pooling feature is input into the fully connected layer, and softmax is used as the activation function to obtain the video behavior prediction classification result. 2. The method according to claim 1, wherein obtaining video data, and preprocessing the video data to obtain training samples, comprising: Get video data; Using the dense sampling method to randomly extract several consecutive frames of images from the video data to form a video block; scaling the image in the video block to a size of 120 pixels by 160 pixels, and randomly cropping an image of a size of 112 pixels by 112 pixels therefrom; Divide the grayscale of the cropped image by 255 and map it to the numerical interval range of [0,1]; Perform de-average normalization operations on the RGB three channels of the cropped image respectively; The video blocks are randomly flipped in the horizontal direction with a probability of 50% to obtain training samples. 3. The method according to claim 1, wherein the spatiotemporal feature extraction module is composed of several residual modules and inter-frame time domain information extraction modules alternately connected in series; Forming unit; the inter-frame time-domain information extraction module includes: an inter-frame time-domain feature extraction unit and a feature fusion unit; the inter-frame time-domain feature extraction unit includes a bilinear operation convolution layer for extracting time-domain features ; The feature fusion unit includes a convolution layer for feature fusion; Inputting the first maximum pooling feature into the first spatiotemporal feature extraction module of the second feature extraction submodule to obtain the first spatiotemporal fusion feature, including: Inputting the first maximum pooling feature into the first residual module in the spatiotemporal feature extraction module of the first and second feature extraction submodules to obtain deep spatial features; Inputting the deep spatial domain feature into the first inter-frame time domain information extraction module in the spatiotemporal feature extraction module of the first and second feature extraction submodules to obtain a fusion feature; Input the fusion feature into the second residual module and the inter-frame time domain information extraction module of the first second feature extraction sub-module, and repeat this until the feature information is extracted by the first second feature extraction All the residual modules in the sub-modules and the inter-frame time domain information extraction module are up to the first fusion feature. 4. method according to claim 3, is characterized in that, described training set is input into described video behavior recognition network to carry out network training, obtains video behavior prediction classification result, also comprises before step: The parameters of the backbone network of the video behavior recognition network are initialized with the parameters pre-trained on the kinetics400 data set by the TSN model; Initializing the parameters of the inter-frame time-domain feature extraction unit in the inter-frame time-domain information extraction module to random numbers, and initializing the parameters of the feature fusion unit in the inter-frame time-domain information extraction module to 0; The parameters of the fully connected layer are initialized to random numbers. 5. The method according to claim 1, wherein the video to be recognized is acquired, and preprocessed, the preprocessed video to be recognized is input into the video behavior recognition network model, and a video behavior classification result is obtained, include: Obtaining the video to be recognized, uniformly sampling the video to be recognized, and obtaining several video sequences of equal length; Scale the image in the video sequence to a size of 120 pixels × 160 pixels, crop the middle area of 112 pixels × 112 pixels, divide the grayscale of the cropped image by 255, and map it to the range of [0,1]. The three RGB channels of the cropped image are de-averaged and normalized respectively; Input the processed video sequence into the video behavior recognition network model to obtain a classification prediction score; The predicted scores are averaged, the obtained average scores are searched, and the category corresponding to the highest average score obtained by the search is used as the video behavior classification result.

Images