CN111462183A - Behavior identification method and system based on attention mechanism double-current network - Google Patents

Abstract

The present disclosure provides a behavior recognition method and system based on an attention mechanism dual-stream network, belonging to the technical field of behavior recognition. The entire acquired video is divided into multiple equal-length video segments, and the RGB image of each frame of each video segment is extracted. and optical flow grayscale images, and perform preprocessing; randomly sample the preprocessed images to obtain the RGB image and optical flow grayscale image of each video segment; use the dual-stream network model that introduces the attention mechanism to extract the sampled images The appearance features and temporal dynamic features of the video are fused respectively according to the temporal network and the spatial network type, and the fusion result of the temporal network and the fusion result of the spatial network are weighted and fused to obtain the recognition result of the entire video; Make full use of it, and can better extract the local key features of the video frame image, highlight the foreground area where the action occurs, suppress the influence of irrelevant information in the background environment, and improve the accuracy of behavior recognition.

Description

A behavior recognition method and system based on attention mechanism dual-stream network

technical field

The present disclosure relates to the technical field of behavior recognition, and in particular, to a behavior recognition method and system based on an attention mechanism dual-stream network.

Background technique

The statements in this section merely provide background related to the present disclosure and do not necessarily constitute prior art.

In recent years, video-based human action behavior recognition has attracted widespread attention and has become an important research direction in the field of computer vision. It has been widely used in human motion analysis, human-computer interaction, video surveillance, medical care and other scientific fields. With the development of deep learning, especially deep convolutional neural networks, the research on behavior recognition at home and abroad has made great progress. However, due to the influence of factors such as viewing angle change, object occlusion, and background environment interference in complex videos, it is difficult to accurately analyze video data.

Currently, video-oriented action recognition mainly uses a two-stream network, which contains two independent parts that do not share the parameters of a spatial network and a temporal network. The spatial network extracts spatial appearance features from RGB images, and the temporal network extracts spatial appearance features from stacked optical Extract temporal dynamic features from images.

The inventor of the present disclosure found that although the dual-stream network has achieved good results in the field of behavior recognition, the traditional dual-stream network still faces the following problems: (1) Since the traditional dual-stream network simply uses the image information of a certain frame in the video data Identifying the entire video results in a low utilization rate of video data, so it is difficult to ensure that the effective information of other frames in the video can be extracted and utilized by the network; (2) Since the traditional dual-stream network does not consider the influence of image feature weights, it is vulnerable to video The interference of irrelevant information in the background environment leads to encoding background motion information into the final feature representation, which affects the classification accuracy.

SUMMARY OF THE INVENTION

In order to solve the deficiencies of the prior art, the present disclosure provides a behavior recognition method and system based on an attention mechanism dual-stream network, which can make full use of video data and better extract video frames when the video time is long. The local key features of the image highlight the foreground area where the action occurs, suppress the influence of irrelevant information in the background environment, and improve the accuracy of action recognition.

In order to achieve the above object, the present disclosure adopts the following technical solutions:

A first aspect of the present disclosure provides a behavior recognition method based on an attention mechanism dual-stream network.

A behavior recognition method based on attention mechanism dual-stream network, comprising the following steps:

Divide the acquired entire video into multiple equal-length video segments, extract the RGB image and optical flow grayscale image of each frame of each video segment, and perform preprocessing;

Randomly sample the preprocessed images to obtain at least one RGB image and at least one stacked optical flow grayscale image of each video segment;

Using the dual-stream network model that introduces the attention mechanism, the appearance features and temporal dynamic features of the sampled images are extracted, and the features of each extracted video clip are fused according to the temporal network and spatial network types, respectively. The fusion results are weighted and fused to obtain the recognition results of the entire video.

A second aspect of the present disclosure provides a behavior recognition system based on an attention mechanism dual-stream network.

A behavior recognition system based on attention mechanism dual-stream network, including:

The data acquisition module is configured to: divide the acquired entire video into multiple equal-length video segments, extract the RGB image and optical flow grayscale image of each frame of each video segment, and perform preprocessing;

The image sampling module is configured to: randomly sample the preprocessed image to obtain at least one RGB image and at least one stacked optical flow grayscale image of each video segment;

The behavior recognition module is configured to: extract the appearance features and temporal dynamic features of the sampled images by using a dual-stream network model that introduces an attention mechanism, and fuse the extracted features of each video segment according to the temporal network and spatial network types. , the fusion results of the temporal network and the spatial network are weighted and fused to obtain the recognition result of the entire video.

A third aspect of the present disclosure provides a medium on which a program is stored, and when the program is executed by a processor, implements the steps in the attention mechanism dual-stream network-based behavior recognition method described in the first aspect of the present disclosure.

A fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, the processor implementing the program as described in the first aspect of the present disclosure when the processor executes the program The steps in the attention mechanism two-stream network-based action recognition method.

Compared with the prior art, the beneficial effects of the present disclosure are:

1. The method, system, medium and electronic device described in this disclosure use a segmented dual-stream network to divide the video into multiple equal-length segments, and randomly sample each segment to obtain static RGB images and dynamic images of each segment. Then, the two images are input into the dual-stream network for video feature extraction, and then the video features of each segment are fused according to the temporal and spatial channel types, which fully utilizes the video data and improves the accuracy of behavior recognition.

2. The method, system, medium and electronic device described in the present disclosure introduce an attention mechanism into the airspace network, so that the network pays more attention to the local key features of the image, highlights the foreground area where the action occurs, and suppresses the influence of irrelevant information in the background environment, The accuracy of action recognition is further improved.

Description of drawings

The accompanying drawings that constitute a part of the present disclosure are used to provide further understanding of the present disclosure, and the exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure.

FIG. 1 is a schematic flowchart of a behavior recognition method based on an attention mechanism dual-stream network according to Embodiment 1 of the present disclosure.

FIG. 2 is a schematic diagram of an overall network structure provided by Embodiment 1 of the present disclosure.

FIG. 3 is an acquired video RGB image and a corresponding optical flow grayscale image provided by Embodiment 1 of the present disclosure.

FIG. 4 is a schematic diagram of the overall structure of the Inception-v3 network provided in Embodiment 1 of the present disclosure.

FIG. 5 is a schematic structural diagram of the channel attention unit provided in Embodiment 1 of the present disclosure.

Detailed ways

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

The embodiments of this disclosure and features of the embodiments may be combined with each other without conflict.

Example 1:

As shown in FIG. 1 , Embodiment 1 of the present disclosure provides a behavior recognition method based on an attention mechanism dual-stream network, including the following steps:

Divide the acquired entire video into multiple equal-length video segments, extract the RGB image and optical flow grayscale image of each frame of each video segment, and perform preprocessing;

Randomly sample the preprocessed images to obtain at least one RGB image and at least one optical flow grayscale image of each video segment;

Using the dual-stream network model that introduces the attention mechanism, the appearance features and temporal dynamic features of the sampled images are extracted, and the features of each extracted video clip are fused according to the temporal network and spatial network types, respectively. The fusion results of the temporal network Weighted fusion is performed with the fusion result of the airspace network to obtain the recognition result of the entire video.

The detailed steps are as follows:

Step 1: Get the RGB image and optical flow grayscale image of each frame from the video. That is to use the OpenCV and denseflow tools compiled by GPU to cut the video frame to obtain the RGB image of each frame of the video, and use the TV-L1 algorithm to calculate the RGB image of each adjacent two frames, so as to obtain the optical flow gray of the video. degree image. Figure 3 shows the acquired video RGB image and its corresponding optical flow grayscale image (horizontal optical flow image and vertical optical flow image. By acquiring the horizontal and vertical optical flow images, the comprehensiveness of the data is greatly improved, and the the accuracy of action recognition in videos).

Step 2: On the basis of acquiring two kinds of images, preprocess the image data. In this step, four data augmentation methods, random cropping, corner cropping, horizontal flipping and scale jittering, are used to preprocess the input data in the temporal and spatial domain networks.

The random cropping method refers to randomly selecting the cropping area from the image for cropping, the corner cropping method refers to selecting the cropping area from the four corners and the center of the image for cropping, and the scale dithering method refers to determining the size of the cropped area from the image according to the dither ratio.

Taking the public dataset UCF101 as an example, the size of the input data is uniformly set to 256×340, the size of the data used for network training is uniformly set to 224×224, and the jitter ratio is set to 1, 0.875, 0.75, and 0.66. The size of the region is randomly determined by the above ratio, and finally the cropped region is uniformly adjusted to 224×224 for network training.

Step 3: On the basis of the above operations, input the image data into the network to train the dual-stream network.

For the overall structure of the network, this embodiment adopts the idea of segmentation. First, the video is divided into K equal-length segments, and each segment is randomly sampled to obtain the static RGB image and dynamic optical flow image of each segment, and then divide the two segments. Then, the video features of each segment are fused according to the temporal and spatial network types, and finally the recognition results of the dual-stream network are integrated to obtain the recognition and classification of the entire video. Figure 2 shows when K is 3 The overall network structure diagram. In Figure 2, the parameters of the three air domain networks are shared, and the parameters of the three time domain networks are shared.

If K video clips are represented by S ₁ , S ₂ , ..., _Sk , images obtained by random sampling are represented by T ₁ , T ₂ , ..., T _k , network parameters are represented by W, and the output of the network is represented by F(T _k ; W), the aggregation function is represented by G, and the normalized exponential function (ie softmax function) is represented by H, then the expression of the entire network structure can be expressed as:

Net(T ₁ ,T ₂ ,...,T _k )=H(G(F(T ₁ ;W),F(T ₂ ;W),...,F(T _k ;W)))

For the network structure in the spatial and temporal domains, the Inception-v3 convolutional neural network is used in this embodiment; this network can effectively solve the problem that image features cannot be accurately extracted due to differences in image content. It is split into multiple small convolution kernels of different sizes. The information obtained by the split convolution kernel is more abundant than the original, and the features of smaller objects can be extracted, which can make more full use of image information.

Figure 4 shows the overall structure of the Inception-v3 network. Thanks to its unique structure, the Inception-v3 network shows some unique advantages, including: increased network depth and width, increased network nonlinearity, and increased network pairing The adaptability of the scale reduces the number of parameters, which can alleviate the gradient disappearance problem, etc.

In addition, an attention mechanism is introduced into the spatial network, a channel attention unit is designed, as shown in Fig. 5, and it is embedded into the Inception-v3 convolutional neural network to suppress the influence of irrelevant information in the background environment. The unit structure includes a channel attention module and a shortcut connection module, which can make the network pay more attention to the local key features of the image and highlight the foreground area where the action occurs.

For network training, this embodiment uses the network parameters trained on the large data set Kinetics as initial parameters, and performs fine-tuning training on the UCF101 data set. This method can effectively solve the problem of insufficient sample training data. At the same time, a gradient clipping scheme is introduced to constrain the network gradient within a fixed range to prevent the gradient explosion phenomenon that may occur during training. Because the network structure is end-to-end, its network parameters can be automatically updated and optimized through the back-propagation algorithm, so that the network parameters can achieve better performance.

The network initializes the hyperparameters of the network, including: learning rate, learning strategy selection, step size, batch scale, momentum, Dropout_ratio, gradient clipping parameters, and number of iterations.

Step 4: On the basis of the above steps, the selection of parameters such as the number of video clips, clip fusion strategy and channel integration ratio is determined through experiments, that is, the number of video clips is 3, the clip fusion method is the mean fusion method, and the channel integration ratio is selected 1. : 1.6. Next, the trained network is validated on the action dataset UCF101.

The UCF101 dataset was proposed in 2012 by Soomro et al. The dataset contains a total of 101 action categories and 13,320 short video clips, which are mainly from YouTube videos with a resolution of 320×240. The dataset is divided into three sub-datasets split1, split2 and split3. Verification is performed on the three sub-data sets respectively, as shown below, Table 1 shows the recognition accuracy rates of the method of this embodiment on the three sub-data sets.

Table 1: Recognition accuracy on UCF101 dataset

In order to demonstrate the performance of the spatial network after introducing the attention mechanism, two groups of basic experiments are tested in this example. The accuracy results obtained by extracting airspace information from the Inception-v3 network of the force mechanism are shown in Table 2.

Table 2: Comparison of recognition accuracy of spatial network after introducing attention mechanism

In order to further demonstrate the superior performance of the present invention, the average value (95.99%) of the experimental results of the three sub-data sets in UCF101 is taken as the final accuracy rate, which is compared with the recognition accuracy rate of the existing advanced methods on the UCF101 data set. The results are shown in Table 3. It can be seen from Table 3 that the recognition accuracy of the method of this embodiment is higher than that of other methods.

Table 3: Comparison of recognition accuracy of various methods

Example 2:

Embodiment 2 of the present disclosure provides a behavior recognition system based on an attention mechanism dual-stream network, including:

The data acquisition module is configured to: divide the acquired entire video into multiple equal-length video segments, extract the RGB image and optical flow grayscale image of each frame of each video segment, and perform preprocessing;

The image sampling module is configured to: randomly sample the preprocessed image to obtain at least one RGB image and at least one stacked optical flow grayscale image of each video segment;

The behavior recognition module is configured to: extract the appearance features and temporal dynamic features of the sampled images by using a dual-stream network model that introduces an attention mechanism, and fuse the extracted features of each video segment according to the temporal network and spatial network types. , the fusion results of the temporal network and the fusion results of the spatial network are weighted and fused to obtain the recognition results of the entire video.

The working method of the behavior recognition system described in this embodiment is the same as the behavior recognition method based on the attention mechanism dual-stream network in Embodiment 1, and will not be repeated here.

Example 3:

Embodiment 3 of the present disclosure provides a medium on which a program is stored, and when the program is executed by a processor, implements the steps in the attention mechanism-based dual-stream network behavior recognition method described in Embodiment 1 of the present disclosure.

Example 4:

Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, the implementation is as described in Embodiment 1 of the present disclosure. The steps in the attention mechanism two-stream network-based action recognition method.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. During execution, the processes of the embodiments of the above-mentioned methods may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM) or the like.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

The above descriptions are only preferred embodiments of the present disclosure, and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

Although the specific embodiments of the present disclosure have been described above in conjunction with the accompanying drawings, they do not limit the protection scope of the present disclosure. Those skilled in the art should understand that on the basis of the technical solutions of the present disclosure, those skilled in the art do not need to pay creative efforts. Various modifications or variations that can be made are still within the protection scope of the present disclosure.

Claims (10)

1. A behavior identification method based on an attention mechanism double-flow network is characterized by comprising the following steps:

dividing the obtained whole video into a plurality of video segments with equal length, extracting an RGB image and an optical flow gray image of each frame of each video segment, and preprocessing;

randomly sampling the preprocessed images to obtain at least one RGB image and at least one stacked optical flow gray image of each video clip;

and extracting appearance characteristics and time dynamic characteristics of the sampled image by using a double-current network model introducing an attention mechanism, fusing the extracted characteristics of each video segment according to the types of a time domain network and a space domain network, and performing weighted fusion on the fusion result of the time domain network and the space domain network to obtain the identification result of the whole video segment.

2. The behavior identification method based on the attention mechanism double-flow network as claimed in claim 1, characterized in that an attention unit is introduced into the spatial domain network of the double-flow network model, and the attention unit at least comprises a channel attention module and a shortcut connection module, so as to make the spatial domain network focus more on the local features of the image.

3. The method for identifying behaviors based on an attention-machine-based dual-flow network as claimed in claim 1, wherein the collected RGB images and optical flow gray scale images are preprocessed by means of random cropping, corner cropping, horizontal flipping and scale dithering.

4. The attention-based dual-flow network behavior recognition method of claim 1, wherein the dual-flow network model comprises two convolutional neural networks of a space domain network and a time domain network, the space domain network takes the RGB image as input, and the time domain network takes the stacked optical flow grayscale image as input.

5. The method for identifying behavior based on attention mechanism dual-flow network as claimed in claim 1, wherein the space domain network and the time domain network both adopt an inclusion-v 3 convolutional neural network.

6. The attention-based dual-flow network behavior recognition method according to claim 1, wherein the training method of the dual-flow network model specifically comprises: and taking the network parameters obtained by training on the first data set as initial parameters, adjusting the network parameters by using the second data set, introducing a gradient cutting scheme, and constraining the network gradient in a fixed range.

7. The attention-driven dual-flow network-based behavior recognition method of claim 1, wherein the optical flow images comprise horizontal optical flow images and vertical optical flow images.

8. An attention-based dual-flow network behavior recognition system, comprising:

a data acquisition module configured to: dividing the obtained whole video into a plurality of video segments with equal length, extracting an RGB image and an optical flow gray image of each frame of each video segment, and preprocessing;

an image sampling module configured to: randomly sampling the preprocessed images to obtain at least one RGB image and at least one optical flow gray image of each video clip;

a behavior recognition module configured to: and extracting appearance characteristics and time dynamic characteristics of the sampled image by using a double-current network model introducing an attention mechanism, fusing the extracted characteristics of each video segment according to the types of a time domain network and a space domain network, and performing weighted fusion on the fusion result of the time domain network and the space domain network to obtain the identification result of the whole video segment.

9. A medium having a program stored thereon, which program, when being executed by a processor, carries out the steps of the method for attention-based dual-flow network behavior recognition according to any of claims 1-7.

10. An electronic device comprising a memory, a processor and a program stored on the memory and executable on the processor, wherein the processor implements the steps in the method for attention-based dual-flow network behavior recognition according to any of claims 1-7 when executing the program.

Images