CN110532959A - Real-time violent behavior detection system based on two-channel three-dimensional convolutional neural network - Google Patents

Abstract

The invention provides a real-time violent behavior detection system based on a two-channel three-dimensional convolutional neural network, which comprises the following components: the video acquisition module captures video frames in real time and respectively sends the video frames to the video processing module and the playing module; the video processing module is used for extracting the characteristics of the received video frames by using the convolutional neural network, combining the extracted characteristics and classifying the image data according to the combined characteristics; the playing module is used for marking the image classification result obtained by the video processing module into the video frame sent by the video acquisition module and playing the video frame to a user; the video acquisition module, the video processing module and the playing module work in parallel. The invention improves the identification accuracy by introducing a double-channel idea, and realizes the accurate positioning of the occurrence time of violent behaviors by introducing the deconvolution layer.

Description

Real-time violent behavior detection system based on two-channel three-dimensional convolutional neural network

technical field

The invention relates to the technical field of video surveillance, in particular, to a real-time violent behavior detection system based on fast and slow dual-channel three-dimensional convolutional neural networks.

Background technique

Video human behavior recognition and detection is one of the most challenging tasks in computer vision, which can be widely used in many fields such as video surveillance, motion retrieval, human-computer interaction, smart home, and healthcare. At present, there are two branches in the field of behavior recognition: the traditional method represented by the IDT (improved Dense Trajectories) algorithm, and the deep learning method represented by 2D convolution, 3D convolution, and RNN-LSTM. From the perspective of development trends, deep learning methods have surpassed traditional methods in performance.

Improved Dense Trajectories (IDT): The main difference between traditional methods and deep learning methods is the source of the features used for classification. The traditional method is to manually extract one or several kinds of features with better classification effect based on experience and mix them for classification. The method of deep learning is that people pass the classified samples to the computer and let them learn a set of models by themselves. Using the learned model, the features of certain feature combinations can be extracted for classification. As for which features are extracted by the model, Humans have no way of knowing. The number of features extracted by traditional methods is limited, and the range of features that can be selected is also limited, so the features extracted by hand are not as accurate as those extracted by the model. This is also the advantage of deep learning.

Two-channel convolutional neural network (Two-Stream-CNN): It is a representative algorithm for two-dimensional convolution to solve the problem of behavior recognition. Main content: Two channels process RGB frame sequence and optical flow frame sequence at the same time. There is no information exchange in the feature extraction process of the two channels. After the feature extraction is completed, the features are fused in some way for classification to obtain the final result. Because the network can only process one image at a time, and each frame of image in the sequence needs to be processed, and there is a lot of repeated information between adjacent frame images of the video, there is a phenomenon of repeated calculation in this algorithm, and the speed of recognition and detection It is greatly restricted and cannot meet the requirements of real-time performance.

Long-Short Term Memory (LSTM): Due to its unique design structure, LSTM is suitable for processing and predicting important events with very long intervals and delays in time series. Therefore, in the direction of behavior recognition and detection, LSTM has a good effect and is also one of the current mainstream directions.

Two-dimensional convolution has matured in image recognition and detection problems, but video comparison images add a temporal dimension of information. The traditional two-dimensional convolution kernel has been unable to meet the needs of extracting three-dimensional features. The speed of 3D convolution operation is an advantage and can capture inter-frame information well, and it is currently the mainstream research direction. However, the existing methods all have the problems of low recognition accuracy and slow recognition speed, which greatly limit the development and application of human behavior recognition detection technology.

SUMMARY OF THE INVENTION

According to the above technical problems of low recognition accuracy and slow recognition speed, a real-time violent behavior detection system based on fast and slow dual-channel three-dimensional convolutional neural network is provided, and the recognition accuracy is improved by introducing dual-channel ideas. , and at the same time, by introducing a deconvolution layer, the precise location of the time when the violent behavior occurs.

The technical means adopted in the present invention are as follows:

A real-time violent behavior detection system based on two-channel three-dimensional convolutional neural network, including:

a video capture module, which captures video frames in real time and sends them to the video processing module and the playback module respectively;

The video processing module uses the convolutional neural network to perform feature extraction on the received video frames, combines the extracted features, and then classifies the image data according to the combined features;

a playing module, marking the image classification result obtained by the video processing module into the video frame sent by the video acquisition module, and playing it to the user;

The video acquisition module, video processing module and playback module work in parallel.

Further, before the video processing module performs feature extraction on the received video frame, it also preprocesses the video frame, including: sending the RGB images into the slow channel and the fast channel respectively for processing, and processing the obtained slow channel. The preprocessing result and the fast channel preprocessing result are used as the input of the video processing module.

Further, the slow channel is used to sample the RGB images into video segments at equal intervals, and input the trained slow channel network model to predict the slow channel preprocessing data.

Further, the fast channel is used to process the RGB image into grayscale image data and extract the optical flow image data, and input the trained fast channel network model to predict the fast channel preprocessing data.

Further, the video processing module further performs horizontal fusion processing based on convolution feature fusion on the slow channel feature extraction results and the fast channel feature extraction results.

Further, the system further includes a storage module for storing data information during the operation of the system.

Compared with the prior art, the present invention has the following advantages:

The invention utilizes the multi-layer convolutional neural network to extract the temporal correlation feature of the video frame, achieves parameter sharing to a certain extent, can capture the inter-frame information with good performance, improves the operation speed, and has strong real-time performance. At the same time, the fast and slow channel settings are combined, and the fast and slow channel features to be fused have similar shapes through convolution without data loss. After adding the feature fusion structure, the slow channel is rolled in the middle. The slow output of the stacked layer is superimposed with the output of the fast channel that has undergone convolution deformation at the same layer as the input of the next convolution layer, which improves the accuracy of recognition.

In addition, the present invention does not use the pooling operation in the time domain, which ensures that the information in the time domain is preserved to the greatest extent, so that the time when the violent behavior occurs can be more accurately located.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a structural and functional block diagram of the detection system of the present invention.

Fig. 2 is the working flow chart of the detection system of the present invention.

FIG. 3 is a working flow chart of the video processing module of the present invention.

Detailed ways

It should be noted that the embodiments of the present invention and the features of the embodiments may be combined with each other under the condition of no conflict. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is only a part of the embodiments of the present invention, but not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

As shown in Figures 1-3, the present invention provides a real-time violent behavior detection system based on a dual-channel three-dimensional convolutional neural network, including three modules, namely a video acquisition module, a video processing module, and a delayed playback module. Real-time, requires three threads to run at the same time.

The video capture module captures video frames in real time and sends them to the video processing module and the delayed playback module respectively. Specifically, the opencv library and webcam are used to capture video frames in real time. The captured frame images are divided into two channels, one is stored in the second queue, which provides input for the delay playback module of thread three, and the other is stored in the queue one, which provides materials for the image preprocessing step of the video processing module.

The video processing module uses a convolutional neural network to perform feature extraction on the received video frames, combines the extracted features, and then classifies the image data according to the combined features. Specifically, it is used to realize functions such as image preprocessing, video feature extraction, and image feature classification. In order to improve the accuracy of identification, the invention introduces the idea of Slow_Fast, and the data is respectively sent to the fast channel and the slow channel for processing.

As a preferred embodiment of the present invention, the following technical solutions are adopted when performing image preprocessing:

Slow channel: RGB frames are sampled at equal intervals: 64 frames are a video unit, the video unit is sampled at equal intervals, and a frame is taken every 16 frames, and the sampling result is a video segment in the shape of 4*h*w*3.

Adjustment: In the prediction stage, the width and height of each frame of RGB image are scaled to 224*224 and the shape of the output result is 4*224*224*3. In the training phase, the data is first scaled to 4*256*256*3, then randomly cropped to 4*224*224*3, and the data augmentation is completed with random flipping. This increases the generalization ability of the network model and prevents overfitting of the model.

Fast channel: It is mainly divided into two parts. The first part converts the RGB image into a grayscale image. RGB contains three color channels, and the grayscale image has only one color channel. Multiply the grayscale values of the three RGB channels by the corresponding weights, and the summation result is used as the grayscale value of the corresponding point in the grayscale image. The formula is as follows:

Gray=R*0.299+G*0.587+B*0.114

The output data shape of this link is 64*w*h*1

The second part converts the grayscale image into optical flow data, where the optical flow reflects the motion information of objects between adjacent frames.

In this embodiment, the Farneback optical flow algorithm is preferably used to extract the dense optical flow. The optical flow is calculated every two frames, and a video unit has 64 frames, so the output data shape of this link is 32*w*h*2 (the optical flow image has 2 channels, which are the optical flow in the x-direction and the optical flow in the y-direction)

Adjustment: In the prediction stage, the width and height of each frame of image are scaled to 224*224, and the shape of the output result is 32*224*224*2. In the training phase, the data is first scaled to 32*256*256*2, then randomly cropped to 32*224*224*2, and the data augmentation is completed with random flipping. This increases the generalization ability of the network model and prevents overfitting of the model. Here, random cropping and random flipping should be consistent with the slow channel.

The data preprocessing results of the two channels are used as the input of the feature extraction step at the same time.

As a preferred embodiment of the present invention, the following technical solutions are adopted when extracting video features:

Input the processed data into the corresponding network channels, input the trained network model, and extract features layer by layer. In the figure, the output shape of the feature after each layer is marked in the form of T*W*H*C. For example, 32*112*112*8 means that the output feature after the previous convolution module is 32 frames wide 112 High 112 convolution kernel channel number is 8 shape.

Fast channel: input 32 frames of optical flow images, each frame is 224 wide and 224 high, divided into 2 channels of X-direction optical flow and Y-direction optical flow. There are five convolution modules in the fast channel. All convolution modules have the same structure, including a 3D convolution layer, a BN layer, a relu excitation layer, and a 3D pooling layer. The name of the convolution module, the convolution layer The kernel size and pooling kernel size are marked on the figure. For example, Conv 1_3*3*3_1*2*2 means that the name of the layer is Conv 1, the size of the convolution kernel is 3*3*3, and the size of the pooling kernel is 1*2*2. After 5 layers of convolutional layers, an Average Pooling 3D layer is added, the size of the pooling kernel is 1*7*7, and the feature shape is reduced from 32*7*7*128 to 32*1*1*128, reducing the computational cost .

Slow channel: Input 4 frames of RGB color images, each frame is 224 wide and 224 high, divided into three RGB color channels. There are also 5 convolution modules in the slow channel, the module names are Convx_S, and x is 1-5. The first and last module hierarchy is the same as the fast channel convolution module hierarchy, including a 3D convolution layer, a BN layer, a relu excitation layer, a 3D pooling layer, and the convolution kernel size is 3*3* 3. The size of the pooling kernel is 1*2*2. In the middle three levels of the slow channel, the joint operation of convolution and deconvolution is used to achieve simultaneous upsampling in the time domain and upsampling in the spatial domain. The convolution and pooling kernel sizes are shown in the figure. Similarly, after 5 layers of convolutional layers, followed by an Average Pooling 3D layer, the size of the pooling kernel is 1*7*7, and the feature shape is reduced from 32*7*7*128 to 32*1*1*128, reducing computing expenses.

Horizontal fusion: In order to enable the two channels to make full use of the content learned by the other channel, many schemes use horizontal feature fusion, and there are many fusion methods. Since the first dimension of the two channels in this embodiment is the time sequence The information dimensions are not the same, and feature fusion cannot be performed by direct superposition, so this paper chooses the method of convolution feature fusion. That is, the two features to be fused have similar shapes while ensuring that the data is not lost by means of convolution, and the output shape of each convolution layer is marked in the accompanying drawing. After adding the feature fusion structure, in the slow channel, the output of the slow channel of this convolution layer and the output of the fast channel that has undergone convolution deformation at the same layer are superimposed as the input of the next convolution layer.

As a preferred embodiment of the present invention, the setting scheme of the fully connected layer and the classifier is as follows:

After the feature information of the two channels is fused and compressed, many local features are extracted. It needs to go through a fully connected layer to reassemble the local features into complete features, and the global features will be used as input for classifier classification. In this embodiment, the number of fully connected layer nodes is preferably 1024.

Since it is only necessary to classify actions as yes or no violence, the sigmoid function is chosen as the classifier. The number of output nodes is 2.

The shape of the output result of the classifier is 32*2, that is, a video unit has 64 frames, and two probability values of yes or no violent behavior can be obtained for every two frames of images. The larger probability value is taken as the prediction result, so the final result is a sequence of length 32, corresponding to 32 frames of images. This enables frame-level prediction.

The sigmoid function formula is as follows:

g(x)=1/(1+e^(-x))

It should be noted that, in the entire network, the pooling operation is not used in the time domain to ensure that the time domain information is preserved to the greatest extent. In addition, since RGB mainly pays attention to detail information, it changes slowly with time, and there is a lot of repeated information. In order to avoid repeated calculation and save computing expenses, the slow channel is performed at a low frame rate, and one frame is sampled every 16 frames. In one time unit, a total of Sample 4 frames. Since the optical flow graph mainly focuses on motion information, it changes rapidly with time, and is performed at a high frame rate. One frame is sampled every two frames, and a total of 32 frames are sampled in one time unit. Finally, although the number of input frames is small, the slow channel needs to pay attention to more detailed information. We know that the more the number of convolution kernels, the more detailed information can be paid attention to, so the number of fast channel convolution kernels in the whole process should be obvious. More than the slow channel, the number of fast channel convolution kernels in this paper is set to 8 times that of the slow channel.

The delayed playback module marks the image classification result obtained by the video processing module into the video frame sent by the video acquisition module, and performs delayed playback to the user.

In addition, in this embodiment, the system further includes a storage module for storing data information during the operation of the system.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.

Claims (6)

1. a real-time violent behavior detection system based on dual-channel three-dimensional convolutional neural network, is characterized in that, comprises: a video capture module, which captures video frames in real time and sends them to the video processing module and the playback module respectively; The video processing module uses the convolutional neural network to perform feature extraction on the received video frames, combines the extracted features, and then classifies the image data according to the combined features; a playing module, marking the image classification result obtained by the video processing module into the video frame sent by the video acquisition module, and playing it to the user; The video acquisition module, video processing module and playback module work in parallel. 2. The real-time violent behavior detection system according to claim 1, is characterized in that, before described video processing module carries out feature extraction to the video frame received, also carries out preprocessing to described video frame, comprising: RGB images are respectively It is sent to the slow channel and fast channel for processing, and the obtained slow channel preprocessing result and fast channel preprocessing result are used as the input of the video processing module. 3. The real-time violent behavior detection system according to claim 2, wherein the slow channel is used to sample the RGB images into video segments at equal intervals, and the slow channel network model after input training predicts that the slow channel is obtained. Preprocess data. 4. The real-time violent behavior detection system according to claim 2, wherein the fast channel is used to process the RGB image into grayscale image data and extract the optical flow image data, and the fast channel network model after input training predicts Get fast-lane preprocessed data. 5 . The real-time violent behavior detection system according to claim 2 , wherein the video processing module further performs horizontal fusion processing based on convolution feature fusion on the slow channel feature extraction results and the fast channel feature extraction results. 6 . 6 . The real-time violent behavior detection system according to claim 1 , wherein the system further comprises a storage module for storing data information during the operation of the system. 7 .