CN114882409A - Intelligent violent behavior detection method and device based on multi-mode feature fusion - Google Patents

Abstract

The invention discloses an intelligent violent behavior detection method and device based on multi-mode feature fusion, wherein the method comprises the following steps: acquiring image characteristic data to be detected, voice characteristic data and optical flow characteristic data; performing cross-modal fusion on the image characteristic data and the voice characteristic data to obtain an image fusion characteristic and a first voice fusion characteristic; performing cross-modal fusion on the optical flow feature data and the voice feature data to obtain an optical flow fusion feature and a second voice fusion feature; and inputting the image fusion feature, the first voice fusion feature, the optical flow fusion feature and the second voice fusion feature into a pre-trained multi-mode fusion deep neural network model to obtain an output violent behavior detection result. According to the invention, cross-modal fusion is carried out on the image characteristic data, the voice characteristic data and the optical flow characteristic data, independent single-modal information is retained, video violent behavior detection is carried out by fusing different modalities, and the average accuracy of violent behavior detection is improved.

Description

A method and device for intelligent violent behavior detection based on multimodal feature fusion

technical field

The invention belongs to the technical field of video content analysis, and in particular relates to an intelligent violent behavior detection method and device based on multimodal feature fusion.

Background technique

In recent years, with the continuous development of Internet technology and the help of video big data technology, various videos have begun to appear in daily work and life, resulting in an exponential increase in the number of videos. While bringing great convenience to people and promoting the development of society, these videos are also filled with a lot of violent, vulgar, and terrifying content, which has adversely affected the physical and mental growth of immature minors. At present, how to efficiently and accurately identify these undesirable elements and protect the physical and mental health of minors has become an urgent issue. In addition, it is obviously very time-consuming and labor-intensive to find out the bad elements contained in the massive videos, only relying on the traditional human content review method. Therefore, an intelligent analysis system with the function of detecting specific video content came into being, focusing on the automatic detection technology of video violence, which can be used not only for Internet video content review, but also for video surveillance in sensitive places in intelligent security.

So far, video violence detection methods have relied on processing a single modality, either audio or video. Although recent studies have considered multimodal approaches, most of them ignore the interaction information between different modalities and the advantages of a single modality by simply fusing video and audio emotional information at the feature level. In most cases, visual information acts as an intuitive signal to accurately identify and localize events, but in most scenarios, audio signals can distinguish visually obscured events. Therefore, these illustrate the importance and feasibility of developing a multimodal approach. By studying a multi-modal video violent behavior detection method, it can cope with the changing and complex environment, and improve the generalization ability and average detection accuracy of the violent behavior detection method.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention proposes an intelligent violent behavior detection method, device and system based on multimodal feature fusion.

In order to achieve the above-mentioned technical purpose and achieve the above-mentioned technical effect, the present invention is achieved through the following technical solutions:

In a first aspect, the present invention provides an intelligent violent behavior detection method based on multimodal feature fusion, including:

Obtain the image feature data, voice feature data and optical flow feature data to be detected;

Perform cross-modal fusion of the image feature data and the voice feature data to obtain the image fusion feature and the first voice fusion feature;

Perform cross-modal fusion of optical flow feature data and speech feature data to obtain optical flow fusion features and second speech fusion features;

Inputting the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into a pre-trained multimodal fusion deep neural network model to obtain an output violent behavior detection result;

来优化模型参数，得到训练好的多模态融合的深度神经网络模型。Wherein, the training method of the multi-modal fusion deep neural network model includes: using the labeled image feature data, voice feature data and optical flow feature data to train the multi-modal fusion deep neural network model, and by minimizing the loss function

To optimize the model parameters, the trained multi-modal fusion deep neural network model is obtained.

In some embodiments, the acquiring the image feature data to be detected, the voice feature data and the optical flow feature data include:

Use the I3D network as an extractor of image data to extract image frame features to obtain image feature data to be detected;

Use the VGGish network as the extractor of the voice data to extract the features of the voice data to obtain the voice feature data to be detected;

The I3D network is used as the optical flow data extractor, and the TV-L1 algorithm is used to calculate the optical flow graph on the GPU to obtain the optical flow feature data to be detected.

In some embodiments, the cross-modal fusion of the image feature data and the voice feature data includes:

Splicing image feature data and voice feature data to obtain fusion features;

By calculating the correlation of the fusion features across the modal attention mechanism layer, the output fusion features based on the attention mechanism are obtained;

The output fusion feature based on the attention mechanism is split to obtain the image fusion feature and the first speech fusion feature.

In some embodiments, the cross-modal fusion of optical flow feature data and speech feature data includes:

Splicing optical flow feature data and speech feature data to obtain fusion features;

By calculating the correlation of the fusion features across the modal attention mechanism layer, the output fusion features based on the attention mechanism are obtained;

The output fusion features based on the attention mechanism are split to obtain optical flow fusion features and second speech fusion features.

In some embodiments, the multi-modal fusion deep neural network model includes three multi-modal fusion modules connected in sequence and a fully connected layer; the multi-modal fusion layer includes cross-modal modules, bidirectional long short-term memory A recurrent neural network layer and a pooling layer; the cross-modality module includes a cross-modal attention layer and a graph convolutional layer; the cross-modal attention layer includes sequentially connected linear layers, Softmax layers, normalization layers layers and linear layers.

In some embodiments, the cross-modal module is represented as:

M _i =Linear(M _q )

M _s =Softmax(M _i , dim=1)

M _n =Norm(M _s , dim=2)

Out=Linear(M _n )

Out'=GCN(Out)

Out ₁ , Out ₂ =Split(Out')

表两个特征向量在第一维度拼接；Linear，Softmax，Norm和GCN分别表示线性层、Softmax层、归一化层和GCN层，M_q、M_i分别为第一个线性层的输入、输出，M_s、M_n分别为Softmax层、归一化层的输出，Out为第二个线性层的输出，Out′为GCN层的输出；Split表示矩阵向量的拆分操作；Out₁和Out₂分别表示跨模态模块的两个输出。where M1 and _M2 represent the feature vectors _of the two inputs of the cross-modal module, respectively,

Table two feature vectors are spliced in the first dimension; Linear, Softmax, Norm and GCN represent the linear layer, Softmax layer, normalization layer and GCN layer respectively, M _q , _Mi are the input and output of the first linear layer, respectively , M _s , _Mn are the outputs of the Softmax layer and the normalization layer, respectively, Out is the output of the second linear layer, Out' is the output of the GCN layer; Split represents the splitting operation of the matrix vector; Out ₁ and Out ₂ represent the two outputs of the cross-modal module, respectively.

表示为：In some embodiments, the loss function

Expressed as:

Among them, _yi is the discrete distribution of real violent events, y′ _i is the discrete distribution of the predicted value of the output video tag of the multimodal feature fusion model, and N is the number of input samples.

In a second aspect, the present invention provides an intelligent violent behavior detection device based on multimodal feature fusion, including:

The data preprocessing module is used to obtain the image feature data, voice feature data and optical flow feature data to be detected;

The feature fusion module is used for cross-modal fusion of image feature data and voice feature data, and cross-modal fusion of optical flow feature data and voice feature data to obtain image fusion features, first voice fusion features and optical flow fusion features, The second speech fusion feature;

The violent behavior detection module is used to input the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into the pre-trained multimodal fusion deep neural network model, and obtain the output violent behavior detection result.

In a third aspect, the present invention provides an intelligent violent behavior detection system based on multimodal feature fusion, including: a storage medium and a processor;

the storage medium is used for storing instructions;

The processor is configured to operate in accordance with the instructions to perform the steps of the method of any one of the first aspects.

In a fourth aspect, a storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method in the first aspect are implemented.

Compared with the prior art, the beneficial effects of the present invention:

The invention proposes an intelligent violent behavior detection method based on multimodal feature fusion, and proposes a multimodal fusion deep neural network model for jointly optimizing multimodal fusion and violent behavior detection, which is the traditional multimodal fusion. The state violent behavior detection method provides significant improvement, and the present invention can effectively extract audio, optical flow and image frame information in violent behavior events. The method of the invention utilizes cross-modal modules to combine different modal features to model the relationship between different modalities, so as to generate video features with stronger discrimination. The present invention effectively fuses image feature data, voice feature data and optical flow feature data across modalities, while retaining independent single-modality information, forming a voice fusion feature, a visual fusion feature, and an optical flow fusion feature. Fusion features are used for video violent event detection, which improves the average accuracy of violent behavior detection.

Description of drawings

In order to make the content of the present invention easier to be understood clearly, the present invention will be described in further detail below according to specific embodiments and in conjunction with the accompanying drawings, wherein:

Fig. 1 is a kind of flow chart of the intelligent violent behavior detection method based on multimodal feature fusion of the present invention;

2 is a frame diagram of a method for detecting intelligent violent behavior based on multimodal feature fusion of the present invention;

FIG. 3 is a schematic diagram of a cross-modal model used in a method for detecting intelligent violent behavior based on multi-modal feature fusion of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the protection scope of the present invention.

The application principle of the present invention will be described in detail below with reference to the accompanying drawings.

In the description of the present invention, the meaning of several means one or more, the meaning of multiple means two or more, greater than, less than, exceeding, etc. are understood as not including this number, above, below, within, etc. are understood as including this number. If it is described that the first and the second are only for the purpose of distinguishing technical features, it cannot be understood as indicating or implying relative importance, or indicating the number of the indicated technical features or the order of the indicated technical features. relation.

In the description of the present invention, reference to the terms "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" or the like is meant to be used in conjunction with the embodiment. A particular feature, structure, material or characteristic described or exemplified is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Example 1

An intelligent violent behavior detection method based on multimodal feature fusion, comprising:

Obtain the image feature data, voice feature data and optical flow feature data to be detected;

Perform cross-modal fusion of the image feature data and the voice feature data to obtain the image fusion feature and the first voice fusion feature;

Perform cross-modal fusion of optical flow feature data and speech feature data to obtain optical flow fusion features and second speech fusion features;

Inputting the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into a pre-trained multimodal fusion deep neural network model to obtain an output violent behavior detection result;

来优化模型参数，得到训练好的多模态融合的深度神经网络模型。Wherein, the training method of the multi-modal fusion deep neural network model includes: using the labeled image feature data, voice feature data and optical flow feature data to train the multi-modal fusion deep neural network model, and by minimizing the loss function

To optimize the model parameters, the trained multi-modal fusion deep neural network model is obtained.

An embodiment of the present invention provides an intelligent violent behavior detection method based on multimodal feature fusion, as shown in FIG. 2 , which specifically includes the following steps:

Step 1: Obtain the image feature data, voice feature data and optical flow feature data to be detected, which are specifically described as follows:

We define multimodal data samples and corresponding labels as I = (V, A, O) and y, where V is the untrimmed image frame, A is speech, O is optical flow, y ∈ {0, 1}, y=1 means I covers violent events.

其中C，T分别是通道数，图像帧序列长度。Step 1.1: Use the I3D network as the extractor F ^V of the image frame data V to extract the image frame features to obtain the image feature data to be detected

where C and T are the number of channels and the length of the image frame sequence, respectively.

其中C，T分别是通道数，语音序列长度。Step 1.2: Use the VGGish network as the extractor F ^A of the voice data A to extract the features of the voice data to obtain the voice feature data to be detected

Among them, C and T are the number of channels and the length of the speech sequence, respectively.

其中C，T分别是通道数，光流帧序列长度。Step 1.3: Use the I3D network as the extractor F ^O of the optical flow data O, and use the TV-L1 algorithm to calculate the optical flow graph on the GPU to obtain the optical flow feature data to be detected

where C and T are the number of channels and the length of the optical flow frame sequence, respectively.

Step 2: Perform cross-modal fusion of the image feature data X ^V and the voice feature data X ^A , as shown in Figure 3, and the specific description is as follows:

Step 2.1: Splicing the image feature data X ^V and the voice feature data X ^A to obtain fusion data

Step 2.2: Calculate the correlation of the fusion data through the cross-modal attention mechanism layer to obtain the output fusion feature based on the attention mechanism

和第一语音融合特征

Step 2.3: Split the output fusion feature X ^M based on the attention mechanism to obtain image fusion features

and the first speech fusion feature

Step 3: Perform cross-modal fusion of the optical flow feature data X ^O and the voice feature data X ^A , as shown in Figure 3, and the specific description is as follows:

Step 3.1: Splicing optical flow feature data X ^O and speech feature data X ^A to obtain fusion data

Step 3.2: Calculate the correlation of the fusion data through the cross-modal attention mechanism layer to obtain the output fusion feature based on the attention mechanism

和第二语音融合特征

Step 3.3: Split the output fusion feature X ^M based on the attention mechanism to obtain optical flow fusion features

and the second speech fusion feature

和特征图

其中T和C分别是通道数、序列长度。跨模态模块可以通过以下步骤进行公式化：Step 4: Build a multi-modal fusion deep neural network model, as shown in Figure 2, including 3 multi-modal fusion modules (MFB) and fully connected layers (FC) connected in sequence, among which 3 multi-modal fusion blocks The output channel parameters are set as: 128, 64, 32 respectively. For each multi-modal fusion module, it includes multiple cross-modal modules, bidirectional long short-term memory recurrent neural network layer (BiGRU) and pooling layer (Pool). As shown in Figure 3, for each cross-modal module, it includes a cross-modal attention layer (Cross-Modal Attention) and a graph convolutional layer (GCN). The cross-modal attention layer includes a linear layer (Linear), a Softmax layer, a normalization layer (Norm) and a linear layer (Linear) connected in sequence. For the cross-modal module accepts two inputs: feature map

and feature map

where T and C are the number of channels and the sequence length, respectively. A cross-modal block can be formulated with the following steps:

M _i =Linear(M _q ) (2)

M _s =Softmax(M _i , dim=1) (3)

M _n =Norm(M _s , dim=2) (4)

Out=Linear(M _n ) (5)

Out′=GCN(Out) (6)

Out ₁ , Out ₂ =Split(Out') (7)

表两个特征向量在第一维度拼接；Linear，Softmax，Norm和GCN分别表示线性层、Softmax层、归一化层和GCN层，M_q、M_i分别为第一个线性层的输入、输出，M_s、M_n分别为Softmax层、归一化层的输出，Out为第二个线性层的输出，Out′为GCN层的输出；Split表示矩阵向量的拆分操作；Out₁和Out₂分别表示跨模态模块的两个输出。where M1 and M2 represent the feature vectors of the two inputs of the cross-modal module, respectively,

Table two feature vectors are spliced in the first dimension; Linear, Softmax, Norm and GCN represent the linear layer, Softmax layer, normalization layer and GCN layer respectively, M _q , _Mi are the input and output of the first linear layer, respectively , M _s , _Mn are the outputs of the Softmax layer and the normalization layer, respectively, Out is the output of the second linear layer, Out' is the output of the GCN layer; Split represents the splitting operation of the matrix vector; Out ₁ and Out ₂ represent the two outputs of the cross-modal module, respectively.

Assuming that the graph convolution layer (GCN) has l output channels and k input channels, so the kl convolution operation needs to be used to convert the number of channels, the graph convolution operation formula is:

表示k个输入通道的特征向量；

表示第l个输出通道的特征向量；

表示第K行和第L列的二维卷积核；A表示数据节点之间的连接矩阵，用于确定两个节点vi、vj之间是否存在连接以及连接的强度。在这里用normalized embedded Gaussian方程来衡量空间中两个节点之间的相关性：in

eigenvectors representing the k input channels;

represents the feature vector of the lth output channel;

Represents the two-dimensional convolution kernel of the Kth row and the Lth column; A represents the connection matrix between the data nodes, which is used to determine whether there is a connection between the two nodes vi and vj and the strength of the connection. The normalized embedded Gaussian equation is used here to measure the correlation between two nodes in the space:

Since the normalized Gaussian and softmax operations are equivalent, equation (9) is equivalent to equation (10):

where A _ij represents the correlation between node v _i and node v _j .

After each graph convolution operation, there is an additional operation of Relu, whose purpose is to add nonlinearity to the graph convolution, because the real-world problems solved by using graph convolution are nonlinear, and the convolution operation is Linear operation, so an activation function such as ReLU must be used to add nonlinear properties.

BiGRU layer: In the BiGRU layer, a bidirectional long short-term memory recurrent neural network layer (BiGRU) is used to extract the timing information of violent videos from a global perspective. The formula function is: X=BiGRU(x). where x is the input vector of the BiGRU layer, and X is the output vector, which is used as the input of the next layer.

Pooling layer (Pool): Compress the input feature map, on the one hand, make the feature map smaller and simplify the network computational complexity; on the other hand, perform feature compression to extract the main features. Pooling layers (Pool) can reduce the dimensionality of feature maps while maintaining the most important information.

Step 5, the training method of the multimodal fusion deep neural network model, the specific steps are as follows:

Step 5.1, randomly initialize the parameters and weights of all multimodal fusion deep neural network models;

Step 5.2, take the labeled image feature data, speech feature data and optical flow feature data as the input of the multi-modal fusion deep neural network model, go through the forward propagation step, and finally reach the fully connected layer FC for detection, and obtain the violent behavior detection. As a result, a vector containing the predicted probability values is output. Since the weights are randomly assigned to the first training example, the output probabilities are also random;

如式(11)所示，采用交叉熵(CrossEntropy)损失函数，定义如下：Step 5.3, calculate the loss function of the fully connected layer FC

As shown in formula (11), the cross entropy (CrossEntropy) loss function is used, which is defined as follows:

表示损失函数，用来评测模型对真实概率分布估计的准确程度，可以通过最小化损失函数

来优化模型，更新所有的网络参数。where _yi is the discrete distribution of real violent events. y′ _i is the discrete distribution of the predicted values of the output video labels of the multimodal feature fusion model, and N is the number of input samples.

Represents the loss function, which is used to evaluate the accuracy of the model's estimation of the true probability distribution, which can be minimized by minimizing the loss function

to optimize the model and update all network parameters.

Step 5.4, compute the error gradient for all weights in the network using backpropagation. And use gradient descent to update all filter values, weights and parameter values to minimize the output loss, that is the value of the loss function as low as possible. The weights are adjusted according to their contribution to the loss. When the same skeleton data is input again, the output probability may be closer to the target vector. This means that the network has learned to correctly classify that particular skeleton by adjusting its weights and filters, reducing the output loss. Parameters such as the number of filters, filter size, and network structure are all fixed before step 5.1, and will not change during the training process, only the filter matrix and connection weights are updated.

Step 5.5, repeat steps 5.2-5.4 for all skeleton data in the training set, until the training times reach the set number of iterations. After completing the above steps, the training set data is trained and learned through the constructed spatiotemporal attention convolutional neural network, which actually means that all the weights and parameters of the multi-modal fusion deep neural network model have been optimized, which can correctly detect violent events .

Step 6, using the trained multimodal fusion deep neural network model to identify the test sample, and output the result of violent behavior detection.

In order to verify the advantages of the multimodal fusion deep neural network model, in this experiment, it and several publicly available baseline methods are tested on the XD-Violence multimodal violence detection dataset, and the average precision ((AveragePrecision, AP) as an experimental evaluation indicator.

Table 1 shows the comparison results of a multi-modal feature fusion-based intelligent violent behavior detection method proposed in this example and other baseline methods on the XD-Violence dataset, where the multi-modal fusion deep neural network model , denoted as Ours. An intelligent violent behavior detection method based on multi-modal feature fusion proposed in this embodiment connects two stages (multi-modal modeling and detection network) and jointly optimizes them to generate a more discriminative Video features, using the attention mechanism-based feature fusion module, interactively learns different modal fusion features, thus obtaining better experimental results.

Table 1 The recognition results of the method of the present invention and other disclosed methods on the XD-Violence dataset

It should be noted that, for the sake of simple description, the method embodiments are described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequences, because According to embodiments of the present invention, certain steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

Example 2

For the embodiment of the present invention, an intelligent violent behavior detection device based on multimodal feature fusion is provided, including the following modules:

The data preprocessing module is used to obtain the image feature data, voice feature data and optical flow feature data to be detected;

The feature fusion module is used for cross-modal fusion of image feature data and voice feature data, and cross-modal fusion of optical flow feature data and voice feature data to obtain image fusion features, first voice fusion features and optical flow fusion features, The second speech fusion feature;

The violent behavior detection module is used to input the image fusion feature, the first speech fusion feature, the optical flow fusion feature, and the second speech fusion feature into the pre-trained multimodal fusion deep neural network model, and obtain the output violent behavior detection result.

Example 3

Based on the same inventive concept as Embodiment 1, an embodiment of the present invention provides an intelligent violent behavior detection device based on multimodal feature fusion, including: a storage medium and a processor;

the storage medium is used for storing instructions;

The processor is configured to operate in accordance with the instructions to perform the steps of the method of the first aspect.

Example 4

A storage medium having a computer program stored thereon, the computer program implementing the steps of the method of the first aspect when executed by a processor.

To sum up, an intelligent violent behavior detection method based on multi-modal feature fusion proposed by the present invention solves the problem of multi-modal data fusion in the process of video violent behavior detection, and extracts audio, optical flow and Image frame features, and then, a multimodal fusion deep neural network model is constructed, from which multimodal fusion features and single-modality features are calculated. The method can fuse multiple modalities in the video and improve the accuracy of violence detection, with an accuracy rate of up to 80.59%.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited by the above-mentioned embodiments, and the descriptions in the above-mentioned embodiments and the description are only to illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have Various changes and modifications fall within the scope of the claimed invention. The claimed scope of the present invention is defined by the appended claims and their equivalents.

Claims (10)

1. An intelligent violent behavior detection method based on multi-modal feature fusion is characterized by comprising the following steps:

acquiring image characteristic data, voice characteristic data and optical flow characteristic data to be detected;

performing cross-mode fusion on the image characteristic data and the voice characteristic data to obtain an image fusion characteristic and a first voice fusion characteristic;

performing cross-modal fusion on the optical flow feature data and the voice feature data to obtain an optical flow fusion feature and a second voice fusion feature;

inputting the image fusion feature, the first voice fusion feature, the optical flow fusion feature and the second voice fusion feature into a pre-trained multi-mode fusion deep neural network model to obtain an output violent behavior detection result;

the training method of the multi-modal fusion deep neural network model comprises the following steps: training a multi-modal fusion deep neural network model by utilizing image feature data, voice feature data and optical flow feature data with labels and minimizing a loss function

And optimizing the model parameters to obtain the trained multi-mode fused deep neural network model.

2. The intelligent violent behavior detection method based on multi-modal feature fusion of claim 1, which is characterized in that: the acquiring of the image feature data to be detected, the voice feature data and the optical flow feature data comprises:

using an I3D network as an extractor of image data to extract image frame characteristics to obtain image characteristic data to be detected;

performing voice data characteristic extraction by using a VGGish network as an extractor of voice data to obtain voice characteristic data to be detected;

and (3) utilizing an I3D network as an extractor of optical flow data, and calculating an optical flow diagram on a GPU by using a TV-L1 algorithm to obtain optical flow characteristic data to be detected.

3. The intelligent violent behavior detection method based on multi-modal feature fusion of claim 1, wherein the cross-modal fusion of the image feature data and the voice feature data comprises:

splicing the image characteristic data and the voice characteristic data to obtain fusion characteristics;

calculating the correlation of the fusion characteristics through a cross-mode attention mechanism layer to obtain output fusion characteristics based on an attention mechanism;

and splitting the output fusion feature based on the attention mechanism to obtain an image fusion feature and a first voice fusion feature.

4. The intelligent violent behavior detection method based on multi-modal feature fusion of claim 1, wherein the cross-modal fusion of optical flow feature data and voice feature data comprises:

splicing the optical flow characteristic data and the voice characteristic data to obtain a fusion characteristic;

calculating the correlation of the fusion characteristics through a cross-mode attention mechanism layer to obtain output fusion characteristics based on an attention mechanism;

and splitting the output fusion feature based on the attention mechanism to obtain an optical flow fusion feature and a second voice fusion feature.

5. The intelligent violent behavior detection method based on multi-modal feature fusion of claim 1, wherein the multi-modal fusion deep neural network model comprises 3 multi-modal fusion modules and a full connection layer which are connected in sequence; the multi-modal fusion layer comprises a cross-modal module, a bidirectional long-short term memory recurrent neural network layer and a pooling layer; the cross-modal module comprises a cross-modal attention layer and a graph convolution layer; the cross-modal attention layer comprises a linear layer, a Softmax layer, a normalization layer and a linear layer which are sequentially connected.

6. The intelligent violent behavior detection method based on multi-modal feature fusion of claim 5, wherein the cross-modal module is expressed as:

M _i ＝Linear(M _q )

M _s ＝Softmax(M _i ,dim＝1)

M _n ＝Norm(M _s ,dim＝2)

Out＝Linear(M _n )

Out′＝GCN(Out)

Out ₁ ，Out ₂ ＝Split(Out′)

wherein M is ₁ And M ₂ Feature vectors representing two inputs across modality modules respectively,

splicing the two characteristic vectors in a first dimension; linear, Softmax, Norm and GCN denote the Linear, Softmax, normalization and GCN layers, respectively, M _q 、M _i Input, output respectively of the first linear layer, M _s 、M _n Respectively the output of the Softmax layer and the normalization layer, Out is the output of the second linear layer, and Out' is the output of the GCN layer; split represents a splitting operation of a matrix vector; out ₁ And Out ₂ Representing the two outputs of the cross modality module, respectively.

7. The intelligent violent behavior detection method based on multi-modal fusion of features as claimed in claim 6, wherein the loss function is a function of loss

Expressed as:

wherein, y _i Is a discrete distribution of true violent events, y _i ' is the discrete distribution of the output video tag predictions for the multi-modal feature fusion model, and N is the number of input samples.

8. An intelligent violent behavior detection device based on multi-modal feature fusion is characterized by comprising the following components:

the data preprocessing module is used for acquiring image characteristic data, voice characteristic data and optical flow characteristic data to be detected;

the feature fusion module is used for performing cross-modal fusion on the image feature data and the voice feature data, performing cross-modal fusion on the optical flow feature data and the voice feature data, and acquiring an image fusion feature, a first voice fusion feature, an optical flow fusion feature and a second voice fusion feature;

and the violent behavior detection module is used for inputting the image fusion feature, the first voice fusion feature, the optical flow fusion feature and the second voice fusion feature into a pre-trained multi-mode fusion deep neural network model to obtain an output violent behavior detection result.

9. An intelligent violent behavior detection device based on multi-modal feature fusion is characterized by comprising the following components:

a storage medium and a processor;

the storage medium is used for storing instructions;

the processor is configured to operate in accordance with the instructions to perform the steps of the method of any of claims 1-7.

10. A storage medium having a computer program stored thereon, the computer program, when being executed by a processor, performing the steps of the method of any one of claims 1 to 7.

Images