CN111414846A - Group behavior identification method based on key spatiotemporal information-driven and group co-occurrence structured analysis - Google Patents

Abstract

The invention discloses a group behavior identification method based on key spatiotemporal information driving and group co-occurrence structural analysis, 1) obtaining the importance weight of each member in the group based on the key person candidate sub-network; The importance weights and bounding box features are input to the main network CNN, and the spatial features input to the stacked LSTM network are obtained; 3) The output of 2) is used as input to model co-occurrence features, and by grouping the internal neurons of the stacked LSTM, the realization of Different groups learn different co-occurrence features to obtain group features; 4) Input the bounding box features into the key time segment candidate sub-network for feature extraction to obtain the importance weight of the current frame; The group feature is combined with the importance weight of the current frame obtained in 4) to obtain the group feature of the current frame, and input it to the softmax for group behavior recognition to complete the classification task. This scheme extracts the characteristics of important members of the group and key scene frames based on the key spatiotemporal information, and combines the co-occurrence to process the interaction information within the group behavior to improve the recognition accuracy of the group behavior.

Description

Group behavior driven by key spatiotemporal information and structured analysis of group co-occurrence recognition methods

technical field

The invention relates to the field of group behavior identification, in particular to a group behavior identification method based on key spatiotemporal information driving and group co-occurrence structural analysis.

Background technique

In recent years, human action recognition in video has achieved world-renowned achievements in the field of computer vision. Human behavior recognition has also been widely used in real life, such as intelligent video surveillance, abnormal event detection, sports analysis, understanding social behavior, etc. These applications make group behavior recognition have important scientific practicability and huge economic value. Group behavior recognition is a complex activity completed by multiple people. The most important method of group behavior recognition is the study of individual characteristics and how to infer group behavior from individuals.

"A Hierarchical Deep Temporal Model for GroupActivity Recognition" published on CVPR in 2016 builds a deep model to capture the dynamics of LSTM-based models, proposing a novel deep architecture that Activities are modeled, individual activities are modeled in the first phase, and then personnel-level information is combined with representative group activities. The temporal representation of the model is based on a long short-term memory (LSTM) network, and the goal is to exploit discriminative information in the hierarchy between individual behaviors and group activities. However, although this method uses a two-layer LSTM network, it simply combines individual features to represent group behavior, cannot use personal interaction, and cannot identify key people in the group, resulting in the accuracy of group behavior recognition. In addition, considering that the importance of each person in group behavior recognition is different, this scheme simply models each person, and also reduces the accuracy of group behavior recognition.

In addition, "Region based multi-stream convolutional neural networks for collective activity recognition" published in JOURNAL OF VISUAL COMMUNICATION AND IMAGEREPRESENTATION in 2019 proposes a new human-based region multi-stay architecture for group activity recognition, In addition to using the overall image information, this method also analyzes multiple local regions. Although the interaction information of people-people and groups-people is well considered, it cannot be well captured due to the lack of better use of the LSTM network. The timing information of the video does not fully consider the optical flow motion information of the individual; and the proposed Sum Fusion, MaxFusion, Concatenation Fusion and other fusion strategies are artificially formulated and cannot be well represented.

SUMMARY OF THE INVENTION

In order to accurately identify the behavior of each individual in the group and infer the group behavior by using the individual and their interaction characteristics, the present invention proposes a group based on key spatiotemporal information-driven and group co-occurrence structural analysis. Group behavior identification methods.

The present invention is realized by adopting the following technical scheme: a group behavior identification method based on key spatiotemporal information driving and group co-occurrence structural analysis, comprising the following steps:

Step A: For the video to be identified, track each member in the video, obtain its bounding box image x _t , input it to the key person candidate sub-network in chronological order to extract static and dynamic features, identify individual behavior attributes, and obtain personal important information. Sex weight α _t ;

Step B, input the personal importance weight α _t and the personal bounding box image x _t obtained in step A to the main network CNN for analysis and processing, and obtain the spatial feature X' _t =x _t *α _t input to the stacked LSTM network;

Step C, use the output of step B as input to perform co-occurrence feature modeling, group the neurons of the stacked LSTM, and learn different co-occurrence features in different groups to obtain group feature Z _t ;

Step D, input the bounding box image x _t in step A to the key time segment candidate sub-network for feature extraction, and obtain the importance weight of the current frame, that is, the importance β _t of the current frame;

Step E. Combine the group feature Z _t obtained in step C with the importance weight β _t of the current frame obtained in step D to obtain the group feature Z' _t of the current frame, and input Z' _t into the softmax, complete group behavior recognition.

Further, in the step A, when obtaining the personal importance weight α _t , the specific implementation is as follows:

(1) First, establish a key person candidate sub-network, and the key person candidate sub-network includes CNN and LSTM layers connected in series, a fully connected layer, a tanh activation function, and a fully connected layer;

(2) Second, obtain the behavioral attribute scores of group members, specifically:

At time t, there are M members in the set scene, the bounding box feature set extracted by the CNN network is x _t =(x _{t, 1} ,,...,x _{t, M} ) ^T , and the behavior attribute score is s _t =(s _t,1 ,...,s _t,M ) ^T , the behavior attribute score represents the behavior category judgment of M members, expressed as:

表示来自LSTM层的隐藏变量。Among them, T is the length of the video sequence, U _s , W _xs , W _hs are the learnable parameter matrices, b _s , _bus are the bias vectors,

represents the hidden variable from the LSTM layer.

(3) Finally, obtain the importance weight of each member, and then determine the key person, specifically:

Given a group behavior category set G _{_action} =(A ₁ ,A ₁ ,...,A _q ) ^T , for the kth person, calculate the projection of its behavior attribute score s _t,k on G _{_action} , specifically using this The cosine similarity of two multidimensional vectors is measured by:

||||表示2范数；

|||| represents the 2 norm;

Converted to cosine angle normalization coefficient is:

The importance weight of each person in the spatial scene is calculated by the following formula:

α _t,k is to determine how much each member contributes to the group behavior recognition task.

Further, the step B is specifically implemented in the following manner:

(1) First, through the modulation of importance weights, the spatial characteristics of the kth person at time t are obtained:

x' _t,k =α _t,k ·x _t,k

(2) Then, aggregate all individual spatial features modulated by importance weights as the input of the stacked LSTM in the main network, that is:

X' _t =(x' _t,1 ,...,x' _t,K ).

Further, in the step C, when the co-occurrence feature learning is performed, the following methods are specifically adopted:

Step C1, first establish an end-to-end fully connected deep LSTM network model to realize automatic learning of time series features and motion modeling;

Based on the alternate deployment of LSTM layers and feedforward layers, a deep network is formed to capture motion information, and the feedforward layer is located between the two layers of LSTM networks, so that the neurons of each layer are fully connected to the neurons of the next layer;

Step C2, then group the neurons of the layered LSTM, and introduce constraints on the weights of the individual members and the neurons in the objective function, so that the neurons in the same group have greater weights for the subsets composed of some individual members. connections, while connecting other nodes with smaller weights, so as to mine the co-occurrence of individual members.

Further, in the step C1, in order to ensure that the fully connected deep LSTM network model learns effective features, different types of regularization are implemented in different parts of the model, including two types of regularization:

1) For fully connected layers, regularization is introduced to drive the model to learn co-occurrence features of individuals in different layers, and co-occurrence feature learning of nodes between LSTM layers;

2) For LSTM neurons, derive a new Dropout layer and apply it to the LSTM neurons in the last LSTM layer.

Further, in the step C2, the co-occurrence mining and utilization are realized by adding group sparse constraints to the connections of each group of neurons and individual members:

(1) Group the neurons of each layer of LSTM according to the number of group behavior categories. For the neurons in the Kth group, this group of neurons will automatically distinguish different individual behaviors after training, and the co-occurrence regularity is added to the loss function. change:

where L is the maximum likelihood loss function of the deep LSTM network, W _xβ = [W _x,1 ;...;W _x,K ] is the weight matrix connected to the β of the input unit, let N denote the number of neurons , these N neurons are divided into θ groups, the number of neurons in each group is ε=[N/θ], for LSTM layer, S={i,f,o,c} represents the input gate in LSTM neurons , forget gate, output gate and cell unit, for the feedforward layer, S={h} represents the neuron itself;

以驱动其选择具有不同描述性的特征做为输入，不同的神经元分组探索不同的共现性特征，然后采用梯度下降法求解。The second term in formula (1) is L1 regularization, which is used to determine a relatively important subset of key figures during training; in the third term, since the L2 norm can encourage the matrix W _xβ,k to become sparse, so Using the L2 norm for each group of cells is defined as

In order to drive it to select features with different descriptiveness as input, different neuron groups explore different co-occurrence features, and then use gradient descent method to solve.

Further, the step D is specifically realized in the following ways:

(1) First, establish a key time segment candidate sub-network, and the key time segment candidate sub-network includes CNN and LSTM layers connected in series, fully connected layers and Relu nonlinear units;

(2) According to the video sequence input to the sub-network, use the Relu unit to obtain the behavior attribute score in the current frame, namely: o _t =RELU(w _x' x _t +w _h' h' _t-1 +b')= (o ₁ ,o ₂ ,...o _C ) ^t , C is the total number of action categories, t is the current frame, the size of which depends on the current input x _t , the hidden state h' _{t-1 of the LSTM layer at time t-1} ;

(3) Finally, according to the correlation degree between the behavior attribute score of the current frame and the group behavior attribute, the importance weight β _t of the current frame in the input sequence T is obtained, specifically:

Given a group action category set G _{_action} =(A ₁ ,A ₁ ,...,A _q ) ^T , the temporal importance weight β _t can be calculated by calculating the set and the current frame action attribute score o _t =(o ₁ , o ₂ ,...o _C ) ^t is represented by the intersection similarity coefficient of these two sets, namely:

Among them, ∩ represents the intersection calculation, and n() represents the number of elements in the solution set.

Further, in the step E, the following methods are specifically adopted when the group behavior identification is performed based on the group feature Z _t and the importance weight β _t of the current frame:

(1) First calculate the frame group characteristics at time t as:

Z' _t = Z _t ·β _t

(2) Then input it to the softmax layer for final group behavior recognition:

y=softmax(Z' _t )

Among them, y is the group behavior category.

Further, based on the complexity of the model, the main network, the key person candidate sub-network and the key time segment candidate sub-network are jointly trained. Specifically, the joint training process of the network is as follows:

Input: The number of training times of the model N1, N2;

(1) Use the Gaussian function to initialize the network parameters;

(2) Fix the weight of the key person candidate sub-network, and jointly train the key time period candidate sub-network main network with only one LSTM layer of the main network to obtain the key time period candidate model;

(3) Repeat the iteration, and after N1 iterations increase its LSTM layer to three layers, train the main network;

(4) Fine-tune the main network and key time period candidate sub-networks through N2 iterations;

(5) Fix the key time period candidate sub-network, and jointly train the key person candidate sub-network and the main network with only one LSTM layer to obtain the key person candidate sub-network;

(6) Repeat the iteration, after N1 iterations increase its LSTM layer to three layers, train the main network;

(7) Fine-tune the main network and the candidate sub-network for this key time period through N2 iterations;

(8) The sub-network obtained by jointly training (4) and (7) through N1 iterations;

(9) Jointly fine-tune the entire network model through N2 iterations;

Output: The entire group behavior recognition model is finally converged.

Compared with the prior art, the advantages and positive effects of the present invention are:

The group behavior recognition method based on spatiotemporal importance and co-occurrence proposed in this scheme uses the importance mechanism to focus on important individual behaviors among group behaviors, extracts more important personal characteristics and important scene frames, and combines co-occurrence Sexual processing of interactive information within group behaviors, through the combination of spatiotemporal importance and co-occurrence, can make better use of individual characteristics and effectively use key information among many information, thereby achieving accuracy and efficiency improvement, and has important scientific Practicality and great economic value.

Description of drawings

1 is a schematic diagram of an overall network architecture according to an embodiment of the present invention;

FIG. 2 is an internal structure diagram of the main network stacked LSTM according to an embodiment of the present invention;

3 is a schematic diagram of grouping neurons in each layer of LSTM according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of the internal structure of an LSTM neuron according to an embodiment of the present invention.

Detailed ways

In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be further described below with reference to the accompanying drawings and embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention, however, the present invention may also be implemented in other ways than those described herein, and therefore, the present invention is not limited to the specific embodiments disclosed below.

In order to accurately identify the behavior of each individual in the group, and use the individual and their interaction characteristics to infer group behavior, this embodiment proposes a structured analysis based on key spatiotemporal information-driven and group co-occurrence. Group behavior identification method, including the following steps:

Step A. For the video to be identified, track each member in the video, obtain its bounding box image x _t , input it to the key person candidate sub-network in chronological order to extract static features and dynamic features, identify individual behavior attributes, and then obtain the individual behavioral attributes. importance weight α _t ;

Step B. Input the personal importance weight α _t and the personal bounding box image x _t obtained in step A to the main network for multiplication processing, in order to obtain the spatial feature X' _t = x _t * input into the main network layered LSTM α _t ;

Step C, take the output of step B as input, carry out co-occurrence feature modeling, group the neurons of the stacked LSTM, and learn different co-occurrence features in different groups to obtain group feature Z _t ;

Step D, input the bounding box image x _t in step A into the key time segment candidate sub-network, perform feature extraction, and obtain the importance weight of the current frame, that is, the importance β _t of the current frame;

Step E. Combine the group feature Z _t obtained in step C and the importance weight β _t of the current frame obtained in step D to obtain the group feature Z' _t of the current frame, and input it into softmax for grouping. Group behavior identification, complete group behavior identification.

In order to achieve the accuracy and efficiency of group behavior recognition, this scheme designs a main network and two sub-networks (key person candidate sub-network and key time period candidate sub-network). The main network is used to extract features and spatial-temporal correlation. Exploitation and final classification; key person candidate sub-network is used to assign appropriate importance to different individuals; key time period candidate sub-network is used to assign appropriate importance to different frames. The single-person behavior recognition is not carried out in the main network, but directly infers the single-person behavior in the sub-network to obtain the ranking of key people, and then controls the useful information input of the main network.

1. In the specific group behavior identification process, although the group activities are completed by many people, the members who decide the group behavior are usually a few members (key people), and these key people complete the group within a certain period of time. group behavior. Therefore, in this scheme, two sub-networks are designed to pay attention to the key person information, so as to shield the useless information interference of other members and optimize the recognition accuracy of the model; the two sub-networks are the key person candidate sub-network and the key time period respectively. The candidate sub-network, in which the key person candidate sub-network quantifies the importance of the spatial position of the group members according to the correlation between the individual behavior and the group behavior, and controls the input of the CNN spatial information in the main network; The candidate sub-network in the key time period quantifies the importance of the time information output by the stacked LSTM in the main network according to the correlation between the category of member behavior and the group behavior, pays attention to the useful time slices, and optimizes the input to the classification layer softmax. Information; through the above two sub-networks, the "critical moment information" in the group behavior is purified, aiming to shield the influence of interference noise from irrelevant personnel in the group.

2. In addition, as a complex group activity, a certain behavior of the group is generally completed by a number of specific individuals within the group. The group members are structured groupings, and these members have a close interaction within this small group. In this regard, in this scheme, we focus on the core group formed by key figures, reduce the influence of irrelevant members on the identification of group behavior, and call the characteristic of core group members' collaborative interaction and cooperation to determine group behavior identification as co-occurrence. There are many kinds of group behaviors, so there will be a corresponding number of such relatively "stable co-occurrence groups". It should be emphasized that the appearance of these co-occurrence groups is time-varying, and that only one "co-occurrence group" plays a dominant role in a certain time segment.

In order to describe such a "co-occurrence group" in the group, in this scheme, a three-layer joint iterative bidirectional LSTM layer is designed in the main network, and the LSTM neurons in each layer are grouped, and each neuron group only focuses on A category of group behavior, each neuron in the group needs to connect each member of the group, (for example, if the neurons in an LSTM are divided into 6 groups, you can focus on 6 different group behaviors, and The behavioral properties of these six groups of concern are invariant). In this way, neurons in the same group are trained to have more weighted connections to a subset of several individuals for a particular behavior, and less weighted connections to some other less related individuals, so that This method learns the co-occurrence features, which highlights the co-occurrence timing information of the key core small groups, and after learning the key time segment candidate sub-network features, it can further suppress the redundant and useless information in the LSTM, and improve the co-occurrence timing information information. The noise ratio is used as the input of the classification layer softmax, thereby improving the accuracy of group behavior recognition.

The following describes the group behavior identification method described in this embodiment in detail, specifically:

In step A, when obtaining the personal importance weight α _t , the specific implementation is as follows:

(1) First, establish a key person candidate sub-network, as shown in Figure 1, the key person candidate sub-network includes CNN and LSTM layers connected in series, a fully connected layer, a tanh activation function and a fully connected layer;

(2) Second, obtain the behavioral attribute scores of the group members, specifically:

At time t, there are M members in the set scene, the bounding box feature set x _t =(x _{t, 1} ,,...,x _{t, M} ) ^T extracted by the CNN network, and the behavior attribute score s _t =(s _t,1 ,...,s _t,M ) ^T represents the behavior category judgment of M members, which is obtained by the following formula:

表示来自LSTM层的隐藏变量；Among them, T is the length of the video sequence, U _s , W _xs , W _hs are the learnable parameter matrices, b _s , _bus are the bias vectors,

represents the hidden variables from the LSTM layer;

(4) Finally, obtain the importance weight of each member, and then determine the key person, specifically:

Given the group behavior category set G _{_action} =(A ₁ ,A ₁ ,...,A _q ) ^T , for the kth person, to calculate the projection of its behavior attribute score s _t,k on G _{_action} , you can use these two The cosine similarity of a multidimensional vector is measured by:

||||表示2范数；

|||| represents the 2 norm;

由以下公式计算空间场景中每个人的重要性权重：Converted to cosine angle normalization coefficient is:

The importance weight of each person in the spatial scene is calculated by the following formula:

α _t,k is to determine how much each member contributes to the group behavior recognition task, and to control the amount of information flowing to the main network.

In step B, multiply the personal importance weight α _t obtained in step A and the feature of the personal bounding box feature x _t input to the main network CNN, in order to obtain the feature X' _t = x _t * input to the stacked LSTM network α _t , which is implemented in the following ways:

(1) First, through the modulation of importance weights, the spatial characteristics of the kth person at time t are obtained:

x' _t,k =α _t,k ·x _t,k

(2) Then, aggregate all individual spatial features modulated by importance weights as the input of the stacked LSTM:

X' _t = (x' _t,1 ,...,x' _t,K )

The key person candidate sub-network proposed in this embodiment determines the importance of the individual's behavior based on all individuals at the current time and the hidden variables in the LSTM layer. The purpose of the key person candidate sub-network is to assign importance to individuals in group activities. Sex weights, considering that the LSTM hidden variable h _t-1 contains information from past frames, so it can explore long-term dynamics.

In step C, the output of step B is used as the input, and the co-occurrence feature modeling is performed, the neurons of the stacked LSTM are grouped, different groups learn different co-occurrence features, and the group feature Z _t is obtained. Do this in the following way:

(1) First, establish an end-to-end fully connected deep layered LSTM network model to achieve time-series feature learning and motion modeling; it aims to reliably model complex relationships between different individuals, using LSTM layers and feedforward layers to alternately deploy A deep network to capture motion information, as shown in Figure 2, the feedforward layer is between the two layers of LSTM networks, its role is to make each layer of neurons fully connected to the neurons of the next layer, there is no existence between neurons Same-layer connections, and no cross-layer connections.

In order to ensure that the model learns effective features, different types of regularization are implemented in different parts of the model to effectively alleviate the problem of overfitting (the model is a deep model composed of a fully connected LSTM network + feedforward layer, and the structure is relatively complex , it is easy to cause the problem of over-fitting, so the design of regularization is to reduce the complexity of the model to solve the problem of over-fitting.) Specifically, this embodiment proposes two types of regularization:

1) For fully connected layers, this embodiment introduces regularization to drive the model to learn co-occurrence features of individuals in different layers, and co-occurrence feature learning of nodes between LSTM layers;

2) For LSTM neurons, derive a new Dropout layer and apply it to the LSTM neurons in the last LSTM layer This helps the network learn complex motion dynamics.

(2) Then group the internal neurons of the stacked LSTM, and introduce constraints on the weights of the individual members and the neurons in the objective function, so that the neurons in the same group have greater weights for the subsets composed of some individual members. connections, while connecting other nodes with smaller weights, so as to mine the co-occurrence of individual members.

As shown in Figure 3, the LSTM layer of the main network consists of several LSTM neurons, which are divided into K groups, and each neuron in the same group has a larger connection weight with some individuals in common (ie A subset of several members closely related to a certain group behavior has a larger connection weight), while other related individuals have a smaller connection weight. Different groups of neurons have different degrees of sensitivity to the behavior of different groups, which is reflected in different groups of neurons corresponding to individual subsets of larger connection weights. In terms of implementation, the above co-occurrence mining and utilization can be achieved by adding group sparse constraints to the connections of each group of neurons and individual members.

1) Group the neurons of each layer of LSTM according to the number of group behavior categories, for example, if there are 10 behavior categories, they are divided into 10 groups. Each neuron and individual are fully connected, and the importance of each individual is also different. After training, the neurons can know which individuals are important, thereby highlighting important groups.

Therefore, this embodiment designs a fully connected main network, allowing each neuron to be connected to any individual to automatically explore the co-occurrence characteristics within the group, dividing the neurons in the same layer into θ groups, allowing different Groups focus on discriminating between different behavioral categories. Taking the neurons in the K group as an example, this group of neurons will automatically distinguish different individual behaviors after training, and the co-occurrence regularization is added to the loss function. The design is as follows: (The model uses regularization for two purposes. One is to prevent overfitting, and the other is to incorporate prior information to force the model to learn the effect we want)

Among them, L is the maximum likelihood loss function of the deep LSTM network, W _xβ = [W _x,1 ;...;W _x,K ] is the weight matrix connected to the input unit β, let N represent the number of neurons , these N neurons are divided into θ groups, the number of neurons in each group is ε=[N/θ], for LSTM layer, S={i,f,o,c} represents the input gate in LSTM neurons , forget gate, output gate and cell unit, for the feedforward layer, S={h} represents the neuron itself;

以驱动其可以选择具有不同描述性的特征做为输入，不同的神经元分组探索不同的共现性特征，以便获得识别多种动作类别的能力，然后采用梯度下降法求解。(矩阵W_xβ,k优化，其值变得稀疏，实际上就是对θ组共现性小组群体的特征重要性实现“厚此薄彼”实现选择)。The second term in formula (1) is L1 regularization. During training, L1 regularization reduces quickly for small weights and slows down for large weights, so the weights of the final model are mainly concentrated in high importance. In terms of features, for unimportant features, the weight will quickly approach 0, and then a relatively important subset of key figures can be determined. In the third term, since the L2 norm can encourage the matrix W _xβ,k to become sparse, the use of the L2 norm for each group of cells is defined as

In order to drive it to select features with different descriptive properties as input, different neuron groups explore different co-occurrence features in order to obtain the ability to identify multiple action categories, and then use the gradient descent method to solve. (The matrix W _xβ,k is optimized, and its value becomes sparse, which is actually the realization of "favor one over the other" for the feature importance of the θ group co-occurrence group group).

During training, the network randomly drops some neurons to force the remaining network units to compensate, and during testing, the network uses all neurons together to make predictions. Extending this idea to LSTM networks, this scheme uses a new gradient descent algorithm that allows selective gradient descent of internal gates, cells and output responses of LSTM neurons, encouraging each cell to learn better parameters, such as Figure 4 shows LSTM neurons in an expanded form. For LSTM networks, it is not desirable to erase all information in the unit because the unit remembers events that occurred over a long period of time in the past. Therefore, as shown in Figure 4, allowing The dropout effects in the LSTM flow along the dashed lines of the layers, prohibiting it from flowing along the time axis.

The response of a cell transmitted in the time direction without loss is:

The response for a cell with a loss is:

where m _i , m _f , m _c , m _o and m _h are the input gate, forget gate, cell memory unit, output gate and the loss binary mask vector of the output response, respectively, and an element value of 0 indicates that a loss has occurred. For the first LSTM layer, the input _xt is the obtained single-person action feature; for higher LSTM layers, the input _xt is the response output of the previous layer.

In step D, the bounding box information in step A is input into the key time segment candidate sub-network for feature extraction to obtain the importance weight of the current frame, that is, the importance β _t of the current frame, which is specifically implemented in the following manner:

(1) First, establish the key time segment candidate sub-network, as shown in Figure 1, the key time segment candidate sub-network includes serially connected CNN and LSTM layers, fully connected layers and Relu nonlinear units;

(2) Then, according to the video sequence input to the sub-network, use the Relu unit to obtain the behavior attribute score in the current frame, that is: o _t =RELU(w _x' x _t +w _h' h' _t-1 +b' )=(o ₁ ,o ₂ ,...o _C ) ^t , C represents the total number of action categories, t represents the current frame, and its size depends on the current input x _t , the hidden state h't of the LSTM layer at time _t -1 _-1 .

For a sequence of video frames, the amount of valuable information provided by different frames is usually unequal, and only some frames contain the most distinguishing information, while other frames provide contextual information as a supplement. For example, for the group behavior of spiking in a volleyball game, action frames such as catching, jumping, etc., are less important than the spiking frame.

(3) Finally, according to the correlation degree between the behavior attribute score of the current frame and the group behavior attribute, the importance weight β _t of the current frame in the input sequence T is obtained, specifically:

Given a group action category set G _{_action} =(A ₁ ,A ₁ ,...,A _q ) ^T , the temporal importance weight β _t can be calculated by calculating the set and the current frame action attribute score o _t =(o ₁ , o ₂ ,...o _C ) ^t is represented by the intersection similarity coefficient of these two sets, namely:

Among them, ∩ means to calculate the intersection of two sets, and n() means to solve the number of elements in the set.

In step E, combine the group feature Z _t obtained in step C with the importance weight β _t of the current frame obtained in step D to obtain the group feature Z' _t of the current frame, and input it to the softmax layer. Carry out the final group behavior identification, which is implemented in the following ways:

(1) First calculate the frame group characteristics at time t as:

Z' _t = Z _t ·β _t

(2) Then input it to the softmax layer for final group behavior recognition:

y=softmax(Z' _t )

Among them, y is the group behavior category.

The purpose of designing the sub-network in the present invention is to enable the network to pay attention to different individuals at different levels and assign different importance to different frames. As shown in Figure 1 of the network model, the key person candidate sub-network mainly acts on the main LSTM. The input, key-time segment candidate sub-network mainly acts on the output of the main LSTM network. The purpose of the two sub-networks is to enable the network to pay different levels of attention to different individuals and assign different levels of importance to different frames.

This scheme integrates key person candidates and key time period candidate sub-networks in the same network. The key person candidate sub-network acts on the input of the main LSTM network, and the key time period candidate sub-network acts on the output of the main LSTM network. The final objective function of these two sub-networks is formulated as a sequence with a regularized cross-entropy loss as follows:

表示将序列预测为第i类的概率标量。λ1，λ2和λ3平衡了三个正则项的贡献。where y=(y1,..,yC) denotes the true label, if it belongs to the i-th class, y _i =1 and y _j =0 for j not equal to i.

A scalar representing the probability of predicting the sequence as class i. λ1, λ2 and λ3 balance the contributions of the three regularization terms.

The first regularization term aims to encourage the key person candidate sub-network to dynamically focus on more spatial nodes in the sequence. This example found that the network model tends to ignore many individuals over time, even if these individuals are valuable for determining the type of action, i.e. trapped in local optimal positions, therefore, this regularization term is introduced to avoid such ill-posed solutions Program. The second regularization term is to regularize the learned key-period candidate sub-network under the control of the L2 norm, instead of increasing them indefinitely. This alleviates vanishing gradients in backpropagation, where the gradient of backpropagation is proportional to 1/βt. The third regularization term of the L1 norm is to reduce the overfitting of the network, and W _uv represents the connection matrix in the network.

Finally, considering the complexity of the model, a joint training strategy of the main network and the sub-network is proposed to enable the model to achieve better results. Specifically, the joint training process of the network is as follows:

Due to the interaction of these three networks, including two loss functions, the optimization work is quite difficult. Therefore, the joint training strategy proposed in this scheme can effectively train the entire model. The training process is as follows:

Input: The number of training times of the model N1, N2 (for example, N1=1000, N2=500)

1. Use the Gaussian function to initialize the network parameters;

2. Fix the weight of the key person candidate sub-network (according to the initialization weight), and jointly train the main network of the key time period candidate sub-network with only one LSTM layer of the main network to obtain the key time period candidate model;

3. Repeat the iteration, and after N1 iterations increase its LSTM layer to three layers, train the main network;

4. Fine-tune the main network and candidate sub-networks for key time periods through N2 iterations;

5. Fix the key time period candidate sub-network, and jointly train the key person candidate sub-network and the main network with only one LSTM layer to obtain the key person candidate sub-network;

6. Repeat the iteration, and after N1 iterations increase its LSTM layer to three layers, train the main network;

7. Fine-tune the main network and the candidate sub-network for this key time period through N2 iterations;

8. The sub-network obtained in steps 4 and 7 is jointly trained by N1 iterations;

9. Jointly fine-tune the entire network model through N2 iterations;

Output: The entire group behavior recognition model is finally converged.

For the group behavior recognition method proposed in this scheme, the key person candidate sub-network is used to assign different importance weights to each person in the group, so as not to lose information, but also to pay attention to those who contribute greatly to group behavior recognition. In addition, the candidate sub-network of the key time period is used to assign different importance weights to each frame, no frame is discarded, and no data is missing, and the model is continuously trained and iterated, which can greatly improve the performance of group behavior recognition. Efficiency and Precision.

In addition, the current expression of the interaction relationship is to use the graph model to model the interaction relationship between the characters in the group, the data is huge, and the model training is difficult, and this solution uses the co-occurrence to deal with the interaction within the group behavior. relationship, using a fully connected layered bidirectional LSTM, and grouping its neurons, different groups recognize different behaviors, and further effectively improve the accuracy of group behavior recognition.

The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications to equivalent changes. The embodiments are applied to other fields, but any simple modifications, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention still belong to the protection scope of the technical solutions of the present invention without departing from the content of the technical solutions of the present invention.

Claims (9)

1. The group behavior identification method based on key spatio-temporal information driving and group co-occurrence structured analysis is characterized by comprising the following steps of:

step A, aiming at a video to be identified, tracking each member in the video to obtain a bounding box image x of the member_tInputting the data into the key character candidate sub-network in time sequence for static and dynamic feature extraction, and identifying the personal behavior attribute to obtain the personal importance weight α_t；

Step B, the personal importance weight α obtained in the step A is weighted_tAnd personal bounding box image x_tInputting the spatial characteristics X 'to the laminated L STM network'_t＝x_t*α_t；

Step C, performing co-occurrence feature modeling by taking the output of the step B as input, grouping neurons of the laminated L STM, learning different co-occurrence features by different groups, and obtaining group feature Z_t；

Step D, the bounding box image x in the step A is processed_tInputting the data into the candidate sub-network of the key time slice for feature extraction to obtain the importance weight β of the current frame_t；

Step E, grouping characteristics Z obtained in the step C_tAnd the importance weight β of the current frame obtained in step D_tIn combination, obtaining a group feature Z 'of the current frame'_tAnd Z 'is'_tAnd inputting to softmax, and finishing the identification of the group behaviors.

2. The base of claim 1The group behavior identification method based on key space-time information drive and group co-occurrence structural analysis is characterized in that in the step A, personal importance weights α are obtained_tThe method is realized by the following specific steps:

(1) firstly, establishing a key character candidate sub-network, wherein the key character candidate sub-network comprises a CNN and L STM layer, a full connection layer, a tanh activation function and a full connection layer which are connected in series;

(2) secondly, acquiring a behavior attribute score of the group members, specifically:

at the t-th moment, setting M members in the scene, and setting the combination of the feature sets of the boundary frames extracted by the CNN network as x_t＝(x_t，1,,...,x_t，M)^TBehavior attribute is given as s_t＝(s_t,1,...,s_t,M)^TThe behavior attribute score represents the behavior category judgment of the M members, and is expressed as:

where T is the video sequence length, U_s,W_xs,W_hsTo learnable parameter matrices, b_s,b_usIn order to be a vector of the offset,

represents hidden variables from the L STM layer;

(3) finally, the importance weight of each member is obtained, and further a key figure is determined, specifically:

given set of group behavior classes G_{_action}＝(A₁,A₁,...,A_q)^TFor the kth individual, its behavioral attribute score s is calculated_t,kAt G_{_action}Can be measured by the cosine similarity of the two multidimensional vectors:

| | represents a 2 norm;

the conversion to cosine angle normalization coefficients is:

the importance weight of each person in the spatial scene is calculated by the following formula:

α_t,kto determine how much each member contributes to the group behavior recognition task.

3. The group behavior identification method based on key spatiotemporal information-driven and group co-occurrence structured analysis according to claim 1, characterized in that: the step B is specifically realized by the following steps:

(1) firstly, the spatial characteristics of the kth individual at time t are obtained through importance weight modulation:

x'_t,k＝α_t,k·x_t,k

(2) then, aggregating all the importance weight modulated individual spatial features as an input of an overlay L STM in the main network, and obtaining:

X'_t＝(x'_t,1,...,x'_t,K)。

4. the group behavior identification method based on key spatiotemporal information-driven and group co-occurrence structured analysis according to claim 1, characterized in that: in the step C, when performing co-occurrence feature learning, the following method is specifically adopted:

step C1, firstly, establishing an end-to-end full connection depth L STM network model to realize time sequence feature learning and motion modeling;

a deep network is formed on the basis of L STM layers and feedforward layers which are alternately deployed to capture motion information, and the feedforward layer is arranged between two layers of L STM networks so that neurons in each layer are completely connected with neurons in the next layer;

and C2, grouping the neurons of the laminated L STM, and introducing constraint on the weight of the connected member individuals and the neurons into an objective function, so that the neurons in the same group have more weight connection on a subset formed by some member individuals and have less weight connection on other nodes, thereby mining the co-occurrence of the member individuals.

5. The method for identifying group behaviors based on key spatio-temporal information driving and group co-occurrence structural analysis according to claim 4, wherein in the step C1, in order to ensure that the full-connectivity depth L STM network model learns effective features, different types of regularization are implemented in different parts of the model, and the method specifically comprises two types of regularization:

1) for fully connected layers, regularization is introduced to drive the model to learn co-occurrence features of individuals of different layers, and co-occurrence feature learning of nodes between L STM layers;

2) for the L STM neuron, a new Dropout layer was derived and applied to the L STM neuron in the last L STM layer.

6. The group behavior identification method based on key spatiotemporal information-driven and group co-occurrence structured analysis according to claim 5, characterized in that: in the step C2, mining and utilizing co-occurrence are realized by adding group sparsity constraint to the connection of each group of neurons and member individuals:

(1) grouping neurons of each layer L STM according to the group behavior category number, for the K-th group of neurons, automatically distinguishing different individual behaviors through training, and adding co-occurrence regularization in a loss function:

where L is the maximum likelihood loss function of the deep L STM network, W_xβ＝[W_x,1,...,W_x,K]Is a weight matrix connected to the input unit β and set withN represents the number of neurons, the N neurons being divided into θ groups, the number of neurons in each group being [ N/θ ═]For the L STM layer, S ═ { i, f, o, c } denotes the input gate, forgetting gate, output gate, and cell in the L STM neuron, and for the feed forward layer, S ═ { h } denotes the neuron itself;

the second term in equation (1) is L1 regularization, which is used during training to determine a relatively important subset of key people, and in the third term, the matrix W may be encouraged due to the L2 norm_xβ,kBecomes sparse, so the L2 norm is used for each set of cells to define

The method is characterized in that the method is driven to select features with different descriptors as input, different neuron groups explore different co-occurrence features, and then a gradient descent method is adopted for solving.

7. The group behavior identification method based on key spatiotemporal information-driven and group co-occurrence structured analysis according to claim 1, characterized in that: the step D is specifically realized by the following steps:

(1) firstly, establishing a key time slice candidate sub-network, wherein the key time slice candidate sub-network comprises a CNN and L STM layer, a full connection layer and a Relu nonlinear unit which are connected in series;

(2) then, according to the video sequence input into the sub-network, a behavior attribute score in the current frame is obtained by using a Relu unit, namely: o_t＝RELU(w_x'x_t+w_h'h'_t-1+b')＝(o₁,o₂,...o_C)^tC denotes the total number of behavior classes, t denotes the current frame, the size of which depends on the current input x_tAnd L hidden state h 'of STM layer at time t-1'_t-1；

(3) Finally, according to the degree of association between the current frame behavior attribute score and the group behavior attribute, the importance weight β of the current frame in the input sequence T is obtained_tThe method specifically comprises the following steps:

given set of group behavior classes G_{_action}＝(A₁,A₁,...,A_q)^TTime importance weight β_tMay be determined by calculating the set and current frame behavior attribute scores o_t＝(o₁,o₂,...o_C)^tThe joint similarity coefficient of the two sets is expressed as:

where ∩ represents the intersection calculation and n () represents the number of elements in the solution set.

8. The group behavior identification method based on key spatiotemporal information-driven and group co-occurrence structured analysis according to claim 1, characterized in that: in said step E, based on the group characteristics Z_tAnd importance weight β of the current frame_tThe following method is specifically adopted when group behavior identification is carried out:

(1) firstly, calculating the frame group characteristics at the t-th moment as follows:

Z'_t＝Z_t·β_t

(2) then inputting the data into a softmax layer for final group behavior identification:

y＝soft max(Z'_t)

where y is the group behavior category.

9. The group behavior identification method based on key spatiotemporal information-driven and group co-occurrence structured analysis according to claim 1, characterized in that: based on the complexity of the model, joint training is performed on the main network, the key character candidate sub-networks and the key time slice candidate sub-networks, specifically, the joint training process of the network is as follows:

inputting: training times of the model N1, N2;

(1) initializing network parameters by using a Gaussian function;

(2) fixing the weights of the key character candidate sub-networks, and jointly training a key time period candidate sub-network main network with only one L STM layer of the main network to obtain a key time period candidate model;

(3) repeating iteration, and training the main network after increasing the L STM layer to three layers through N1 iterations;

(4) fine-tuning the primary network and the critical-period candidate sub-network through N2 iterations;

(5) fixing the key time slice candidate sub-networks, and jointly training the key character candidate sub-networks and the main network with only one L STM layer to obtain key character candidate sub-networks;

(6) repeating iteration, and training the main network after increasing the L STM layer to three layers through N1 iterations;

(7) fine-tuning the primary network and the critical time period candidate sub-network through N2 iterations;

(8) obtaining a sub-network through N1 times of iterative joint training (4) and (7);

(9) collectively fine-tuning the entire network model through N2 iterations;

and (3) outputting: and finally converging to obtain the whole group behavior recognition model.

Images