CN110580456A - Group Activity Recognition Method Based on Coherence Constraint Graph Long Short-Term Memory Network - Google Patents

Abstract

The present invention provides a group activity recognition method based on the coherence constraint map long-term memory network, comprising the following steps: (1) using the CNN features of all people as the input of the coherence constraint map long-term and short-term memory network, and jointly learning all people in time and space individual motion states over time under the constraints of contextual coherence; (2) using the attention mechanism of global contextual coherence to quantify the contribution of related motions by learning attention factors corresponding to different motions; (3) in each time step Aggregate LSTM is adopted to aggregate all individual motion states weighted by different attention factors into a hidden representation of the whole activity, and the hidden representation of each activity is input into the softmax classifier; (4) the softmax classifier for each time step The output of the group is averaged to obtain the probability class vector of the group activity, so as to infer the category of the group activity.

Description

Group Activity Recognition Method Based on Coherence Constraint Graph Long Short-Term Memory Network

technical field

The invention relates to an action recognition technology in the field of computer vision, in particular to a group action recognition method based on a coherence constraint graph long-short-term memory network.

Background technique

Traditional action recognition such as single-person recognition and two-person interaction action recognition usually consists of one or two persons appearing in a video, and such tasks have achieved satisfactory performance over the past few decades. Compared with traditional human behavior, group activities are more complex but more common behaviors in the scene. Unlike solo activities and two-person interactions, group activities are usually performed by multiple people at the same time. Therefore, in group activity recognition, we need to model the behavior of multiple individuals and their interactions. This is a fine-grained recognition task that is much more difficult than traditional single-person action recognition or two-person interaction recognition.

Benefiting from the success of Recurrent Neural Networks (RNNs), especially for the development of Long Short-Term Memory Networks (LSTMs), group activity recognition has made some progress in recent years. By reviewing existing deep learning methods related to group activity recognition, a common solution is to first learn individual-level action representations for each person, and then integrate all individual representations to identify group-level activities. Specifically, some of the earlier methods assumed that everyone in the active scene was independent of each other. Subsequently, some methods assume that all persons in the activity scene are interdependent, and model the individual motion of each person with reference to the motion state of others in the group activity. However, the above method considers that all people's movements contribute equally to the group activity, which suppresses the contribution of some coherent movements to the whole activity and exaggerates some outlier movements that have nothing to do with the group activity.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a group activity identification method based on the coherence constraint graph long and short-term memory network.

The technical scheme for realizing the object of the present invention is: a method for identifying group activity based on a coherent constraint graph long-short-term memory network, comprising the following steps:

Step 1, use a pre-trained convolutional neural network CNN model to extract the CNN features of each person in the tracked bounding box;

Step 2: Use the CNN features of all people as the input of the CCG-LSTM coherence constraint map long-term memory network, and jointly learn the individual motion states of all people over time under the constraints of spatiotemporal context coherence;

Step 3: Use the attention mechanism of global context coherence to learn the attention factors corresponding to different movements, and obtain the movement state of each individual under the constraint of global context coherence through the attention factors;

Step 4, in each time step, the aggregated long-short-term memory network LSTM in the coherence constraint graph long-short-term memory network CCG-LSTM aggregates all the motion states of a single individual obtained by different attention factors into a hidden representation of the entire activity;

Step 5. Input the hidden representation of each activity into the softmax classifier at each time step;

In step 6, the output of the softmax classifier at each time step is averaged to infer the class of group activity.

Further, step 1 specifically includes the following steps: for each video segment, the object tracker in the Dlib library is used to track a set of bounding boxes around each person within a certain time step, and the object tracker extracts each bounding box. CNN features of each person in .

Further, in step 1, if the tracker does not track a person in a certain frame, an all-zero matrix is used to make up for the missing feature of the target person in this frame of image.

Further, step 2 specifically includes the following steps:

Step 201, given a video clip of T frames, which describes the group activity containing V individuals, represents the CNN features of the vth person in the tth frame, where t∈{1,2,...,T}, v∈{1,2,...,V};

Step 202, representing the characteristics of group activities as a graph structure in the spatial and temporal domains in E ^t is the adjacency matrix;

Step 203, construct the coherence constraint graph long and short-term memory network CCG-LSTM, the motion state of the vth node in the CCG-LSTM at time step t Calculated as follows

in

where, for the vth node, is the input gate, For the gate of oblivion, is the output gate, For adjoining the gate of oblivion, is the time confidence gate, is the spatial context confidence gate, φ( ) is a multilayer perceptron, represents the motion state of the vth node at time step t-1, Indicates that the motion state i of the i-th node at time step t-1 belongs to the adjacent nodes of the v-th node, represents the relationship weight between the vth node and the ith node at time step t-1, is the spatial context state of the vth person, W _* , U _* , G _* are the weight matrices, b _* is the bias vector, and * refers to the subscripts i, g, o, f and σ( ) represents the sigmoid activation function, represents the tanh activation function, ⊙ represents element-wise multiplication, Φ(v) represents the adjacent node of the vth node, represents the spatial context memory state of the vth node, Yes The spatial context state transformed by the dimension of the _Wp transformation matrix, Yes The feature transformed by the dimension of the W _x transformation matrix, is the variable of the motion state projected to another dimension space, the parameter ρ is the input range of the control function, is the spatial context memory state of the vth node, is the memory state of the corresponding neighbors of the vth node.

Further, step 3 specifically includes the following steps:

Step 301, obtain the average motion state of all individual motion states to represent the hidden representation of the overall activity at that time step, i.e.

Step 302, use an attention model to learn attention factors to measure the contribution of individual activities to the overall activity where γ is a parameter;

Step 303, obtain the motion state of the vth node under the constraint of global context coherence through the attention factor

Further, step 4 specifically includes the following steps:

Use an aggregated LSTM in the spatial domain to aggregate the motion states of all people into a hidden person-to-person representation of the entire activity at time step t:

in, represents the hidden state of the aggregated LSTM, z ^t is the hidden representation of the entire activity at time step t, is the motion state of the vth individual under the constraint of global context coherence.

Further, step 5 specifically includes the following steps:

Putting the hidden representation z ^t (t=1,2,...,T) of the group activity at time step t into the softmax classifier gets y ^t =softmax(z ^t ),t=1,2,... , T.

Further, step 6 specifically includes the following steps:

Average the outputs of all softmax classifiers to get the probability class vector of group activity Thereby, the classification results are obtained.

Compared with the prior art, the present invention has the following advantages: (1) Consider expanding graph LSTMs under the constraints of spatiotemporal context coherence (STCC), and understand group activities by exploring individual motions in both spatial and temporal domains; (2) Measure individuals To quantify the contribution of the movement to the group activity, the consistency of the movement itself with the whole activity under the constraint of Global Context Coherence (GCC) can effectively identify the group activity.

The present invention will be further described below with reference to the accompanying drawings.

Description of drawings

Figure 1 is a flow chart of the present invention.

Figure 2 is a visualization of the group activity recognition method based on the coherence constraint graph long and short-term memory network.

Table 1 shows the recognition accuracy of different methods on the volleyball dataset.

Detailed ways

1. A method for group activity recognition based on coherence constraint graph long-term and short-term memory network, including learning the motion state of individuals under the constraints of spatio-temporal context coherence, quantifying the contribution of individual motion to group activities under the constraints of global context coherence, and using aggregated LSTM to obtain There are four processes of hidden representation of group activity and acquisition of probability class vector of group activity.

Learning the motion state of an individual under the coherence constraints of the spatiotemporal context includes the following steps:

Step 1, use a pre-trained convolutional neural network (CNN) model to extract the CNN features of each person in the bounding box being detected and tracked, and the convolutional neural network used is compatible with AlexNet, VGG, ResNet and GoogLeNet.

Step 2, add a temporal confidence gate to the ordinary graph long short-term memory network (Graph LSTM) and spatial context confidence gates to learn temporal context coherence constraints and spatial context coherence constraints for all individuals.

Step 3: The individual CNN features obtained in step 1 are used as the input of the coherence constraint graph long-term memory network to jointly learn the individual motion states of all individuals over time under the constraints of spatiotemporal context coherence. The steps taken are as follows:

a. Given a video clip of T frames, it describes the group activity of V individuals represents the CNN features of the vth person at frame t, where t∈{1,2,...,T}, v∈{1,2,...,V}.

b. Represent the characteristics of group activities as a graph structure in the spatial and temporal domains:

in E ^t is the adjacency matrix.

c. Construct a long and short-term memory network of coherence constraint graph, for the vth node, is the input gate, For the gate of oblivion, is the output gate, For adjoining the gate of oblivion, is the time confidence gate, is the spatial context confidence gate, φ( ) is a multilayer perceptron, represents the motion state of the vth node at time step t-1, Indicates that the motion state i of the i-th node at time step t-1 belongs to the adjacent nodes of the v-th node, represents the relationship weight between the vth node and the ith node at time step t-1, is the spatial context state of the vth person, W _* , U _* , G _* are the weight matrices, b _* is the bias vector, and * refers to the subscripts i, g, o, f and σ( ) represents the sigmoid activation function, represents the tanh activation function, ⊙ represents element-wise multiplication, Φ(v) represents the adjacent node of the vth node, represents the spatial context memory state of the vth node, Yes The spatial context state transformed by the dimension of the _Wp transformation matrix, Yes The feature transformed by the dimension of the W _x transformation matrix, is the variable of the motion state projected to another dimension space, the parameter ρ is the input range of the control function, is the spatial context memory state of the vth node, refers to the memory state of the corresponding neighbors of the vth node; at time step t, the motion state of the vth node in CCG-LSTM Calculated as follows:

Get the state of individual movement

Quantifying the contribution of individual motion to group activity under the constraints of global context coherence involves the following steps:

Step 4, using the attention mechanism of global context coherence, by learning the attention factors corresponding to different individual movements, the individual states obtained in step 3 are Contribution to group activity is quantified by first taking the average motion state of all individual motion states to approximate the hidden representation of this part of the overall activity, i.e.

Step 5, use an attention model to learn attention factors to measure the contribution of the individual activities of step 3 to the overall activities of step 4 where γ is a parameter.

Step 6, obtain the motion state of the vth node under the constraint of global context coherence through the attention factor of step 5

Using an aggregated LSTM to obtain hidden representations of group activity involves the following steps:

Step 7. Aggregate the motion states of all individuals in step 6) into a hidden person-to-person representation of the entire activity at time step t using an aggregated LSTM in the spatial domain:

in represents the hidden state of the aggregated LSTM, z ^t is the hidden representation of the entire activity at time step t.

Obtaining the probability class vector of group activity involves the following steps:

Step 8, put the hidden representation z ^t (t=1,2,...,T) of the group activity at time step t in step 7 into the softmax classifier to get: y ^t =softmax(z ^t ),t=1 ,2,...,T.

Step 9), average the outputs of all the softmax classifiers in step 8) to obtain the probability class vector of the group activity: Thereby, the classification results are obtained.

Table 1 shows the recognition accuracy of different methods on the volleyball dataset

Claims (8)

1. a group activity recognition method based on coherence constraint graph long short-term memory network, is characterized in that, comprises the following steps: Step 1, use a pre-trained convolutional neural network CNN model to extract the CNN features of each person in the tracked bounding box; Step 2: Use the CNN features of all people as the input of the CCG-LSTM coherence constraint map long-term memory network, and jointly learn the individual motion states of all people over time under the constraints of spatiotemporal context coherence; Step 3: Use the attention mechanism of global context coherence to learn the attention factors corresponding to different movements, and obtain the movement state of each individual under the constraint of global context coherence through the attention factors; Step 4, in each time step, the aggregated long-short-term memory network LSTM in the coherence constraint graph long-short-term memory network CCG-LSTM aggregates all the motion states of a single individual obtained by different attention factors into a hidden representation of the entire activity; Step 5. Input the hidden representation of each activity into the softmax classifier at each time step; In step 6, the output of the softmax classifier at each time step is averaged to infer the class of group activity. 2. method according to claim 1, is characterized in that, step 1 specifically comprises the steps: For each video clip, an object tracker in the Dlib library is used to track a set of bounding boxes around each person within a certain time step, and the object tracker extracts the CNN features of each person in each bounding box. 3 . The method according to claim 2 , wherein in step 1, if the tracker does not track a person in a certain frame, an all-zero matrix is used to make up for the missing feature of the target person in this frame of image. 4 . 4. method according to claim 1, is characterized in that, step 2 specifically comprises the steps: Step 201, given a video clip of T frames, which describes the group activity containing V individuals, represents the CNN features of the vth person in the tth frame, where t∈{1,2,...,T}, v∈{1,2,...,V}; Step 202, express the characteristics of the group activity as a graph structure θ ^t ={S ^t ,E ^t }(t=1,2,...,T) in the spatial and temporal domains, where E ^t is the adjacency matrix; Step 203, construct the coherence constraint graph long and short-term memory network CCG-LSTM, the motion state of the vth node in the CCG-LSTM at time step t Calculated as follows in where, for the vth node, is the input gate, For the gate of oblivion, is the output gate, For adjoining the gate of oblivion, is the time confidence gate, is the spatial context confidence gate, φ( ) is a multilayer perceptron, represents the motion state of the vth node at time step t-1, Indicates that the motion state i of the i-th node at time step t-1 belongs to the adjacent nodes of the v-th node, represents the relationship weight between the vth node and the ith node at time step t-1, is the spatial context state of the vth person, W _* , U _* , G _* are the weight matrices, b _* is the bias vector, and * refers to the subscripts i, g, o, f and σ( ) represents the sigmoid activation function, represents the tanh activation function, ⊙ represents element-wise multiplication, Φ(v) represents the adjacent node of the vth node, Represents the spatial context memory state of the vth node, W _p : W _x : _Wq : Yes The spatial context state transformed by the dimension of the _Wp transformation matrix, Yes The feature transformed by the dimension of the W _x transformation matrix, is the variable of the motion state projected to another dimension space, the parameter ρ is the input range of the control function, is the spatial context memory state of the vth node, is the memory state of the corresponding neighbors of the vth node. 5. method according to claim 1, is characterized in that, step 3 specifically comprises the steps: Step 301, obtain the average motion state of all individual motion states to represent the hidden representation of the overall activity at that time step, i.e. Step 302, use an attention model to learn attention factors to measure the contribution of individual activities to the overall activity where γ is a parameter; Step 303, obtain the motion state of the vth node under the constraint of global context coherence through the attention factor 6. method according to claim 1, is characterized in that, step 4 specifically comprises the steps: Use an aggregated LSTM in the spatial domain to aggregate the motion states of all people into a hidden person-to-person representation of the entire activity at time step t: in, represents the hidden state of the aggregated LSTM, z ^t is the hidden representation of the entire activity at time step t, is the motion state of the vth individual under the constraint of global context coherence. 7. The method according to claim 1, wherein step 5 specifically comprises the steps: Putting the hidden representation z ^t (t=1,2,...,T) of the group activity at time step t into the softmax classifier gets y ^t =soft max(z ^t ),t=1,2,... ., T. 8. method according to claim 1, is characterized in that, step 6 specifically comprises the steps: Average the outputs of all softmax classifiers to get the probability class vector of group activity Thereby, the classification results are obtained.