CN108229355B - Behavior recognition method and device, electronic device, computer storage medium - Google Patents

Abstract

Embodiments of the present disclosure disclose a method and device, electronic device, computer storage medium, and program for behavior recognition, wherein the method includes: performing human key point detection on at least one frame of video image, and obtaining multiple persons of the at least one frame of video image. body key points; based on the feature information of multiple human body key points of the at least one frame of video image and the associated information of the multiple human body key points, the behavior recognition result of the at least one frame of video image is obtained. By combining the feature information of the key points of the human body and the association information between the key points of the human body, the embodiments of the present disclosure make full use of the local information and the overall information, and improve the accuracy of behavior recognition.

Description

Behavior recognition method and device, electronic device, computer storage medium

technical field

The present disclosure relates to computer vision technology, and in particular, to a behavior recognition method and apparatus, electronic device, and computer storage medium.

Background technique

Behavior recognition is to identify the actions or behaviors of people from videos, such as swimming, running, sweeping the floor, etc. Behavior recognition plays an important role in understanding the content and meaning of videos. Behavior recognition can take video images, speech, or coordinates of human key points as input, and use neural networks to output behavior categories.

SUMMARY OF THE INVENTION

A behavior recognition technology provided by an embodiment of the present disclosure.

According to an aspect of the embodiments of the present disclosure, a behavior recognition method is provided, including:

Performing human body key point detection on at least one frame of video image to obtain multiple human key points of the at least one frame of video image;

Based on the feature information of multiple human body key points of the at least one frame of video image and the associated information of the multiple human body key points, a behavior recognition result of each frame of video image in the at least one frame of video image is obtained.

In another embodiment based on the above method of the present invention, the feature information of the human body key points includes coordinate information of the human body key points; or,

The feature information of the human body key points includes coordinate information of the human body key points, estimated confidence levels of the human body key points and/or initial features corresponding to the human body key points.

In another embodiment based on the above method of the present invention, the association information of the multiple human body key points includes any one or more of the following: the space between at least two human body key points in the same frame of video image association information, and temporal association information between at least two human body key points corresponding to the same human body part and belonging to adjacent frames of video images in the at least one frame of video image.

The temporal correlation information between at least two human body key points corresponding to the same human body part and belonging to different frames of video images in the at least one frame of video image is used to indicate that the human body part will follow in the at least one frame of video image. The trajectory of movement in time.

In another embodiment based on the above method of the present invention, the at least one frame of video image is specifically multiple frames of continuous video images in the video; and/or

The spatial correlation information between at least two human body key points in the same frame of video image is determined according to the connectivity relationship of human body structures.

In another embodiment based on the above method of the present invention, the spatial association information between the at least two human body key points includes a spatially adjacent relationship between the at least two key points, and/or

The temporal association information between the at least two key points includes: the adjacent relationship of the frames to which the at least two key points belong.

In another embodiment based on the above method of the present invention, after performing human body key point detection on at least one frame of video image and obtaining multiple human body key points of the at least one frame of video image, the method further includes:

A spatiotemporal map is established based on multiple human body key points in the at least one frame of video image, wherein the spatiotemporal map includes feature information of multiple human key points in the at least one frame of video image and the multiple human key points in the at least one frame of video image. Relevant information of key points of the individual;

The obtaining of the behavior recognition result of the at least one frame of video image based on the feature information of the multiple human body key points and the associated information of the multiple human body key points in the at least one frame of video image includes:

Based on the space-time map, a behavior recognition result of the at least one frame of video image is obtained.

In another embodiment based on the above method of the present invention, the space-time graph includes a plurality of nodes corresponding to the plurality of human body key points, and each node in the plurality of nodes includes a feature of a corresponding human body key point information;

Each of the plurality of nodes has at least one edge, and the plurality of edges of the plurality of nodes indicate the association relationship of the plurality of human body key points.

At least one edge of the human body key point indicates the relationship between the human body key point and other human body key points.

In another embodiment based on the above method of the present invention, a first node in the plurality of nodes has a spatial edge with each second node in the at least one second node, wherein the first node corresponding to the first node has a spatial edge. A human body key point and a second human body key point corresponding to each of the at least one second node belong to the same frame, and the first human body key point and the human body corresponding to each of the second human body key points sites are directly connected, and/or

There is a time edge between the first node and each of the at least one third node, wherein the first human body key point corresponds to the same third human body key point corresponding to each of the third nodes body parts and belong to adjacent frames.

The number of the plurality of nodes is equal to the number of the plurality of human body key points, and the plurality of nodes are in one-to-one correspondence with the plurality of human body key points.

In another embodiment based on the above method of the present invention, the establishing a space-time map based on a plurality of human body key points in the at least one frame of video image includes:

According to the connection relationship of human body structure, use spatial edge to connect at least two human body key points located in the same frame of video image;

At least two human body key points of the same body part in adjacent frames of the at least one frame of video image are connected using a temporal edge.

In another embodiment based on the above method of the present invention, the obtaining the behavior recognition result of the at least one frame of video image based on the space-time graph includes:

The space-time graph is input into a convolutional neural network to obtain a behavior recognition result of the at least one frame of video image.

In another embodiment based on the above method of the present invention, the step of inputting the space-time graph into a convolutional neural network for processing to obtain a behavior recognition result of the at least one frame of video image includes:

Based on the association information between the multiple human body key points, convolution processing is performed on the multiple human body key points to obtain a convolution processing result of the multiple human body key points;

Based on the convolution processing result of the multiple human body key points, the behavior recognition result of the at least one frame of video image is obtained.

In another embodiment based on the above method of the present invention, the convolution process is performed on the plurality of human body key points based on the associated information of the plurality of human body key points to obtain the volume of the plurality of human body key points Product processing results, including:

Based on the association information of the plurality of human body key points, determining at least one fifth human body key point that has an associated relationship with the fourth human body key point in the plurality of human body key points;

Based on the fourth human body key point and the feature information of each human body key point in the at least one fifth human body key point, a convolution processing result of the fourth human body key point is obtained.

The at least one fifth human body key point includes at least one human body key point that has a spatial relationship with the fourth human body key point; or

The at least one fifth human body key point includes at least one human body key point having a spatial relationship with the fourth human body key point, and at least one human body key point having a temporal relationship with the fourth human body key point.

In another embodiment based on the above method of the present invention, the fourth human body key point is obtained based on the feature information of each human body key point in the fourth human body key point and the at least one fifth human body key point Convolution processing results of points, including:

Using the convolution parameters corresponding to the set of human body key points to which each of the fourth human body key points and the at least one fifth human body key point belongs, the convolution process is performed on each of the human body key points, obtain the initial convolution result of each human body key point;

Based on the fourth human body key point and the initial convolution result of each human body key point in the at least one fifth human body key point, a convolution processing result of the fourth human body key point is obtained.

In another embodiment based on the above method of the present invention, before performing the convolution processing on each of the human body key points, the method further includes:

dividing the fourth human body key point and the at least one fifth human body key point into at least one human body key point set, wherein each human body key point set includes at least one human body key point;

The convolution parameter of each human body key point is determined based on the set of human body key points to which each human body key point in the fourth human body key point and the at least one fifth human body key point belongs, wherein different human body key points belong to different human body key points. The human keypoints of the point set correspond to different convolution parameters.

In another embodiment based on the above method of the present invention, the at least one human body key point set includes a first human body key point set and a second human body key point set;

The described fourth human body key point and the at least one fifth human body key point are divided into at least one human body key point set, including:

The fourth human body key point is divided into the first human body key point set, and the at least one fifth human body key point is divided into the second human body key point set.

In another embodiment based on the above method of the present invention, the dividing the fourth human body key point and the at least one fifth human body key point into at least one human body key point set includes:

Based on the distance between each of the fourth human body key points and the at least one fifth human body key point and the reference point, the fourth human body key point and the at least one fifth human body key point Divide into at least one human body keypoint set.

In another embodiment based on the above method of the present invention, based on the distance between each of the fourth human body key points and the at least one fifth human body key point and a reference point, the The fourth human body key point and the at least one fifth human body key point are divided into at least one human body key point set, including:

determining a first distance between the fourth human body key point and the reference point based on the feature information of the fourth human body key point;

determining the each person based on the magnitude relationship between the distance between each of the fourth human body key points and the at least one fifth human body key point and the reference point and the first distance The set of keypoints to which the body keypoint belongs.

In another embodiment based on the above method of the present invention, the distance between each of the fourth human body key points and the at least one fifth human body key point and the reference point is the same as the distance between each human body key point and the reference point. The size relationship between the first distances, and determine the key point set to which each human body key point belongs, including:

It is determined that the human body key points whose distance from the reference point is less than the first distance belong to the first set of key points; and/or

It is determined that the human body key point whose distance from the reference point is equal to the first distance belongs to the second set of key points; and/or

It is determined that the human body key points whose distance from the reference point is greater than the first distance belong to the third set of key points.

In another embodiment based on the above method of the present invention, obtaining the behavior recognition result of the at least one frame of video image based on the convolution processing result of the plurality of human body key points includes:

Perform global pooling processing on the convolution processing result of each human body key point in the plurality of human body key points included in the space-time map to obtain a pooling processing result;

Based on the pooling processing result, a behavior recognition result of the at least one frame of video image is obtained.

In another embodiment based on the above method of the present invention, the pooling processing result includes a one-dimensional feature vector;

The behavior recognition result of each frame of video image in the at least one frame of video image is obtained based on the pooling processing result, including:

Using the fully connected layer, the one-dimensional feature vector is processed to obtain a recognition vector, and the recognition vector includes a vector value corresponding to the number of behavior classification categories;

The human action classification in the video image is obtained based on each of the vector values in the identification vector.

According to another aspect of the embodiments of the present disclosure, a behavior recognition device is provided, comprising:

a key point detection unit, configured to perform human key point detection on at least one frame of video image, and obtain a plurality of human key points of the at least one frame of video image;

A behavior recognition unit, configured to obtain a behavior recognition result of the at least one frame of video image based on the feature information of multiple human body key points in the at least one frame of video image and the associated information of the multiple human body key points.

In another embodiment based on the above device of the present invention, the feature information of the human body key points includes coordinate information of the human body key points; or,

The feature information of the human body key points includes coordinate information of the human body key points, estimated confidence levels of the human body key points and/or initial features corresponding to the human body key points.

In another embodiment of the above device based on the present invention, the association information of the multiple human body key points includes any one or more of the following: the space between at least two human body key points in the same frame of video image association information, and temporal association information between at least two human body key points corresponding to the same human body part and belonging to adjacent frames of video images in the at least one frame of video image.

In another embodiment based on the above device of the present invention, the at least one frame of video image is specifically multiple frames of continuous video images in the video; and/or

The spatial correlation information between at least two human body key points in the same frame of video image is determined according to the connectivity relationship of human body structures.

In another embodiment based on the above-mentioned apparatus of the present invention, the spatial correlation information between the at least two human key points includes a spatially adjacent relationship between the at least two key points, and/or

The temporal association information between the at least two key points includes: the adjacent relationship of the frames to which the at least two key points belong.

In another embodiment of the above device based on the present invention, it also includes:

A graph establishing unit, configured to establish a space-time graph based on a plurality of human body key points in the at least one frame of video image, wherein the space-time graph includes a plurality of human body key points in the at least one frame of video image. feature information and associated information of the multiple human body key points;

The behavior recognition unit is specifically configured to obtain a behavior recognition result of the at least one frame of video image based on the space-time graph.

In another embodiment of the above device based on the present invention, the space-time graph includes a plurality of nodes corresponding to the plurality of human body key points, and each node of the plurality of nodes includes a feature of a corresponding human body key point information;

Each of the plurality of nodes has at least one edge, and the plurality of edges of the plurality of nodes indicate the association relationship of the plurality of human body key points.

In another embodiment of the above device based on the present invention,

A first node in the plurality of nodes has a spatial edge with each second node in the at least one second node, wherein the first human body key point corresponding to the first node and the at least one second node have a space edge. The second human body key point corresponding to each second node belongs to the same frame, and the first human body key point is directly connected to the human body part corresponding to each second human body key point, and/or

There is a time edge between the first node and each of the at least one third node, wherein the first human body key point corresponds to the same third human body key point corresponding to each of the third nodes body parts and belong to adjacent frames.

In another embodiment based on the above device of the present invention, the behavior recognition unit is specifically configured to input the space-time graph into a convolutional neural network to obtain a behavior recognition result of the at least one frame of video image.

In another embodiment of the above device based on the present invention, the behavior identification unit includes:

a convolution processing module, configured to perform convolution processing on the plurality of human body key points based on the association information between the plurality of human body key points to obtain convolution processing results of the plurality of human body key points;

The convolution identification module is configured to obtain the behavior identification result of the at least one frame of video image based on the convolution processing results of the multiple human body key points.

In another embodiment based on the above device of the present invention, the convolution processing module includes:

an association determination module, configured to determine at least one fifth human body key point that has an associated relationship with the fourth human body key point in the plurality of human body key points based on the association information of the plurality of human body key points;

A feature processing module, configured to obtain a convolution processing result of the fourth human body key point based on the feature information of each human body key point in the fourth human body key point and the at least one fifth human body key point.

In another embodiment based on the above device of the present invention, the feature processing module is specifically configured to use the human body to which each human body key point belongs to the fourth human body key point and the at least one fifth human body key point Convolution parameters corresponding to the set of key points, performing convolution processing on each of the human body key points to obtain an initial convolution result of each of the human body key points;

Based on the fourth human body key point and the initial convolution result of each human body key point in the at least one fifth human body key point, a convolution processing result of the fourth human body key point is obtained.

In another embodiment of the above device based on the present invention, the behavior identification unit further includes:

a classification module, configured to divide the fourth human body key point and the at least one fifth human body key point into at least one human body key point set, wherein each human body key point set includes at least one human body key point;

A parameter determination module, configured to determine the convolution parameter of each human body key point based on the human body key point set to which each human body key point in the fourth human body key point and the at least one fifth human body key point belongs, Among them, the human body key points belonging to different human body key point sets correspond to different convolution parameters.

In another embodiment based on the above device of the present invention, the at least one human body key point set includes a first human body key point set and a second human body key point set;

The classification module is specifically configured to divide the fourth human body key point into the first human body key point set, and divide the at least one fifth human body key point into the second human body key point set.

In another embodiment based on the above device of the present invention, the classification module is specifically configured to be based on the relationship between each of the fourth human body key points and the at least one fifth human body key point and a reference point The distance between the fourth human body key point and the at least one fifth human body key point is divided into at least one human body key point set.

In another embodiment of the above device based on the present invention, the classification module includes:

a first distance module, configured to determine a first distance between the fourth human body key point and the reference point based on the feature information of the fourth human body key point;

a first relationship module, configured to be based on the size of the distance between each of the fourth human body key points and the at least one fifth human body key point and the reference point and the first distance relationship, to determine the set of key points to which each of the human body key points belongs.

In another embodiment based on the above device of the present invention, the first relationship module is specifically configured to determine that a human body key point whose distance from the reference point is smaller than the first distance belongs to a first set of key points; and / or

It is determined that the human body key point whose distance from the reference point is equal to the first distance belongs to the second set of key points; and/or

It is determined that the human body key points whose distance from the reference point is greater than the first distance belong to the third set of key points.

In another embodiment based on the above device of the present invention, the convolution identification module is specifically configured to perform the convolution processing result of each human body key point in the multiple human body key points included in the space-time map. Global pooling processing to obtain the pooling processing result;

Based on the pooling processing result, a behavior recognition result of the at least one frame of video image is obtained.

According to another aspect of the embodiments of the present disclosure, an electronic device is provided, including a processor, where the processor includes the above-mentioned behavior recognition apparatus.

According to another aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a memory for storing executable instructions;

and a processor for communicating with the memory to execute the executable instructions to complete the behavior recognition method as described above.

According to another aspect of the embodiments of the present disclosure, there is provided a computer storage medium for storing computer-readable instructions, and when the instructions are executed, the above-described behavior recognition method is performed.

According to another aspect of the embodiments of the present disclosure, there is provided a computer program, comprising computer-readable code, when the computer-readable code is executed on a device, the processor in the device executes the program for implementing the above-mentioned Instructions for behavior recognition methods.

According to yet another aspect of the embodiments of the present disclosure, a computer program product is provided for storing computer-readable instructions, which, when executed, cause a computer to execute the behavior recognition method described in any of the above possible implementation manners.

In an optional implementation manner, the computer program product is specifically a computer storage medium, and in another optional implementation manner, the computer program product is specifically a software product, such as SDK or the like.

Embodiments of the present disclosure further provide another behavior recognition method and a corresponding device, electronic device, computer storage medium, computer program, and computer program product, wherein the method includes: performing human key point detection on at least one frame of video image to obtain multiple human body key points of at least one frame of video image; based on the feature information of multiple human body key points of at least one frame of video image and the associated information of multiple human key points, the behavior recognition result of at least one frame of video image is obtained.

Based on the behavior recognition method and device, electronic device, computer storage medium, and program provided by the foregoing embodiments of the present disclosure, key point detection is performed on at least one frame of video image, and multiple human key points of at least one frame of video image are obtained; The feature information of multiple human body key points of at least one frame of video image and the associated information of multiple human body key points are obtained, and the behavior recognition result of at least one frame of video image is obtained. The disadvantages of local information, by combining the feature information of human body key points and the correlation information between human body key points, make full use of local information and overall information, and improve the accuracy of behavior recognition.

The technical solutions of the present disclosure will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

The accompanying drawings, which form a part of the specification, illustrate embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure.

The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, wherein:

FIG. 1 is a schematic flowchart of a behavior recognition method provided by an embodiment of the present disclosure.

FIG. 2 is a schematic structural diagram of a space-time graph constructed in an optional example of the behavior recognition method of the present disclosure.

3a-d are specific example diagrams of the behavior recognition method provided by the embodiment of the present disclosure.

FIG. 4 is a schematic flowchart of a specific example of the behavior recognition method of the present disclosure.

FIG. 5 is a schematic structural diagram of an embodiment of the behavior recognition apparatus of the present disclosure.

FIG. 6 is a schematic structural diagram of an electronic device used to implement an embodiment of the present application.

Detailed ways

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

Meanwhile, it should be understood that, for the convenience of description, the dimensions of various parts shown in the accompanying drawings are not drawn in an actual proportional relationship.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application or uses in any way.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

Embodiments of the present disclosure may be applied to computer systems/servers that are operable with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments and/or configurations suitable for use with computer systems/servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, Microprocessor systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the foregoing, among others.

A computer system/server may be described in the general context of computer system-executable instructions, such as program modules, being executed by the computer system. Generally, program modules may include routines, programs, object programs, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer systems/servers may be implemented in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

In the process of realizing the present disclosure, the inventors found that there are at least the following problems: the methods in the prior art, whether using the LSTM network or the ResNet network, do not consider the natural spatial connection between the key points of the human body.

The existing technology ignores the connection of adjacent key points, and indiscriminately processes the key points as vectors and directly inputs them into the network. The network considers all human body key points at the beginning, which is not conducive to the model's attention to local information.

FIG. 1 is a schematic flowchart of a behavior recognition method provided by an embodiment of the present disclosure. As shown in Figure 1, the method of this embodiment includes:

Step 110: Perform key point detection on at least one frame of video image to obtain multiple human body key points of at least one frame of video image.

Specifically, the at least one frame of video image may be derived from a video collected by the device itself, a video input by a user, or a video obtained from other devices. The embodiment of the present disclosure does not limit the manner of obtaining the at least one frame of video image. The at least one frame of video images may specifically be one or more frames of video images, and the embodiment of the present disclosure does not limit the number of the at least one frame of video images. Optionally, the at least one frame of video image may be a continuous video image, for example, the at least one frame of video image may belong to a video segment corresponding to an action or behavior, and a behavior recognition result can be obtained based on this video segment. Or, optionally, each frame of video image in the at least one frame of video image may correspond to an action or behavior, for example, the at least one frame of video image is discontinuous and one frame is collected by spacing a set number of frames in the video obtained from a video image, but this is not limited in this embodiment of the present disclosure.

Specifically, human body key point detection can be performed on each frame of video image in at least one frame of video image to obtain at least one human body key point in each frame of video image. In this way, multiple human body key points of the at least one frame of video image can be obtained. point. Wherein, optionally, a machine learning method can be used to perform key point detection, for example, a neural network, a support vector machine (Support Vector Machine), a random forest (Random Forest, RF), etc. The implementation of the key point detection in the embodiments of the present disclosure Not limited.

Step 120: Obtain a behavior recognition result of at least one frame of video image based on feature information of multiple human body key points and associated information of multiple human body key points in at least one frame of video image.

In one or more optional embodiments, the feature information of the human body key points may include coordinate information of the human body key points, for example, the 2D coordinates of the human body key points or the 3D coordinates of the human body key points, or the features of the human body key points The information may further include the estimated confidence of the key points of the human body and/or the initial features corresponding to the key points of the human body. The feature information obtained by feature extraction, or the feature information of the human body key point may also include other information related to the key point, and the embodiment of the present disclosure does not limit the specific information included in the feature information of the human body key point.

In 120, in addition to the feature information of each human body key point among the multiple human body key points obtained from the at least one frame of video image, the association information between the multiple human body key points can also be used to perform behavior recognition , to improve the accuracy of behavior recognition. Optionally, the association information of the multiple human body key points may include the spatial association information of the multiple human body key points and/or the temporal association relationship of the multiple human body key points. As an example, the at least one frame of video image may specifically be multiple frames of continuous video images in the video. In this case, the association information of the multiple human body related points may include the spatial association information and temporal association relationship of multiple human body key points, Wherein, the spatial correlation information may indicate the spatial correlation between human key points. For example, the spatial correlation information may include the positional relationship of human key points in the same frame of video images, such as adjacent or directly connected, etc.; temporal correlation information may Indicates the temporal association relationship between key points of the human body. For example, the temporal association information may include the relationship between the frames to which the key points corresponding to the same body part (such as the same joint point) belong, such as adjacent frames to which they belong, etc. The present disclosure implements The example is not limited to this.

Optionally, the spatial relationship between the human body key points in the same frame of video images may be determined according to the connectivity relationship of the human body structure. For example, the key point corresponding to the elbow may be considered to be adjacent to the key point corresponding to the wrist and the key point corresponding to the shoulder, or may be predefined, and the embodiment of the present disclosure does not limit the way of determining the association relationship between the key points of the human body .

As an optional example, while the human body key point detection is performed, the information of the human body part corresponding to the human body key point, such as the human body part label, can also be output, and based on the human body part information of the human body key point, the relationship between the human body key points can be determined. connection relation. Optionally, the temporal correlation information between the human body key points corresponding to the same human body part and belonging to different frames of video images may indicate the movement track of the human body part in the at least one frame of video image over time.

In some optional examples, the spatial association information between the human body key points may include the adjacent relationship between the human body key points in the spatial position, for example, may include direct body parts indicating at least two human body key points in the same video image. Connectivity information. Optionally, the temporal association information between the human body key points may include the adjacent relationship of the frames to which the human body key points belong, for example, may include a frame indicating that at least two human body key points corresponding to the same body part belong to adjacent frames. information, but embodiments of the present disclosure are not limited thereto.

In 120, optionally, a machine learning method may be used to process the feature information and associated information of multiple human body key points to obtain a behavior recognition result of at least one frame of video image. In an optional example, a neural network can be used to process the feature information and associated information of multiple human body key points. For example, the feature information and associated information of multiple human body key points can be processed and input into a convolutional neural network for processing. processing, to obtain the behavior recognition result of at least one frame of video image, wherein the feature information and related information of multiple human body key points can be directly input into the convolutional neural network, or the feature information and related information of multiple human body key points can be directly input into the convolutional neural network. After processing, it is input to the convolutional neural network, and the specific form of the input of the convolutional neural network is not limited in this embodiment of the present disclosure.

In one or more optional embodiments, a space-time map may be established based on feature information and associated information of multiple human body key points in at least one frame of video image. Correspondingly, in 120, a space-time map may be established based on the , to obtain the behavior recognition result of at least one frame of video image. Optionally, the feature information and associated information of multiple human body key points may also be embodied in other ways, and the embodiment of the present disclosure is not limited thereto.

Specifically, a space-time map may be established based on temporal correlation information and/or spatial correlation information of multiple human body key points. The space-time graph may include multiple nodes and multiple edges, and each node in the multiple nodes may have at least one edge. In some optional examples, the human body key points can be used as nodes, and the association information between the human body key points can be embodied as edges in the spatiotemporal graph, that is, the human body key points with temporal and/or spatial correlations Edges can be used to connect them, and edges can be used to indicate the relationship between key points of the human body. As an optional example, a spatial edge and a temporal edge can be used to indicate the spatial and temporal relationship between the key points of the human body. In addition, optionally, the key points of the human body can also have self-connected edges with themselves. This The disclosed embodiments do not limit the specific implementation of the space-time graph.

As an example, assuming that multiple human body key points come from a frame of video image, the space-time graph may include the spatial edge obtained by connecting the node corresponding to each human body key point and the node corresponding to at least one other human key point, and each person The nodes corresponding to the body key points are connected to the self-connected edge obtained by themselves, wherein the body parts corresponding to the two human body key points that constitute the spatial edge are adjacent or directly connected.

As another example, assuming that multiple human body key points come from at least two frames of video images, the spatiotemporal graph can also include temporal edges on the basis of spatial edges and self-connected edges, wherein the temporal edges are obtained by connecting two adjacent frames. obtained from two human body key points corresponding to the same human body part in the video image, but the embodiment of the present disclosure is not limited thereto.

Optionally, at least one edge of a human body key point may indicate an association relationship between the human body key point and other human body key points. For example, different human body key points in the same video image correspond to different human body joints, and two points located at both ends of one bone. There is an association relationship between the key points of the human body corresponding to the joints of the human body. In an optional space-time map example, there are "temporal edges" between human key points corresponding to the same human joint in adjacent frames (such as the "left elbow" in the third and fourth frames), and adjacent human body key points in the same frame There is a "spatial edge" between the key points of the human body (such as the "left elbow" and "left wrist" in the fifth frame), where the "adjacent human key points" can be defined manually or determined according to the connection relationship of the human body structure or It is determined in other ways, for example, the human body key points corresponding to the human body joints connected or directly connected by the same bone are adjacent human body key points (such as the left elbow joint and the left wrist joint), but the embodiment of the present disclosure is not limited thereto.

In one or more optional embodiments, a first node of the plurality of nodes has a spatial edge with each of the at least one second node, wherein the first human body key point corresponding to the first node and The second human body key point corresponding to each second node in the at least one second node belongs to the same frame of video image, and the first human body key point is directly connected with the human body part corresponding to each second human body key point.

In one or more optional embodiments, there is a time edge between the first node and each of the at least one third node, wherein the first human body key point corresponds to a third node corresponding to each third node. Human body keypoints correspond to the same human body parts and belong to adjacent frames.

Optionally, when constructing the space-time graph, first, based on the connectivity relationship of the human body structure, use spatial edges to connect at least two human body key points located in the same frame of video images, and then use temporal edges to connect the same body part in the at least two human body parts. At least two human body key points in adjacent frames of a frame of video image, in this way, automatic construction can be realized without manual assignment, so that the same network structure has universality and can be applied to different nodes and node connection structures scenario, but this is not limited in the embodiments of the present disclosure.

Optionally, the number of multiple nodes included in the space-time graph may or may not be equal to the number of multiple human body key points, wherein if the number of multiple nodes is equal to the number of multiple human body key points, the multiple nodes and Multiple human body key points may be in one-to-one correspondence, but the embodiment of the present disclosure is not limited thereto.

Optionally, the behavior recognition result obtained in 120 may be the behavior recognition result corresponding to each frame of video image in at least one frame of video image, or the behavior recognition result corresponding to all video images in at least one frame of video image, for example, The at least one frame of video image belongs to a video segment of the video stream, the video segment corresponds to a human action, and accordingly, a behavior recognition result can be obtained based on the at least one frame of video image in the video segment. Alternatively, space-time graphs can also be constructed through other processes.

FIG. 2 is a schematic diagram of an example of a space-time graph constructed in an embodiment of the present disclosure. In the space-time graph, the key points of the human body are associated with the space edge and the time edge. Each key point of the human body has at least one edge with other key points of the human body. Correspondingly, each key point of the human body can be obtained through the space-time graph. The adjacent key points in the time dimension and/or the adjacent key points in the space dimension, but the specific implementation of the space-time graph is not limited in this embodiment of the present disclosure.

As an example, a space-time graph based on N joints and a skeleton sequence of T frames of video images can be expressed as G=(V, E), where V is a node set composed of multiple nodes, and E is a node set composed of multiple nodes. A collection of edges composed of edges.

Specifically, the node set V={v _ti |t=1,...,T,i=1,...,N} includes all key points of the skeleton sequence, and the feature vector of the node may include the coordinate vector and the estimated confidence, or further Include additional information. As an optional example, the edge set E may include two subsets, the first subset describes the connection between parts in each frame of video image, denoted as E _s ={v _t v _tj |(i,j)∈H }, where H is the set of naturally connected human joint points. The second subset contains inter-frame edges that connect the same joints in consecutive frames, denoted EF = _{ v tiv ( _{t +1)i} }. All the edges in _EF for a specific node can represent the trajectory of the human body part corresponding to the node over time.

In this way, based on other key points associated with the human body key points, such as adjacent key points, more information of the human body key points can be obtained, which is beneficial to obtain a more accurate action recognition result.

Based on the behavior recognition method provided by the above-mentioned embodiments of the present disclosure, key point detection is performed on at least one frame of video image, and multiple human body key points of at least one frame of video image are obtained; based on the features of multiple human key points of at least one frame of video image Compared with other methods, the local information and overall information of human body features can be fully utilized, and the accuracy of behavior recognition can be improved.

In one or more optional embodiments, a neural network may be used to process the space-time graph to obtain a behavior recognition result of at least one frame of video image. In an optional example, convolution processing may be performed on the space-time graph, and at this time, the space-time graph may optionally be input into the convolutional neural network, for example, the feature vector of the nodes in the space-time graph may be input In a convolutional neural network, the feature vector of a node may include feature information of a human body key point corresponding to the node and information of an edge corresponding to the node, but the embodiment of the present disclosure is not limited thereto. The convolutional neural network can process the input space-time graph to obtain the behavior recognition result of at least one frame of video image.

In some optional embodiments, convolution processing may be performed on multiple human body key points based on the association information between multiple human body key points to obtain convolution processing results of multiple human body key points. For example, a convolutional neural network can be used to process each human body key point in the space-time map separately, and the convolution processing result of each human body key point can be obtained, such as the image features corresponding to each human body key point, and based on multiple human body key points. The convolution processing result of each human body key point in the body key points is obtained, and the behavior recognition result of at least one frame of video image is obtained. For the convenience of description, the neural network based on the space-time graph can be referred to as the space-time graph neural network (Spatial-Temporal Graph Neural Networks, ST-GNCs), or the neural network can also have other names, the name of the neural network is not It should be understood as a limitation on the embodiments of the present disclosure.

Optionally, at least one associated key point of a certain human body key point (which may be referred to as a current human body key point) may be determined based on the associated information of multiple human body key points, and based on the feature information of the current human body key point and the at least one associated key point of the human body. The feature information of each associated key point in an associated key point determines the image feature of the current human body key point. Wherein, optionally, the associated key points may include adjacent key points. Taking the space-time graph as an example, assuming that the human body key point corresponding to the current node is called the fourth human body key point, and the human body key point corresponding to each edge of at least one edge of the current node is called the fifth human body key point, then At least one fifth key point may be determined as an adjacent key point of the fourth human body key point, wherein, optionally, the at least one edge may be part or all of the edges of the current node, and may include at least one temporal edge and/or or at least one spatial edge, correspondingly, at least one adjacent key point of the fourth human body key point (ie at least one fifth human body key point) may include adjacent key points in the time dimension and/or phase in the space dimension. adjacent key points, but the embodiments of the present disclosure are not limited thereto. Alternatively, the current node also has self-connected edges, and the human body key point corresponding to each edge of the current node can be determined as the adjacent key point of the fourth human body key point. At this time, at least one of the fourth human body key points The adjacent key points may include itself and at least one fifth key point. Accordingly, the fourth human body key point may be determined based on the feature information of each adjacent key point in the at least one adjacent key point of the fourth human body key point. , but the embodiment of the present disclosure is not limited thereto.

Optionally, convolution processing may be performed on each associated key point of the current human body key point and at least one associated key point of the current human body key point to obtain an initial convolution result of each human body key point, and based on the current human body key point. The initial convolution result of the key point and the initial convolution result of at least one associated key point. For example, the initial convolution result of the current human body key point and the initial convolution result of at least one associated key point are superimposed to obtain the current Convolutional processing results of human keypoints. For example, each fifth human body key point among the fourth human body key point and at least one fifth human body key point may be convolved by using the convolution parameter to obtain an initial convolution result of each human body key point. Wherein, optionally, the convolution parameters corresponding to each human body key point may be the same or different.

In some optional embodiments, at least one associated key point of the current human body key point may be classified, or the current human body key point and at least one associated key point thereof may be classified, a classification result may be obtained, and a corresponding corresponding key point may be assigned based on the classification result Convolution parameters. Optionally, the type of each adjacent node of the node may be determined, and then convolution kernels are assigned to perform convolution processing according to the type of adjacent nodes. For example, the fourth human body key point and the at least one fifth human body key point may be divided into at least one human body key point set, wherein each human body key point set includes at least one human body key point, and each human body key point set may correspond to A set of convolution parameters, and then, the convolution parameter of each human body key point may be determined based on the human body key point set to which each human body key point in the fourth human body key point and at least one fifth human body key point belongs. Wherein, optionally, different human body key point sets may correspond to different convolution parameters, such as different convolution kernels, wherein different human body key point sets may correspond to different convolution kernels; and the convolution kernels The weight value can be initialized in advance, for example, it can be obtained through training. As an example, assuming that the human body key points are divided into two human body key point sets, numbers can be assigned to the two human body key point sets respectively, and two sets of weight values are pre-initialized. The corresponding weight values are obtained from the numbers of the human body key point sets, that is, different convolution kernels are obtained; but the embodiment of the present disclosure is not limited to this.

In order to treat the human body key points of different classifications differently, on the basis of not changing the network structure, different network parameters (such as convolution kernel parameters) are assigned to each human body key point set, and the neural network The human body key points in the human body key point set can be processed to highlight local information; specifically, the specific parameters allocated can be determined by training or pre-defined. After training based on different tasks, the most suitable combination of the classification can be obtained. Network parameters. The structure of the neural network with different network parameters is unchanged. Specifically, different classification sets can also be input into different convolutional neural networks respectively, and the convolution operation is performed on the key points of the human body through the convolutional neural network. Human features corresponding to different classifications of human key points.

The following describes the manner of classifying the associated key points of the human body key points in the embodiment of the present disclosure with reference to FIGS. 3a-d. 3a is an example of a frame of input skeleton, and the input skeleton includes 18 key points. Optionally, the number of key points in the embodiment of the present disclosure may be any number, which is not limited in the embodiment of the present disclosure. .

In one or more optional embodiments, a uniform classification (Uni-labeling) manner as shown in FIG. 3b may be adopted. Specifically, all human body key points may be classified into the same key point classification, that is, the current human body key point and at least one associated key point may adopt the same convolution parameters.

In one or more optional embodiments, the distance-partitioning method shown in FIG. 3c may be used. Specifically, at least one associated key point can be classified according to the distance from the current human body key point. As an example, the distance between the current human body key point connected by the self-connected edge and itself is 0, and the distance between the human body key point connected by the space edge or the time edge and the current human key point is 1 (that is, the adjacent key point), then you can Divide the current human body key point distance into one class, and divide all other associated key points of the current human body key point into another class, for example, at least one human body key point set may include a first human body key point set and a second human body key point set. Point set, at this time, optionally, the fourth human body key point may be divided into the first human body key point set, and the at least one fifth human body key point may be divided into the second human body key point set. In one or more optional embodiments, the Spatial Configuration Partitioning method shown in FIG. 3d may be used. Specifically, the human body key points can be classified based on the distance between the human body key points and the reference point, wherein the reference point can be any predefined point, such as a center of gravity point, a center point or other types of reference points. For example, in the process of convolution, for the neighbors of the current node, those that are closer to the reference point than the current discussion node belong to one class, those that are farther belong to one class, and those that are equal belong to one class, or more or more can be set. There are fewer classifications, which are not limited in this embodiment of the present disclosure. At this time, optionally, the first distance between the fourth human body key point and the reference point may be determined based on the coordinate information of the fourth human body key point, and based on the difference between the distance between the human body key point and the reference point and the first distance The size relationship between the human body key points is determined to determine the human body key point set to which the human body key points belong. For example, three sets of key points can be set: the first set of key points, the second set of key points and the third set of key points. At this time, it can be determined that the human body key points whose distance from the reference point is less than the first distance belong to the first set of key points. a set of key points, and/or it can be determined that the human body key points whose distance from the reference point is equal to the first distance belong to the second set of key points, and/or the human body key points whose distance from the reference point is greater than the first distance can be determined The points belong to the third set of keypoints.

Compared with the convolution processing without distinguishing adjacent key points, the convolution processing of different human key points with different convolution parameters can reflect the local information of human key points, which is beneficial to improve the accuracy of behavior recognition results. .

Optionally, the embodiments of the present disclosure may also be classified in other manners, which are not limited in the embodiments of the present disclosure.

Optionally, after obtaining a plurality of human body key points, the above classification process can be performed, and label information can be added to each human body key point, wherein the label information can indicate the category to which the human body key points belong. Label information is added to each human body key point or each edge to indicate the category to which the human body key point or the human body key point corresponding to the edge belongs. At this time, the convolutional neural network may assign corresponding convolution parameters to each human body key point to perform convolution processing according to the label information, but the embodiment of the present disclosure is not limited to this.

Optionally, in the embodiment of the present disclosure, the convolution operation of the convolutional neural network can be replaced by a graph convolution operation, that is, a convolutional neural network based on a graph model, and the space-time based convolutional neural network can be realized without changing the network structure. Convolution processing of graphs. After passing through the convolutional neural network, the space-time graph can still maintain the structure of the graph model. However, through layer-by-layer convolution, each node already contains high-level semantic information extracted from the underlying coordinate information.

In one or more optional embodiments, the convolutional neural network may further include a global pooling layer and a fully connected layer after one or more convolutional layers. At this time, correspondingly, the convolution processing result of each human body key point in the multiple human body key points included in the space-time map can be globally pooled to obtain the pooling processing result, and based on the pooling processing result, A behavior recognition result of at least one frame of video image is obtained.

Since the input feature for behavior recognition is the feature of the space-time graph structure composed of human features, in order to synthesize the information of all nodes, all nodes in the space-time graph are passed through the global pooling layer to obtain a one-dimensional vector, where the global pool The transformation layer is only a transformation method, and the present application does not limit the transformation method of the feature dimension.

Specifically, the pooling processing result includes a one-dimensional feature vector; based on the pooling processing result, the behavior recognition result of each frame of video image in at least one frame of video image is obtained, which may specifically include: using a fully connected layer to perform the one-dimensional feature vector processing to obtain an identification vector, where the identification vector includes vector values corresponding to the number of action classification categories; the action classification in the video image is obtained based on each vector value in the identification vector.

Specifically, the number of vector values in the one-dimensional vector obtained above does not necessarily correspond to the behavior classification category (the behavior classification category is determined by the data set used, for example, there are 400 categories of behaviors in Kinetics, and 200 categories in ActivityNet). In order to realize the behavior classification, the one-dimensional vector can be input into the fully connected layer to obtain the recognition vector, and the classification result of the behavior recognition can be obtained based on the recognition vector. The behavior recognition can also use a more complex network structure instead of a single fully connected layer, and this application does not limit the network structure of the behavior recognition.

FIG. 4 is a schematic flowchart of a specific embodiment of the disclosed behavior recognition method. As shown in the figure, at least one frame of video image is recognized for human body key points, and pose estimation is performed based on the identified human body key points, and a space-time map of the skeleton sequence is constructed based on all human body key points. The application of spatiotemporal graph convolutional networks (st-gcn) will gradually produce high-quality feature maps. The corresponding action category classification is obtained through the standard softmax classifier.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, execute It includes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

FIG. 5 is a schematic structural diagram of an embodiment of the behavior recognition apparatus of the present disclosure. The apparatus of this embodiment can be used to implement the above method embodiments of the present disclosure. As shown in Figure 5, the device of this embodiment includes:

The key point detection unit 51 is configured to perform human key point detection on at least one frame of video image, and obtain multiple human key points of at least one frame of video image.

Specifically, the at least one frame of video image may be derived from a video collected by the device itself, a video input by a user, or a video obtained from other devices. The embodiment of the present disclosure does not limit the manner of obtaining the at least one frame of video image. The at least one frame of video images may specifically be one or more frames of video images, and the embodiment of the present disclosure does not limit the number of the at least one frame of video images. Optionally, the at least one frame of video image may be a continuous video image, for example, the at least one frame of video image may belong to a video segment corresponding to an action or behavior, and a behavior recognition result can be obtained based on this video segment.

The behavior recognition unit 52 is configured to obtain the behavior recognition result of at least one frame of video image based on the feature information of multiple human body key points and the associated information of multiple human body key points in at least one frame of video image.

In one or more optional embodiments, the feature information of the human body key points may include coordinate information of the human body key points, for example, the 2D coordinates of the human body key points or the 3D coordinates of the human body key points, or the features of the human body key points The information may further include the estimated confidence of the key points of the human body and/or the initial features corresponding to the key points of the human body. The feature information obtained by feature extraction, or the feature information of the human body key point may also include other information related to the key point, and the embodiment of the present disclosure does not limit the specific information included in the feature information of the human body key point.

In one or more optional embodiments, the association information of multiple human body key points includes any one or more of the following: spatial association information between at least two human body key points in the same frame of video image, and temporal correlation information between at least two human body key points corresponding to the same human body part and belonging to adjacent frames of video images in at least one frame of video image.

Based on the behavior recognition device provided by the above embodiments of the present disclosure, key point detection is performed on at least one frame of video image to obtain multiple human body key points of at least one frame of video image; based on multiple human key points of at least one frame of video image The feature information and the associated information of multiple human body key points are obtained, and the behavior recognition result of at least one frame of video image is obtained. The correlation information between the feature information and the key points of the human body can make full use of the local information and the overall information, and improve the accuracy of behavior recognition.

In one or more optional embodiments, the at least one frame of video image is specifically multiple frames of continuous video images in the video; and/or

The spatial correlation information between at least two human body key points in the same frame of video image is determined according to the connectivity relationship of human body structures.

In some optional examples, the spatial association information between the human body key points may include the adjacent relationship between the human body key points in the spatial position, for example, may include direct body parts indicating at least two human body key points in the same video image. Connectivity information. Optionally, the temporal association information between the human body key points may include the adjacent relationship of the frames to which the human body key points belong, for example, may include a frame indicating that at least two human body key points corresponding to the same body part belong to adjacent frames. information, but embodiments of the present disclosure are not limited thereto.

In one or more optional embodiments, the apparatus of the present disclosure further includes:

A graph establishment unit, configured to establish a space-time graph based on a plurality of human body key points in at least one frame of video image;

Correspondingly, the behavior recognition unit is specifically configured to obtain the behavior recognition result of at least one frame of video image based on the space-time graph.

The space-time map includes feature information of multiple human body key points and associated information of multiple human body key points in at least one frame of video image.

Specifically, a space-time map may be established based on temporal correlation information and/or spatial correlation information of multiple human body key points. The space-time graph may include multiple nodes and multiple edges, and each node in the multiple nodes may have at least one edge. In some optional examples, the human body key points can be used as nodes, and the association information between the human body key points can be embodied as edges in the spatiotemporal graph, that is, the human body key points with temporal and/or spatial correlations Edges can be used to connect them, and edges can be used to indicate the relationship between key points of the human body. As an optional example, a spatial edge and a temporal edge can be used to indicate the spatial and temporal relationship between the key points of the human body. In addition, optionally, the key points of the human body can also have self-connected edges with themselves. This The disclosed embodiments do not limit the specific implementation of the space-time graph.

In one or more optional embodiments, a first node of the plurality of nodes has a spatial edge with each of the at least one second node, wherein the first human body key point corresponding to the first node and The second human body key point corresponding to each second node in the at least one second node belongs to the same frame of video image, and the first human body key point is directly connected with the human body part corresponding to each second human body key point.

In one or more optional embodiments, there is a time edge between the first node and each of the at least one third node, wherein the first human body key point corresponds to a third node corresponding to each third node. Human body keypoints correspond to the same human body parts and belong to adjacent frames.

Optionally, when constructing the space-time graph, first, based on the connectivity relationship of the human body structure, use spatial edges to connect at least two human body key points located in the same frame of video images, and then use temporal edges to connect the same body part in the at least two human body parts. At least two human body key points in adjacent frames of a frame of video image, but this is not limited in this embodiment of the present disclosure.

In one or more optional embodiments, the behavior recognition unit 52 is specifically configured to input the space-time graph into the convolutional neural network to obtain the behavior recognition result of at least one frame of video image.

Optionally, the behavior recognition result obtained by the behavior recognition unit 52 may be the behavior recognition result corresponding to each frame of video image in at least one frame of video image, or the behavior recognition result corresponding to all video images in at least one frame of video image, For example, the at least one frame of video image belongs to a video segment of the video stream, the video segment corresponds to a human action, and accordingly, a behavior recognition result can be obtained based on the at least one frame of video image in the video segment. Alternatively, space-time graphs can also be constructed through other processes.

In one or more optional embodiments, the behavior recognition unit 52 includes:

The convolution processing module is used to perform convolution processing on multiple human body key points based on the association information between multiple human body key points to obtain the convolution processing results of multiple human body key points;

The convolution recognition module is used to obtain the behavior recognition result of at least one frame of video image based on the convolution processing results of multiple human key points.

Optionally, a convolution processing module, including:

an association determination module, configured to determine at least one fifth human body key point that has an associated relationship with the fourth human body key point in the multiple human body key points based on the association information of the multiple human body key points;

The feature processing module is configured to obtain the convolution processing result of the fourth human body key point based on the feature information of each human body key point in the fourth human body key point and the at least one fifth human body key point.

In some optional embodiments, the feature processing module is specifically configured to use the convolution parameters corresponding to the human body key point set to which each human body key point in the fourth human body key point and at least one fifth human body key point belongs to, for each human body key point. Perform convolution processing on the key points of the human body to obtain the initial convolution result of each key point of the human body;

The convolution processing result of the fourth human body key point is obtained based on the initial convolution result of each human body key point in the fourth human body key point and at least one fifth human body key point.

Optionally, convolution processing may be performed on each associated key point of the current human body key point and at least one associated key point of the current human body key point to obtain an initial convolution result of each human body key point, and based on the current human body key point. The initial convolution result of the key point and the initial convolution result of at least one associated key point. For example, the initial convolution result of the current human body key point and the initial convolution result of at least one associated key point are superimposed to obtain the current Convolutional processing results of human keypoints. For example, each fifth human body key point among the fourth human body key point and at least one fifth human body key point may be convolved by using the convolution parameter to obtain an initial convolution result of each human body key point. Wherein, optionally, the convolution parameters corresponding to each human body key point may be the same or different.

In some optional embodiments, the behavior identification unit 52 further includes:

a classification module, configured to divide the fourth human body key point and at least one fifth human body key point into at least one human body key point set; wherein each human body key point set includes at least one human body key point;

The parameter determination module is configured to determine the convolution parameter of each human body key point based on the human body key point set to which each human body key point belongs in the fourth human body key point and the at least one fifth human body key point.

Among them, the human body key points belonging to different human body key point sets correspond to different convolution parameters.

In order to treat the human body key points of different classifications differently, on the basis of not changing the network structure, different network parameters (such as convolution kernel parameters) are assigned to each human body key point set, and the neural network The human body key points in the human body key point set can be processed to highlight local information; specifically, the specific parameters allocated can be determined by training or pre-defined. After training based on different tasks, the most suitable combination of the classification can be obtained. Network parameters. The structure of the neural network with different network parameters is unchanged. Specifically, different classification sets can also be input into different convolutional neural networks respectively, and the convolution operation is performed on the key points of the human body through the convolutional neural network. Human features corresponding to different classifications of human key points.

In one or more optional embodiments, the at least one human body key point set includes a first human body key point set and a second human body key point set;

The classification module is specifically configured to divide the fourth human body key point into the first human body key point set, and divide the at least one fifth human body key point into the second human body key point set.

In one or more optional embodiments, the classification module is specifically configured to classify the fourth human body based on the distance between each of the fourth human body key points and the at least one fifth human body key point and the reference point. The key points and the at least one fifth human body key point are divided into at least one human body key point set.

Optionally, a classification module, including:

a first distance module, configured to determine the first distance between the fourth human body key point and the reference point based on the feature information of the fourth human body key point;

a first relationship module, configured to determine each human body key point based on the magnitude relationship between the distance between each human body key point and the reference point in the fourth human body key point and the at least one fifth human body key point and the first distance The set of keypoints to which it belongs.

Specifically, a first relationship module, specifically configured to determine that a human body key point whose distance from the reference point is less than the first distance belongs to the first set of key points; and/or

It is determined that the human body keypoint whose distance from the reference point is equal to the first distance belongs to the second set of keypoints; and/or

It is determined that the human body key points whose distance from the reference point is greater than the first distance belong to the third set of key points.

Compared with the convolution processing without distinguishing adjacent key points, the convolution processing of different human key points with different convolution parameters can reflect the local information of human key points, which is beneficial to improve the accuracy of behavior recognition results. .

Optionally, the embodiments of the present disclosure may also be classified in other manners, which are not limited in the embodiments of the present disclosure.

Optionally, after obtaining a plurality of human body key points, the above classification process can be performed, and label information can be added to each human body key point, wherein the label information can indicate the category to which the human body key points belong. Label information is added to each human body key point or each edge to indicate the category to which the human body key point or the human body key point corresponding to the edge belongs. At this time, the convolutional neural network may assign corresponding convolution parameters to each human body key point to perform convolution processing according to the label information, but the embodiment of the present disclosure is not limited to this.

In one or more optional embodiments, the convolution identification module is specifically configured to perform global pooling processing on the convolution processing result of each human body key point in the multiple human body key points included in the space-time map, Get the pooling result;

Based on the pooling processing result, the behavior recognition result of at least one frame of video image is obtained. At this time, correspondingly, the convolution processing result of each human body key point in the multiple human body key points included in the space-time map can be globally pooled to obtain the pooling processing result, and based on the pooling processing result, A behavior recognition result of at least one frame of video image is obtained.

Since the input feature for behavior recognition is the feature of the space-time graph structure composed of human features, in order to synthesize the information of all nodes, all nodes in the space-time graph are passed through the global pooling layer to obtain a one-dimensional vector, where the global pool The transformation layer is only a transformation method, and the present application does not limit the transformation method of the feature dimension.

Specifically, the pooling processing result includes a one-dimensional feature vector; based on the pooling processing result, a behavior recognition result of each frame of video image in at least one frame of video image is obtained.

According to an aspect of an embodiment of the present disclosure, an electronic device is provided, including a processor, where the processor includes the behavior recognition apparatus of any of the foregoing embodiments of the present disclosure.

According to an aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a memory for storing executable instructions;

and a processor configured to communicate with the memory to execute executable instructions so as to accomplish any of the above embodiments of the behavior recognition method of the present disclosure.

According to an aspect of the embodiments of the present disclosure, a computer storage medium is provided for storing computer-readable instructions, and when the instructions are executed, any of the foregoing embodiments of the behavior identification method of the present disclosure are executed.

According to an aspect of the embodiments of the present disclosure, a computer program is provided, including computer-readable codes. When the computer-readable codes are executed on a device, a processor in the device executes any of the above-mentioned embodiments of the present disclosure. Instructions for behavior recognition methods.

In one or more optional implementation manners, the embodiments of the present disclosure further provide a computer program product for storing computer-readable instructions, which, when executed, cause the computer to perform the behavior recognition in any of the above possible implementation manners method.

The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional example, the computer program product is embodied as a computer storage medium, and in another optional example, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

In one or more optional implementation manners, the embodiments of the present disclosure further provide another behavior recognition method and its corresponding apparatus, electronic device, computer storage medium, computer program, and computer program product, wherein the method includes: Performing human body key point detection on at least one frame of video image to obtain multiple human body key points of at least one frame of video image; A behavior recognition result of at least one frame of video image is obtained.

In some embodiments, the behavior identification instruction may be specifically an invocation instruction, and the first device may instruct the second device to perform image processing by means of invocation. Accordingly, in response to receiving the invocation instruction, the second device may perform the above behavior Steps and/or processes in any embodiment of the method are identified.

Embodiments of the present disclosure also provide an electronic device, which may be, for example, a mobile terminal, a personal computer (PC), a tablet computer, a server, and the like. 6, which shows a schematic structural diagram of an electronic device 600 suitable for implementing the embodiments of the present application: As shown in FIG. 6, the electronic device 600 includes one or more processors, communication parts, etc. or multiple processors such as: one or more central processing units (CPUs) 601, and/or one or more graphics processing units (GPUs) 613, etc., the processors may Executable instructions or executable instructions loaded from storage section 608 into random access memory (RAM) 603 perform various suitable actions and processes. The communication part 612 may include, but is not limited to, a network card, and the network card may include, but is not limited to, an IB (Infiniband) network card.

The processor can communicate with the read-only memory 602 and/or the random access memory 630 to execute executable instructions, connect with the communication part 612 through the bus 604, and communicate with other target devices through the communication part 612, thereby completing the provision of the embodiments of the present application. The operation corresponding to any one of the methods, for example, performing human body key point detection on at least one frame of video image to obtain multiple human body key points of at least one frame of video image; based on the characteristics of multiple human key points of at least one frame of video image information and the associated information of multiple human body key points to obtain the behavior recognition result of at least one frame of video image.

In addition, in the RAM 603, various programs and data necessary for the operation of the device can also be stored. The CPU 601 , the ROM 602 and the RAM 603 are connected to each other through a bus 604 . In the case of RAM 603, ROM 602 is an optional module. The RAM 603 stores executable instructions, or writes executable instructions into the ROM 602 at runtime, and the executable instructions enable the processor 601 to perform operations corresponding to the above communication methods. An input/output (I/O) interface 605 is also connected to bus 604 . The communication unit 612 may be integrated, or may be configured to have multiple sub-modules (eg, multiple IB network cards) and be connected to the bus.

The following components are connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc. ; and a communication section 609 including a network interface card such as a LAN card, a modem, and the like. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I/O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

It should be noted that the architecture shown in FIG. 6 is only an optional implementation. In the specific practice process, the number and type of components in the above-mentioned FIG. 6 can be selected, deleted, added or replaced according to actual needs; For the setting of different functional components, separate settings or integrated settings can also be adopted. For example, the GPU and the CPU can be set separately or the GPU can be integrated on the CPU, and the communication department can be set separately or integrated on the CPU or GPU. and many more. These alternative embodiments all fall within the scope of the present disclosure.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the methods illustrated in the flowcharts, the program code may include corresponding Execute the instructions corresponding to the method steps provided by the embodiments of the present application, for example, perform human body key point detection on at least one frame of video image, and obtain multiple human body key points of at least one frame of video image; The feature information of the key points and the association information of multiple human body key points are used to obtain the behavior recognition result of at least one frame of video image. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 609 and/or installed from the removable medium 611 . When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the method of the present application are performed.

It should be understood that terms such as "first" and "second" in the embodiments of the present disclosure are only for distinction, and should not be construed as limitations on the embodiments of the present disclosure.

It should also be understood that in the present disclosure, "a plurality" may refer to two or more, and "at least one" may refer to one, two, or more.

It should also be understood that any reference to any component, data or structure in the present disclosure may generally be construed as one or more in the absence of an explicit definition or a contrary indication in the context.

It should also be understood that the description of the various embodiments in the present disclosure emphasizes the differences between the various embodiments, and the same or similar points can be referred to each other, and for the sake of brevity, they will not be repeated.

The methods and apparatuses of the present disclosure may be implemented in many ways. For example, the methods, apparatuses, and devices of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Furthermore, in some embodiments, the present disclosure can also be implemented as programs recorded in a recording medium, the programs including machine-readable instructions for implementing methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

The description of the present disclosure has been presented for purposes of example and description, and is not intended to be exhaustive or to limit the disclosure to the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to better explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use.

Claims (29)

1. a method for behavior recognition, is characterized in that, comprises: Performing human body key point detection on at least one frame of video image to obtain multiple human key points of the at least one frame of video image; A spatiotemporal map is established based on multiple human body key points in the at least one frame of video image, wherein the spatiotemporal map includes feature information of multiple human key points in the at least one frame of video image and the multiple human key points in the at least one frame of video image. Relevant information of key points of the individual; determining at least one fifth human body key point that has an associated relationship with the fourth human body key point in the plurality of human body key points based on the association information between the plurality of human body key points included in the space-time graph; Using the convolution parameters corresponding to the set of human body key points to which each of the fourth human body key points and the at least one fifth human body key point belongs, the convolution process is performed on each of the human body key points, obtain the initial convolution result of each human body key point; Obtain the convolution processing result of the fourth human body key point based on the initial convolution result of each human body key point in the fourth human body key point and the at least one fifth human body key point; Based on the convolution processing result of the multiple human body key points, the behavior recognition result of the at least one frame of video image is obtained. 2. The method according to claim 1, wherein the feature information of the human body key points comprises coordinate information of the human body key points; or, The feature information of the human body key points includes coordinate information of the human body key points, estimated confidence levels of the human body key points and/or initial features corresponding to the human body key points. 3. The method according to claim 1, wherein the associated information of the multiple human body key points comprises any one or more of the following: between at least two human body key points in the same frame of video image and the temporal correlation information between at least two human body key points corresponding to the same human body part and belonging to adjacent frames of video images in the at least one frame of video image. 4. The method according to claim 3, wherein the at least one frame of video image is specifically multiple frames of continuous video images in the video; and/or The spatial correlation information between at least two human body key points in the same frame of video image is determined according to the connectivity relationship of human body structures. 5. The method of claim 3, wherein The spatial association information between the at least two human body key points includes the spatially adjacent relationship of the at least two key points, and/or The temporal association information between the at least two key points includes: the adjacent relationship of the frames to which the at least two key points belong. 6 . The method according to claim 1 , wherein the space-time graph includes a plurality of nodes corresponding to the plurality of human body key points, and each node in the plurality of nodes includes a corresponding human body key point. 7 . characteristic information; Each of the plurality of nodes has at least one edge, and the plurality of edges of the plurality of nodes indicate the association relationship of the plurality of human body key points. 7. The method of claim 6, wherein A first node in the plurality of nodes has a spatial edge with each second node in the at least one second node, wherein the first human body key point corresponding to the first node and the at least one second node have a space edge. The second human body key point corresponding to each second node belongs to the same frame, and the first human body key point is directly connected to the human body part corresponding to each second human body key point, and/or There is a time edge between the first node and each of the at least one third node, wherein the first human body key point corresponds to the same third human body key point corresponding to each of the third nodes body parts and belong to adjacent frames. 8. The method according to any one of claims 1-7, wherein before the convolution processing is performed on each of the human body key points, the method further comprises: dividing the fourth human body key point and the at least one fifth human body key point into at least one human body key point set, wherein each human body key point set includes at least one human body key point; The convolution parameter of each human body key point is determined based on the set of human body key points to which each human body key point in the fourth human body key point and the at least one fifth human body key point belongs, wherein different human body key points belong to different human body key points. The human keypoints of the point set correspond to different convolution parameters. 9. The method according to claim 8, wherein the at least one human body key point set comprises a first human body key point set and a second human body key point set; The described fourth human body key point and the at least one fifth human body key point are divided into at least one human body key point set, including: The fourth human body key point is divided into the first human body key point set, and the at least one fifth human body key point is divided into the second human body key point set. 10. The method according to claim 9, wherein the dividing the fourth human body key point and the at least one fifth human body key point into at least one human body key point set, comprising: Based on the distance between each of the fourth human body key points and the at least one fifth human body key point and the reference point, the fourth human body key point and the at least one fifth human body key point Divide into at least one human body keypoint set. 11. The method according to claim 10, wherein, based on the distance between each human body key point and a reference point in the fourth human body key point and the at least one fifth human body key point, the The fourth human body key point and the at least one fifth human body key point are divided into at least one human body key point set, including: determining a first distance between the fourth human body key point and the reference point based on the feature information of the fourth human body key point; determining the each person based on the magnitude relationship between the distance between each of the fourth human body key points and the at least one fifth human body key point and the reference point and the first distance The set of keypoints to which the body keypoint belongs. 12 . The method according to claim 11 , wherein the distance between each human key point and the reference point is based on the fourth human key point and the at least one fifth human key point. 13 . The size relationship with the first distance, to determine the key point set to which each human body key point belongs, including: It is determined that the human body key points whose distance from the reference point is less than the first distance belong to the first set of key points; and/or It is determined that the human body key point whose distance from the reference point is equal to the first distance belongs to the second set of key points; and/or It is determined that the human body key points whose distance from the reference point is greater than the first distance belong to the third set of key points. 13. The method according to any one of claims 1-7, wherein the obtaining a behavior recognition result of the at least one frame of video image based on the convolution processing results of the plurality of human body key points, comprising: Perform global pooling processing on the convolution processing result of each human body key point in the plurality of human body key points included in the space-time map to obtain a pooling processing result; Based on the pooling processing result, a behavior recognition result of the at least one frame of video image is obtained. 14. A behavior recognition device, comprising: a key point detection unit, configured to perform human key point detection on at least one frame of video image, and obtain a plurality of human key points of the at least one frame of video image; A graph establishing unit, configured to establish a space-time graph based on a plurality of human body key points in the at least one frame of video image, wherein the space-time graph includes a plurality of human body key points in the at least one frame of video image. feature information and associated information of the multiple human body key points Behavioural recognition unit, including: A convolution processing module, including: an association determination module and a feature processing module; The association determination module is configured to determine, based on the association information between the plurality of human body key points included in the space-time graph, at least a A fifth key point of the human body; The feature processing module is configured to use the convolution parameter corresponding to the human body key point set to which each human body key point in the fourth human body key point and the at least one fifth human body key point belongs to Perform convolution processing on the body key points to obtain the initial convolution result of each human body key point; based on the initial convolution of each human body key point in the fourth human body key point and the at least one fifth human body key point As a result, the convolution processing result of the fourth human body key point is obtained; The convolution identification module is configured to obtain the behavior identification result of the at least one frame of video image based on the convolution processing results of the multiple human body key points. 15. The apparatus according to claim 14, wherein the feature information of the human body key points comprises coordinate information of the human body key points; or, The feature information of the human body key points includes coordinate information of the human body key points, estimated confidence levels of the human body key points and/or initial features corresponding to the human body key points. 16. The apparatus according to claim 14, wherein the association information of the multiple human body key points comprises any one or more of the following: between at least two human body key points in the same frame of video image and the temporal correlation information between at least two human body key points corresponding to the same human body part and belonging to adjacent frames of video images in the at least one frame of video image. 17. The apparatus according to claim 16, wherein the at least one frame of video image is specifically multiple frames of continuous video images in the video; and/or The spatial correlation information between at least two human body key points in the same frame of video image is determined according to the connectivity relationship of human body structures. 18. The apparatus of claim 16, wherein The spatial association information between the at least two human body key points includes the spatially adjacent relationship of the at least two key points, and/or The temporal association information between the at least two key points includes: the adjacent relationship of the frames to which the at least two key points belong. 19. The apparatus according to claim 14, wherein the space-time graph comprises a plurality of nodes corresponding to the plurality of human body key points, and each node in the plurality of nodes comprises a corresponding human body key point characteristic information; Each of the plurality of nodes has at least one edge, and the plurality of edges of the plurality of nodes indicate the association relationship of the plurality of human body key points. 20. The apparatus of claim 19, wherein A first node in the plurality of nodes has a spatial edge with each second node in the at least one second node, wherein the first human body key point corresponding to the first node and the at least one second node have a space edge. The second human body key point corresponding to each second node belongs to the same frame, and the first human body key point is directly connected to the human body part corresponding to each second human body key point, and/or There is a time edge between the first node and each of the at least one third node, wherein the first human body key point corresponds to the same third human body key point corresponding to each of the third nodes body parts and belong to adjacent frames. 21. The device according to any one of claims 14-20, wherein the behavior recognition unit further comprises: a classification module, configured to divide the fourth human body key point and the at least one fifth human body key point into at least one human body key point set, wherein each human body key point set includes at least one human body key point; A parameter determination module, configured to determine the convolution parameter of each human body key point based on the human body key point set to which each human body key point in the fourth human body key point and the at least one fifth human body key point belongs, Among them, the human body key points belonging to different human body key point sets correspond to different convolution parameters. 22. The apparatus according to claim 21, wherein the at least one human body key point set comprises a first human body key point set and a second human body key point set; The classification module is specifically configured to divide the fourth human body key point into the first human body key point set, and divide the at least one fifth human body key point into the second human body key point set. 23. The apparatus according to claim 22, wherein the classification module is specifically configured to be based on each human body key point and reference point in the fourth human body key point and the at least one fifth human body key point The distance between the fourth human body key point and the at least one fifth human body key point is divided into at least one human body key point set. 24. The apparatus according to claim 23, wherein the classification module comprises: a first distance module, configured to determine a first distance between the fourth human body key point and the reference point based on the feature information of the fourth human body key point; a first relationship module, configured to be based on the size of the distance between each of the fourth human body key points and the at least one fifth human body key point and the reference point between each human key point and the first distance relationship, to determine the set of key points to which each of the human body key points belongs. 25. The apparatus according to claim 24, wherein the first relationship module is specifically configured to determine that a human body key point whose distance from the reference point is smaller than the first distance belongs to the first key point collection; and/or It is determined that the human body key point whose distance from the reference point is equal to the first distance belongs to the second set of key points; and/or It is determined that the human body key points whose distance from the reference point is greater than the first distance belong to the third set of key points. 26. The apparatus according to any one of claims 14 to 20, wherein the convolution identification module is specifically configured to identify each human body key point in a plurality of human body key points included in the space-time map The result of convolution processing is globally pooled to obtain the result of pooling processing; Based on the pooling processing result, a behavior recognition result of the at least one frame of video image is obtained. 27. An electronic device, characterized by comprising a processor, wherein the processor comprises the behavior recognition device according to any one of claims 14 to 26. 28. An electronic device, characterized in that it comprises: memory for storing executable instructions; and a processor for executing the executable instructions to complete the behavior recognition method according to any one of claims 1 to 13. 29. A computer storage medium for storing computer-readable instructions, wherein the behavior recognition method according to any one of claims 1 to 13 is executed when the instructions are executed.

Images