CN116403288B - Motion gesture recognition method and device and electronic equipment - Google Patents

Abstract

The embodiment of the invention provides a motion gesture recognition method, a motion gesture recognition device and electronic equipment. The method comprises the following steps: and acquiring images of a plurality of persons at a plurality of view angles at the same moment, identifying the boundary box of each person in the images, matching each image according to the boundary box, checking according to the matching result, and identifying and determining the gesture information of each person after the boundary box passes the checking. According to the process, the multiple images of multiple persons under multiple visual angles are obtained, so that the boundary box of each person in the images is identified, the gesture information of each person is determined through the boundary box, and the accuracy of gesture identification under the multiple visual angles of multiple persons is improved.

Description

Motion posture recognition method, recognition device and electronic equipment

Technical Field

The present invention relates to the technical field of image recognition, and in particular to a motion posture recognition method, a recognition device and an electronic device.

Background technique

In the field of single-view action recognition, the common feature of the existing commonly used algorithms is that they perform action recognition based on input such as pictures or videos from a single viewpoint, and use a neural network model to process image or video frame information to obtain the key point positions to achieve the action recognition task. This method performs very well when the complete human body is not blocked, but the recognition effect is poor when the human body information is missing in the image due to human body occlusion.

In the field of multi-view action recognition, it is mainly about single-person multi-view action recognition. The extraction and processing of single-person image information is much simpler than that of multiple people. The multi-view information of a single person only needs to do feature extraction and fusion, but the recognition effect for multiple people is not good.

Therefore, the action recognition effect of single-view action recognition or multi-view action recognition is poor in special situations, such as when the action is blocked or there are multiple people.

Summary of the invention

Based on this, the first aspect of the present invention provides a method for identifying a motion posture, which improves the accuracy of posture recognition. The method comprises:

Acquire images from multiple perspectives of multiple people at the same time;

Identify the bounding box of each person in each image based on a preset object detection algorithm;

Select images from any two perspectives, match the bounding boxes of the same person in the two images, and get the matching score;

Construct a matching score matrix of the same person under multiple viewing angles according to the matching scores, wherein the rows and columns of the matching score matrix are bounding boxes in the image;

Perform matching verification on multiple bounding boxes of each person at multiple viewpoints according to the matching score matrix;

When multiple bounding boxes of each person are verified, the posture information of each person is determined based on each bounding box.

In the embodiment of the present invention, images from any two perspectives are selected, and the bounding boxes of the same person in the two images are matched to obtain a matching score, including:

Using the first preset neural network, convolve the bounding boxes of each person in the two images to obtain the appearance features of each person;

Calculate the appearance matching score of each person based on their appearance characteristics;

Identify key points in each bounding box using a second preset neural network;

Calculate the action matching score based on the relative position of the corresponding perspectives of any two images and the coordinates of each key point;

Calculate the product of the action matching score and the appearance matching score to get the matching score.

In the embodiment of the present invention, the appearance matching score of each person is calculated according to the appearance features of each person, including:

Calculate the similarity score of two images based on appearance features;

The similarity scores are normalized to obtain the appearance matching score.

In the embodiment of the present invention, the action matching score is calculated according to the relative position of the viewing angles corresponding to any two images and the coordinates of each key point, including:

Construct a corresponding transformation matrix according to the relative positions of the viewing angles corresponding to any two images;

Using the transformation matrix, the key point coordinates of the same person in the first image of any two images are mapped to the second image to obtain the mapped coordinates;

Calculate the distance between the coordinates of each key point in the second image and the mapped coordinates;

The sum of multiple distances is normalized to obtain the action matching score.

In an embodiment of the present invention, matching verification is performed on multiple bounding boxes of each person under multiple viewing angles according to a matching score matrix, including:

For any two perspective images, the first loop step is performed until all values in the matching score matrix are the first preset values, and the matching results of the any two perspective images are obtained, wherein the first loop step includes:

Searching for a maximum matching score in the matching score matrix, obtaining a matching result and determining a matching bounding box in the image according to the row and column where the maximum matching score is located, and setting all values of the row and column where the maximum matching score is located to a first preset value;

Select the image with the largest number of bounding boxes as the target image;

According to the matching results and the target image, the bounding boxes of the same person in all images except the target image are grouped into a bounding box set;

Check cycle consistency against a set of bounding boxes of the same person.

In an embodiment of the present invention, checking cycle consistency based on a set of bounding boxes of the same person includes:

According to the bounding boxes of the same person in the target image, the bounding boxes in the bounding box set are divided into a plurality of groups according to a second preset value;

In the case where the bounding boxes in each group match each other, it is determined that the bounding boxes of the same person meet cycle consistency.

In the embodiment of the present invention, after multiple bounding boxes of each person have passed verification, the posture information of each person is determined according to each bounding box, including:

When the multiple bounding boxes of each person pass the verification, the image features of each image are extracted using the third preset neural network model;

The bounding boxes of each person in the multiple images are input into the fourth preset neural network model to obtain the posture information of each person.

A second aspect of the present invention provides a motion gesture recognition device, comprising:

An image acquisition module, used to acquire images from multiple perspectives of multiple people at the same time;

A recognition module, which is used to identify the bounding box of each person in each image based on a preset object detection algorithm;

A matching score determination module is used to select images from any two perspectives, match the bounding boxes of the same person in the two images, and obtain a matching score;

A matching score matrix determination module is used to construct a matching score matrix of the same person under multiple viewing angles according to the matching scores, wherein the rows and columns of the matching score matrix are bounding boxes in the image;

A matching verification module is used to perform matching verification on multiple bounding boxes of each person under multiple viewing angles according to a matching score matrix;

The posture information determination module is used to determine the posture information of each person according to each bounding box after multiple bounding boxes of each person have passed the verification.

A third aspect of the present invention provides an electronic device, the electronic device includes a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement any of the above-mentioned motion posture recognition methods.

A fourth aspect of the present invention provides a machine-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, a method for recognizing a motion posture as described above is implemented.

Through the above technical solution, images of multiple perspectives including multiple people are obtained at the same time; the bounding box of each person in each image is identified based on a preset target detection algorithm; images of any two perspectives are selected, and the bounding boxes of the same person in the two images are matched to obtain a matching score; a matching score matrix of the same person under multiple perspectives is constructed according to the matching score, wherein the rows and columns of the matching score matrix are the bounding boxes in the image; multiple bounding boxes of each person under multiple perspectives are matched and verified according to the matching score matrix; when multiple bounding boxes of each person pass the verification, the posture information of each person is determined according to each bounding box. This process improves the accuracy of posture recognition under multiple perspectives of multiple people by obtaining multiple images of multiple people under multiple perspectives, identifying the bounding boxes of each person in the image according to the multiple images, and determining the posture information of each person through the bounding box.

Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

FIG1 is a schematic flow chart of a method for identifying a motion posture provided by an embodiment of the present invention;

FIG2 is a schematic diagram of the structure of a motion posture recognition device provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

Detailed ways

The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

Based on this, the present invention provides a method for identifying a motion posture. FIG1 is a flow chart of a method for identifying a motion posture provided by an embodiment of the present invention. As shown in FIG1 , the method includes:

Step S101: Acquire images from multiple perspectives of multiple persons at the same time.

In actual applications, multiple cameras are respectively set at eight positions: front, rear, left, right, left front, right front, left rear and right rear of a crowd of multiple people.

In practical applications, multiple cameras at various locations are used to capture images of multiple people from multiple perspectives at the same time. If a video of multiple people is captured by a camera, images of each frame or frames at a certain interval in the video are extracted.

Step S102: Identify the bounding box of each person in each image based on a preset object detection algorithm.

In practical applications, the preset target detection algorithm can be an SSD (Single Shot MultiBox Detector) target detection algorithm. Images from multiple perspectives are input into the SSD target detection algorithm to obtain bounding boxes of each person in the multiple images, where the bounding box is a 2D bounding box, which is a part of the original input image and is a rectangular box that frames all body parts of a person.

Step S103: Select images from any two perspectives, match the bounding boxes of the same person in the two images, and obtain a matching score.

In practical applications, two images corresponding to two perspectives are arbitrarily selected, and the bounding boxes belonging to the same person in the two images are matched to obtain the matching scores of the bounding boxes of the same person.

Step S104: construct a matching score matrix of the same person under multiple viewing angles according to the matching scores, wherein the rows and columns of the matching score matrix are bounding boxes in the image.

In practical applications, the matching score matrix is composed of matching scores, the rows of the matching score matrix are all bounding boxes in one of the multiple views, the columns of the matching matrix are all bounding boxes in another of the multiple views, and the matrix values are the matching scores between different bounding boxes corresponding to the rows and columns.

Step S105: performing matching verification on multiple bounding boxes of each person under multiple viewing angles according to the matching score matrix.

In practical applications, the matching score matrix is used to perform matching verification on the matching results of each person's bounding box under multiple viewing angles.

In practical applications, the bounding boxes belonging to the same person in different perspectives should have the same matching results in another perspective. Matching verification of the matching results can ensure the accuracy of the matching results. If the matching verification fails, the remaining bounding boxes are re-matched based on the relative positions of the perspectives until all bounding boxes of the same person pass the matching verification.

Step S106: After multiple bounding boxes of each person have passed verification, the posture information of each person is determined according to each bounding box.

In practical applications, after multiple bounding boxes of each person are verified, the posture information of each person is determined according to the bounding box of each person, and the posture information is 3D posture information.

Through the above embodiment, images of multiple perspectives including multiple people are obtained at the same time; the bounding box of each person in each image is identified based on a preset target detection algorithm; images of any two perspectives are selected, and the bounding boxes of the same person in the two images are matched to obtain a matching score; a matching score matrix of the same person at multiple perspectives is constructed based on the matching score, wherein the rows and columns of the matching score matrix are the bounding boxes in the image; multiple bounding boxes of each person at multiple perspectives are matched and verified based on the matching score matrix; when multiple bounding boxes of each person pass the verification, the posture information of each person is determined based on each bounding box. This process improves the accuracy of posture recognition under multiple perspectives of multiple people by obtaining multiple images of multiple people at multiple perspectives, identifying the bounding boxes of each person in the image based on the multiple images, and determining the posture information of each person through the bounding boxes.

In one embodiment, step S103 includes:

Using the first preset neural network, convolve the bounding boxes of each person in the two images to obtain the appearance features of each person;

Calculate each person's appearance matching score based on each person's appearance features;

Identify key points in each bounding box using a second preset neural network;

The action matching score is calculated based on the relative position of the corresponding perspectives of any two images and the coordinates of each key point;

Calculate the product of the action matching score and the appearance matching score to get the matching score.

In practical applications, the matching score is calculated from the appearance matching score and the action matching score. The first preset neural network may be a convolutional neural network (CNN), which convolves the bounding boxes of each person in the images corresponding to any two perspectives to obtain the appearance features of each person. The appearance features of each person are calculated to obtain the appearance matching score of each person.

In actual applications, the second preset neural network can also be a convolutional neural network, which uses the convolutional neural network to identify the key points of the person in each bounding box. The key points include but are not limited to the person's nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Other important points on the human body.

In practical applications, the relative positions of images corresponding to any two perspectives are calculated based on the positions of multiple perspectives. The key point coordinates of the same person in the two images are obtained. The action matching score of the corresponding person is calculated based on the relative positions of the images and the key point coordinates. The matching score is the product of the appearance matching score and the action matching score.

Through the above embodiment, the first preset neural network is used to convolve the bounding boxes of each person in the two images to obtain the appearance features of each person; the appearance matching score of each person is calculated based on the appearance features of each person; the key points in each bounding box are identified using the second preset neural network; the action matching score is calculated based on the relative position of the perspectives corresponding to any two images and the coordinates of each key point; the product of the action matching score and the appearance matching score is calculated to obtain the matching score. In this process, the appearance matching score is obtained by convolving the bounding boxes of the same person in the two images, the action matching score is calculated by identifying the key points in the bounding boxes, and the matching score of the bounding boxes of the same person in the two images is determined based on the action matching score and the appearance matching score, thereby ensuring the accuracy of the bounding box matching.

In one embodiment, the appearance matching score of each person is calculated based on the appearance characteristics of each person, including:

Calculate the similarity score of two images based on appearance features;

The similarity scores are normalized to obtain the appearance matching score.

In practical applications, the similarity scores of the same person in two images are calculated based on the appearance features. The obtained similarity scores are normalized to obtain the appearance matching score.

In practical applications, the Sigmoid function can be used to map the similarity score to the interval [0, 1 ^] to obtain the final appearance matching score. If the two images are exactly the same, the appearance matching score is 1; if the two images are completely different, the appearance matching score is 0.

Through the above embodiment, the similarity scores of the two images are calculated according to the appearance features; the similarity scores are normalized to obtain the appearance matching scores. This process calculates the similarity scores of the two images belonging to the same person according to the appearance features, and normalizes the similarity scores to obtain the appearance matching scores, and uses the similarity scores to determine the appearance matching scores, thereby improving the accuracy of the appearance matching score calculation.

In one embodiment, the action matching score is calculated according to the relative position of the viewing angles corresponding to any two images and the coordinates of each key point, including:

Construct a corresponding transformation matrix according to the relative positions of the viewing angles corresponding to any two images;

Using the transformation matrix, the key point coordinates of the same person in the first image of any two images are mapped to the second image to obtain the mapped coordinates;

Calculate the distance between the coordinates of each key point in the second image and the mapped coordinates;

The sum of multiple distances is normalized to obtain the action matching score.

In practical applications, the relative positions of the perspectives corresponding to the two images are calculated respectively, and the transformation matrices of the two images are constructed according to the relative positions. The key point coordinates of the same person in the first image of the two images are mapped to the second image using the corresponding transformation matrix to obtain the mapping coordinates.

In practical applications, the distances between multiple key point coordinates and multiple mapping coordinates of the same person in the second image are calculated respectively. The sum of the distances between all key point coordinates and mapping coordinates of the same person is calculated, and the sum is normalized to obtain the action matching score. Specifically, the distance sum can be mapped to the interval [0, 1] using the Sigmoid function to obtain the action matching score.

Through the above embodiment, a corresponding transformation matrix is constructed according to the relative position of the corresponding perspectives of any two images; the key point coordinates of the same person in the first image of any two images are mapped to the second image using the transformation matrix to obtain the mapping coordinates; the distance between each key point coordinate in the second image and the mapping coordinates is calculated; the sum of multiple distances is normalized to obtain the action matching score. This process determines the change matrix based on the relative position of the target image and other images, maps the key point coordinates of the same person in other images to the target image according to the change matrix, calculates the distance between the key point coordinates of the same person in the target image and the mapping coordinates, and determines the action matching score according to the sum of the distances, thereby improving the accuracy of action matching.

In one embodiment, step S105 includes:

For any two perspective images, the first loop step is performed until all values in the matching score matrix are the first preset values, and the matching results of the any two perspective images are obtained, wherein the first loop step includes:

Searching for a maximum matching score in the matching score matrix, obtaining a matching result and determining a matching bounding box in the image according to the row and column where the maximum matching score is located, and setting all values of the row and column where the maximum matching score is located to a first preset value;

Select the image with the largest number of bounding boxes as the target image;

According to the matching results and the target image, the bounding boxes of the same person in all images except the target image are grouped into a bounding box set;

Check cycle consistency against a set of bounding boxes of the same person.

In practical applications, multiple bounding boxes of each person under multiple perspectives are matched and verified, including uniqueness check and cycle consistency check. The uniqueness check means that a bounding box in one perspective corresponds to at most one bounding box in another perspective. When this condition is met, the matching score matrix can be used to obtain preliminary matching results. The cycle consistency check means that bounding boxes that match each other in any three perspectives should be connected into a ring, that is, the bounding boxes that match two bounding boxes in the third perspective should be the same.

In practical applications, the matching score matrix is used to find the maximum matching score in the matrix, and the bounding boxes that match each other in the image are determined according to the row and column where the maximum matching score is located. All values in the row and column where the maximum matching score is located are set to the first preset value, that is, 0, and the matching score maximum value in the matching score matrix is continued. The above operation is repeated until all matrix values are 0, so as to obtain the bounding box matching relationship between each perspective, so that the bounding box meets the uniqueness test.

In practical applications, the image with the most bounding boxes is found among all images as the target image. Taking the target image as the benchmark, each bounding box in it is grouped with its matching bounding box in other images according to the matching results to obtain a bounding box set. This bounding box set represents images of the same person from different perspectives, and the cycle consistency is checked based on the bounding box set of the same person.

Through the above embodiment, for images of any two perspectives, the first loop step is executed until all values in the matching score matrix are the first preset values, and the matching results of the images of any two perspectives are obtained, wherein the first loop step includes: searching for the maximum matching score in the matching score matrix, obtaining the matching result and determining the matching bounding box in the image according to the row and column where the maximum matching score is located, and setting all values in the row and column where the maximum matching score is located to the first preset value; selecting the image with the largest number of bounding boxes as the target image; according to the matching result and the target image, classifying the bounding boxes of the same person in all images except the target image into a bounding box set; and checking the cyclic consistency according to the bounding box set of the same person. This process improves the accuracy of gesture recognition by checking the uniqueness and cyclic consistency of multiple bounding boxes of each person under multiple perspectives.

In one embodiment, checking cycle consistency based on a set of bounding boxes of the same person includes:

According to the bounding boxes of the same person in the target image, the bounding boxes in the bounding box set are divided into a plurality of groups according to a second preset value;

In the case where the bounding boxes in each group match each other, it is determined that the bounding boxes of the same person meet cycle consistency.

In practical applications, the bounding boxes in the bounding box set are subjected to a cyclic consistency check in different combinations in which the number of bounding boxes in each group is a second preset value, such as 3, and individual bounding boxes that do not meet the cyclic consistency check are excluded. If the number of bounding boxes in a group is equal to three and does not form a cycle or the number of bounding boxes is less than three, the group is deleted so that each group of bounding boxes can pass the check.

Through the above embodiment, according to the bounding boxes of the same person in the target image, the bounding boxes in the bounding box set are divided into a plurality of groups by the second preset value; when the bounding boxes in each group match each other, it is determined that the bounding boxes of the same person meet the cycle consistency. This process determines the accuracy of the bounding box matching by checking the cycle consistency of the bounding boxes.

In one embodiment, step S106 includes:

When the multiple bounding boxes of each person pass the verification, the image features of each image are extracted using the third preset neural network model;

The bounding boxes of each person in the multiple images are input into the fourth preset neural network model to obtain the posture information of each person.

In practical applications, the pose information of each person is determined based on the bounding boxes belonging to the same person from different perspectives.

In actual applications, the third preset neural network model includes a HRNet (High-Resolution Net) neural network model. The bounding boxes belonging to the same person obtained from multiple perspectives are input into the third preset neural network model to extract image features from each perspective.

In actual applications, the fourth preset neural network model includes a Transformer neural network model, which inputs the extracted image features of each image into the Transformer neural network model to aggregate image features from different perspectives, and finally connects to a fully connected layer to output predicted posture information.

Through the above embodiment, when multiple bounding boxes of each person pass the verification, the image features of each image are extracted using the third preset neural network model; the bounding boxes of each person in the multiple images are input into the fourth preset neural network model to obtain the posture information of each person. This process can make it possible to identify and obtain the action posture of any individual that has appeared in the viewing angle, making the posture recognition more accurate.

Based on the above-mentioned motion posture recognition method, an embodiment of the present invention further provides a motion posture recognition device 200. FIG2 is a schematic diagram of the structure of a motion posture recognition device provided by an embodiment of the present invention. The device 200 includes:

An image acquisition module 201 is used to acquire images from multiple perspectives of multiple people at the same time;

A recognition module 202, for recognizing a bounding box of each person in each image based on a preset object detection algorithm;

A matching score determination module 203 is used to select images from any two perspectives, match the bounding boxes of the same person in the two images, and obtain a matching score;

A matching score matrix determination module 204 is used to construct a matching score matrix of the same person under multiple viewing angles according to the matching scores, wherein the rows and columns of the matching score matrix are bounding boxes in the image;

A matching verification module 205 is used to perform matching verification on multiple bounding boxes of each person under multiple viewing angles according to a matching score matrix;

The posture information determination module 206 is used to determine the posture information of each person according to each bounding box after multiple bounding boxes of each person have passed the verification.

The motion posture recognition device provided in the embodiment of the invention can implement each process of the motion posture recognition method in the method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

An embodiment of the present invention further provides an electronic device, as shown in FIG3 , which includes a processor 130 and a memory 131 , wherein the memory 131 stores machine executable instructions that can be executed by the processor 130 , and the processor 130 executes the machine executable instructions to implement the above-mentioned motion posture recognition method.

Furthermore, the electronic device shown in FIG. 3 further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

Among them, the memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 132 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bidirectional arrow is used in Figure 3, but it does not mean that there is only one bus or one type of bus.

The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned motion posture recognition method. The specific implementation can be found in the method embodiment, which will not be repeated here.

A motion posture recognition method, recognition device and electronic device provided in an embodiment of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims (9)

1. A method for recognizing a motion gesture, comprising:

acquiring images of multiple visual angles of multiple persons at the same moment;

Identifying a bounding box of each person in each image based on a preset target detection algorithm;

Selecting the images of any two visual angles, and matching the boundary boxes of the same person in the two images to obtain matching scores;

Constructing a matching score matrix of the same person under the multiple view angles according to the matching scores, wherein the rows and columns of the matching score matrix are boundary boxes in the image;

Performing matching verification on the multiple bounding boxes of each person under the multiple view angles according to the matching score matrix;

when the multiple bounding boxes of each person pass verification, determining gesture information of each person according to each bounding box;

the performing the matching check on the bounding boxes of each person under the multiple view angles according to the matching score matrix includes:

And executing a first circulation step for the images of any two view angles until all values in the matching score matrix are first preset values, so as to obtain a matching result of the images of any two view angles, wherein the first circulation step comprises the following steps:

Searching a matching score maximum value in the matching score matrix, obtaining a matching result according to the row and the column where the matching score maximum value is located, determining the matched boundary box in the image, and setting all the values of the row and the column where the matching score maximum value is located as the first preset value;

selecting the image with the largest number of bounding boxes as a target image;

classifying the boundary boxes of the same person in all the images except the target image into a boundary box set according to the matching result and the target image;

the loop consistency is checked against the set of bounding boxes of the same person.

2. The method according to claim 1, wherein the selecting the images from any two perspectives, and matching the bounding boxes of the same person in the two images, to obtain a matching score, includes:

Convolving the boundary frame of each person in the two images by using a first preset neural network to obtain the appearance characteristics of each person;

Calculating appearance matching scores of each person according to the appearance characteristics of each person;

identifying key points in each boundary box by using a second preset neural network;

calculating an action matching score according to the relative positions of the visual angles corresponding to any two images and the coordinates of each key point;

And calculating the product of the action matching score and the appearance matching score to obtain the matching score.

3. The method of claim 2, wherein said calculating an appearance matching score for each person based on the appearance characteristics of each person comprises:

calculating similarity scores of the two images according to the appearance characteristics;

and normalizing the similarity score to obtain the appearance matching score.

4. A method according to claim 3, wherein said calculating an action matching score from the relative positions of the viewing angles corresponding to any two of said images and the coordinates of each of said keypoints comprises:

constructing a corresponding transformation matrix according to the relative positions of the view angles corresponding to any two images;

mapping the coordinates of key points of the same person in a first image in any two images to a second image by using the transformation matrix to obtain mapped coordinates;

calculating the distance between each key point coordinate in the second image and the mapping coordinate;

And normalizing the sum of the distances to obtain the action matching score.

5. The method of claim 1, wherein the verifying cyclical compliance from the set of bounding boxes of the same person comprises:

dividing the boundary boxes in the boundary box set into a plurality of subgroups according to the boundary boxes of the same person in the target image by a second preset value;

In the case that the bounding boxes in each of the subgroups match each other, it is determined that the bounding boxes of the same person conform to the cyclical consistency.

6. The method according to claim 1, wherein said determining pose information of each person from each of said bounding boxes after said plurality of said bounding boxes of each person pass verification comprises:

When the multiple bounding boxes of each person pass the verification, extracting the image characteristics of each image by using a third preset neural network model;

And inputting the boundary box of each person in the plurality of images to a fourth preset neural network model to obtain the gesture information of each person.

7. A motion gesture recognition apparatus, comprising:

the image acquisition module is used for acquiring images of multiple visual angles of multiple persons at the same moment;

the identification module is used for identifying the boundary box of each person in each image based on a preset target detection algorithm;

The matching score determining module is used for selecting the images of any two visual angles, and matching the boundary boxes of the same person in the two images to obtain a matching score;

A matching score matrix determining module, configured to construct a matching score matrix of the same person under the multiple view angles according to the matching score, where rows and columns of the matching score matrix are bounding boxes in the image;

The matching verification module is used for carrying out matching verification on the plurality of boundary boxes of each person under the plurality of view angles according to the matching score matrix;

the gesture information determining module is used for determining gesture information of each person according to each bounding box after the bounding boxes of each person pass verification;

the performing the matching check on the bounding boxes of each person under the multiple view angles according to the matching score matrix includes:

And executing a first circulation step for the images of any two view angles until all values in the matching score matrix are first preset values, so as to obtain a matching result of the images of any two view angles, wherein the first circulation step comprises the following steps:

Searching a matching score maximum value in the matching score matrix, obtaining a matching result according to the row and the column where the matching score maximum value is located, determining the matched boundary box in the image, and setting all the values of the row and the column where the matching score maximum value is located as the first preset value;

selecting the image with the largest number of bounding boxes as a target image;

classifying the boundary boxes of the same person in all the images except the target image into a boundary box set according to the matching result and the target image;

the loop consistency is checked against the set of bounding boxes of the same person.

8. An electronic device comprising a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of identifying a motion gesture of any one of claims 1 to 6.

9. A machine-readable storage medium having instructions stored thereon, which when executed by a processor implement the method of motion gesture recognition of any one of claims 1 to 6.