CN108960178A - A kind of manpower Attitude estimation method and system - Google Patents

Abstract

The estimation method and system of a kind of manpower posture provided by the invention, wherein method includes: the depth image for obtaining target hand, and depth image is input to the convolutional layer of the first default neural network, exports the characteristic pattern of target hand；The Attitude estimation result of characteristic pattern and last iteration is input to the decision-making level of the first default neural network, exports the Attitude estimation result of current iteration；If the deviation between the Attitude estimation result of current iteration and the Attitude estimation result of last iteration is less than preset threshold, using the Attitude estimation result of current iteration as the final carriage estimated result of target hand.This method and system can farthest improve the accuracy of Attitude estimation result, solve the problems, such as to cause Attitude estimation result inaccurate due to the self-similarity between finger.

Description

Method and system for estimating human hand pose

technical field

The present invention relates to the technical field of image recognition, and more specifically, to a method and system for estimating a human hand pose.

Background technique

The problem of hand pose estimation refers to accurately estimating the three-dimensional coordinate positions of the human hand skeleton nodes from the image. This is a key issue in the fields of computer vision and human-computer interaction, and has important implications in fields such as virtual reality, augmented reality, non-contact interaction, and gesture recognition. With the rise and development of commercial and cheap depth cameras, hand pose estimation algorithms based on depth images have become a hot topic.

Existing methods for hand pose estimation generally fall into three categories: model fitting methods, discriminative methods, and hybrid methods. The model fitting method uses an optimization method to fit a predefined hand model to the input depth image. The discriminative method is a completely data-driven method, and its goal is to learn a regressor through labeled training data to predict the pose information of the hand from the input depth image. The hybrid method is a combination of the above two methods. Usually, the initial estimate is obtained through the discriminant method, and then the model fitting method is used to correct the result.

However, due to the small area of the hand itself, the noise of the depth image is large when the distance from the camera is relatively large; the degree of freedom of the hand is high, the relationship between the joints is complicated, and self-occlusion is easy to occur; in addition, the finger itself has a high self-occlusion. similarity. The above problems make it difficult for the existing hand note estimation methods to obtain high-precision hand pose estimation results.

In view of this, there is an urgent need to provide a method and system for estimating the pose of a human hand, so as to effectively improve the accuracy of the result of estimating the pose of the human hand.

Contents of the invention

In order to overcome the problem that it is difficult to obtain high-precision human hand pose estimation results in the existing hand pose estimation methods, the present invention provides a human hand pose estimation method and system.

On the one hand, the present invention provides a kind of estimation method of human hand posture, comprising:

Obtain the depth image of the target hand, input the depth image to the convolutional layer of the first preset neural network, and output the feature map of the target hand;

The feature map and the attitude estimation result of the last iteration are input to the decision-making layer of the first preset neural network, and the attitude estimation result of the current iteration is output;

If the deviation between the pose estimation result of the current iteration and the pose estimation result of the previous iteration is less than a preset threshold, the pose estimation result of the current iteration is taken as the final pose estimation result of the target hand.

Preferably, the output of the pose estimation result of the current iteration further includes:

If the deviation between the attitude estimation result of the current iteration and the attitude estimation result of the previous iteration is greater than a preset threshold, the attitude estimation result of the current iteration is input to the decision-making layer as the attitude estimation result of the previous iteration.

Preferably, the decision-making layer includes a feature optimization layer and a fully connected layer;

Correspondingly, the said feature map and the attitude estimation result of the last iteration are input to the decision-making layer of the first preset neural network, and the attitude estimation result of the current iteration is output, specifically:

Input the feature map and the pose estimation result of the last iteration into the feature optimization layer, and use the feature optimization layer to extract the regional features corresponding to each joint point from the feature map according to the pose estimation result of the last iteration ;

Input the regional features corresponding to all related nodes to the fully connected layer, and output the pose estimation result of the current iteration.

Preferably, the feature optimization layer is used to extract the region features corresponding to each joint point from the feature map according to the pose estimation result of the last iteration, specifically:

For any joint point in the pose estimation result of the last iteration, use the feature optimization layer to obtain the projection point corresponding to the joint point in the feature map, and extract the area features of a preset size centered on the projection point , to obtain the regional features corresponding to the joint point.

Preferably, the fully connected layer includes a first fully connected layer, a second fully connected layer and a third fully connected layer;

Correspondingly, the regional features corresponding to all related nodes are input to the fully connected layer, and the pose estimation result of the current iteration is output, specifically:

Inputting the regional features corresponding to all related nodes to the first fully connected layer, using the first fully connected layer to concatenate the regional features corresponding to the joint points belonging to the same finger to obtain the local features corresponding to each finger;

Inputting the local features corresponding to all fingers to the second fully connected layer, using the second fully connected layer to concatenate the local features corresponding to all fingers to obtain the overall features corresponding to the target hand;

The overall feature is input to the third fully connected layer, and the pose estimation result of the current iteration is output.

Preferably, the input of the feature map and the pose estimation result of the last iteration to the decision-making layer of the first preset neural network also includes a step of obtaining an initial pose estimation result, specifically:

The depth image is input to a second preset neural network, and an initial pose estimation result is obtained according to an output result of the second preset neural network.

Preferably, the input of the depth image to the second preset neural network also includes:

Obtaining a plurality of depth image samples of hands, marking a preset number of joint points in each of the depth image samples, and using each of the marked depth image samples as training samples;

Using all the training samples to train the second preset neural network.

In one aspect, the present invention provides a system for estimating the posture of a human hand, comprising:

The feature extraction module is used to obtain the depth image of the target hand, and input the depth image to the convolutional layer of the first preset neural network, and output the feature map of the target hand;

The result iteration module is used to input the feature map and the attitude estimation result of the last iteration to the decision-making layer of the first preset neural network, and output the attitude estimation result of the current iteration;

The result determination module is used to use the pose estimation result of the current iteration as the final pose estimation result of the target hand if the deviation between the pose estimation result of the current iteration and the pose estimation result of the previous iteration is less than a preset threshold.

In one aspect, the present invention provides an electronic device, comprising:

at least one processor; and

at least one memory communicatively coupled to the processor, wherein:

The memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute any of the methods described above.

In one aspect, the present invention provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute any one of the methods described above.

A method and system for estimating the posture of a human hand provided by the present invention obtains a depth image of the target hand, inputs the depth image to the convolutional layer of the first preset neural network, and outputs a feature map of the target hand; combines the feature map and The attitude estimation result of the last iteration is input to the decision-making layer of the first preset neural network, and the attitude estimation result of the current iteration is output; if the deviation between the attitude estimation result of the current iteration and the attitude estimation result of the previous iteration is less than the preset threshold, The pose estimation result of the current iteration is taken as the final pose estimation result of the target hand. The method and system guide the feature extraction of the current iteration through the attitude estimation result of the previous iteration, so as to further perform feature extraction on the basis of the feature map obtained by convolution to obtain more optimized features, and finally according to the more optimized features Perform attitude estimation to obtain the attitude estimation result of the current iteration, which can make the attitude estimation result of the current iteration more accurate than the estimation result of the previous iteration. Finally, when the iteration process basically converges, the attitude estimation result of the current iteration As the final pose estimation result of the hand, the accuracy of the pose estimation result can be improved to the greatest extent, and the problem of inaccurate pose estimation results due to the self-similarity between fingers is solved.

Description of drawings

1 is a schematic diagram of an overall flow of a method for estimating a human hand pose according to an embodiment of the present invention;

2 is a schematic diagram of an iterative process of a first preset neural network according to an embodiment of the present invention;

3 is a schematic diagram of the overall structure of a human hand pose estimation system according to an embodiment of the present invention;

FIG. 4 is a schematic structural frame diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The specific implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

Fig. 1 is a schematic diagram of the overall flow of a method for estimating the posture of a human hand according to an embodiment of the present invention. As shown in Fig. 1 , the present invention provides a method for estimating the posture of a human hand, including:

S1, acquiring a depth image of the target hand, inputting the depth image to the convolutional layer of the first preset neural network, and outputting a feature map of the target hand;

Specifically, in this embodiment, the hand that needs pose estimation is taken as the target hand. First, obtain the depth image of the target hand, where the gray value of each pixel in the depth image can be used to represent the distance of a point in the scene from the camera. The methods for obtaining depth images can be divided into two categories: passive ranging sensing and active depth sensing. Among them, the most commonly used method in passive ranging sensing is binocular stereo vision, which uses two cameras separated by a certain distance to simultaneously Obtain two images of the same scene, find the corresponding pixel points in the two images through the stereo matching algorithm, and then calculate the time difference information according to the triangulation principle, and the parallax information can be used to represent the depth information of the object in the scene through conversion; the active ranging transmission The sense is that the device itself completes the collection of depth information by emitting energy. In this embodiment, the acquisition method of the depth image of the target hand can be set according to actual needs, and is not specifically limited here.

Further, the depth image of the target hand is input to the convolution layer of the first preset neural network, and the depth image is convolved through the convolution layer of the first preset neural network, and finally the convolution layer is processed by convolution Output the feature map of the target hand. Wherein, the first preset neural network is a convolutional neural network, and the first preset neural network may include multiple convolutional layers, and each convolutional layer may be followed by a corresponding pooling layer. Generally, the convolutional The convolutional layer and the pooling layer are alternately set to pool the output of the convolutional layer through the pooling layer. In this embodiment, the number of convolutional layers and pooling layers can be set according to actual needs, and is not specifically limited here.

It should be noted that if the first preset neural network includes multiple convolutional layers, the result output by the last convolutional layer is used as the feature map of the target hand.

S2, input the feature map and the attitude estimation result of the last iteration to the decision-making layer of the first preset neural network, and output the attitude estimation result of the current iteration;

It should be noted that after the feature map of the target hand is obtained through the above method steps, since each finger in the feature map has a high self-similarity, in view of this, in this embodiment, the gesture-guided iterative method is adopted in the feature map On the basis of further feature extraction, more optimized features are extracted.

Specifically, in this embodiment, a decision-making layer is customized in the first preset neural network, and the decision-making layer is set after the convolution layer. The decision layer has two input interfaces, one of which is used to input the feature map output by the convolutional layer, and the other input interface is used to input the pose estimation result of the previous iteration. In this embodiment, an operation of an output result of the first preset neural network is regarded as an iteration, wherein the pose estimation result of the previous iteration is the pose estimation result output by the first preset neural network in the last computation.

After the feature map and the pose estimation result of the last iteration are input to the decision-making layer of the first preset neural network, the pose estimation result of the last iteration can be used to guide the decision-making layer to extract more optimized features from the feature map . It can be understood that the estimated value of the three-dimensional coordinates of each joint point of the target hand has been obtained in the attitude estimation result of the last iteration. The corresponding features of each joint point in the current iteration can be obtained in a targeted manner from the feature map, and the extracted features corresponding to each joint point are more accurate, and more optimized features can be obtained. Finally, the decision-making layer can make decisions based on the extracted more optimized features, and output the pose estimation results of the current iteration.

It should be noted that, during each iteration of the first preset neural network, the decision-making layer therein needs to simultaneously input a feature map and a pose estimation result. For the current iteration, the pose estimation result input to the decision-making layer is the pose estimation result of the previous iteration. It can be seen from this that the steps of the above method start from the second iteration by default. Therefore, for the first iteration, a pose estimation result needs to be initialized, so that the first preset neural network inputs the initialized pose estimation result to the decision-making layer during the first iteration. In practical applications, the initialized attitude estimation result can be obtained through an existing attitude estimation method, and the specific acquisition method can be set according to actual needs, which is not specifically limited here.

S3. If the deviation between the pose estimation result of the current iteration and the pose estimation result of the previous iteration is less than a preset threshold, use the pose estimation result of the current iteration as the final pose estimation result of the target hand.

Specifically, in the above iterative process, if the deviation between the pose estimation result of the current iteration and the pose estimation result of the previous iteration is smaller than the preset threshold, that is, the result output by the first preset neural network in the current iteration is different from the previous If the deviation between the output results in the iteration is less than the preset threshold, it can be determined that the attitude estimation result of the current iteration is basically consistent with the attitude estimation result of the previous iteration, that is, the iterative process of the first preset neural network basically converges, and no further Execute subsequent iteration steps. At this point, the pose estimation result of the current iteration is taken as the final pose estimation result of the target hand. Wherein, the preset threshold may be set according to actual requirements, and is not specifically limited here.

A method for estimating the posture of a human hand provided by the present invention is to obtain a depth image of the target hand, input the depth image to the convolution layer of the first preset neural network, and output the feature map of the target hand; combine the feature map with the previous The iterative attitude estimation result is input to the decision-making layer of the first preset neural network, and the attitude estimation result of the current iteration is output; if the deviation between the attitude estimation result of the current iteration and the attitude estimation result of the previous iteration is less than the preset threshold, the The pose estimation result of the current iteration is used as the final pose estimation result of the target hand. This method guides the feature extraction of the current iteration through the pose estimation result of the previous iteration, and further performs feature extraction on the basis of the feature map obtained by convolution to obtain more optimized features, and finally performs pose based on the more optimized features. Estimation, the attitude estimation result of the current iteration is obtained, which can make the attitude estimation result of the current iteration more accurate than the estimation result of the previous iteration, and finally when the iteration process basically converges, the attitude estimation result of the current iteration is used as the manual The final pose estimation result of the part can maximize the accuracy of the pose estimation result and solve the problem of inaccurate pose estimation results due to the self-similarity between fingers.

Based on any of the above-mentioned embodiments, a method for estimating the posture of a human hand is provided, which outputs the posture estimation result of the current iteration, and then further includes: if the deviation between the posture estimation result of the current iteration and the posture estimation result of the previous iteration is greater than a preset threshold , then the attitude estimation result of the current iteration is input to the decision-making layer as the attitude estimation result of the previous iteration.

Specifically, in the above iterative process, if the deviation between the pose estimation result of the current iteration and the pose estimation result of the previous iteration is greater than the preset threshold, that is, the result output by the first preset neural network in the current iteration is different from the previous If the deviation between the output results in the iteration is greater than the preset threshold, it can be determined that the iterative process of the first preset neural network has not converged, and at this time, the attitude estimation result of the current iteration is input to the decision-making level. That is, after the current iteration is completed, the pose estimation result of the current iteration and the feature map output by the convolutional layer of the first preset neural network are input to the decision-making layer of the first preset neural network, and the next iteration is continued to obtain The pose estimation result of the next iteration. Wherein, the preset threshold may be set according to actual requirements, and is not specifically limited here.

In the method for estimating the posture of a human hand provided by the present invention, after outputting the posture estimation result of the current iteration, if the deviation between the posture estimation result of the current iteration and the posture estimation result of the previous iteration is greater than a preset threshold, the The attitude estimation result is input to the decision-making layer as the attitude estimation result of the previous iteration. The method can ensure that the first preset neural network continuously iterates until the iteration converges, so that the attitude estimation result of the last iteration is used as the final attitude estimation result, which is beneficial to ensure the accuracy of the attitude estimation result.

Based on any of the above-mentioned embodiments, a method for estimating the pose of a human hand is provided, the decision-making layer includes a feature optimization layer and a fully connected layer; correspondingly, the feature map and the pose estimation result of the last iteration are input to the first preset neural network The decision-making layer outputs the pose estimation result of the current iteration, specifically: input the feature map and the pose estimation result of the previous iteration to the feature optimization layer, and use the feature optimization layer to extract each The regional features corresponding to the joint points; input the regional features corresponding to all relevant nodes to the fully connected layer, and output the pose estimation result of the current iteration.

Specifically, in this embodiment, the decision-making layer of the first preset neural network further includes a feature optimization layer and a fully-connected layer. On this basis, after the feature map and the pose estimation result of the last iteration are input to the decision-making layer of the first preset neural network, the feature map and the pose estimation result of the last iteration will first be input to the feature optimization in the decision-making layer. After obtaining the feature map and the pose estimation result of the last iteration, the feature optimization layer extracts the region features corresponding to each joint point of the target hand from the feature map according to the pose estimation result of the last iteration. It can be understood that the estimated value of the three-dimensional coordinates of each joint point of the target hand has been obtained in the pose estimation result of the last iteration. On this basis, the feature optimization layer is based on the three-dimensional The estimated value of the coordinates can be targeted to obtain the features corresponding to each joint point in the current iteration from the feature map, and the features of a certain area can be extracted for each joint point as the area feature corresponding to each joint point.

Through the above method steps, the regional features corresponding to all relevant nodes of the target hand can be obtained. Afterwards, the regional features corresponding to all related nodes are input to the fully connected layer in the decision-making layer, and the regional features corresponding to all related nodes are spliced by using the fully connected layer, and finally the pose of the target hand in the current iteration is calculated according to the spliced features. Estimate and output the pose estimation result of the current iteration.

A method for estimating the posture of a human hand provided by the present invention, the feature map and the pose estimation result of the last iteration are input to the feature optimization layer, and each joint point is extracted from the feature map according to the pose estimation result of the last iteration by using the feature optimization layer Corresponding regional features; input the regional features corresponding to all related nodes to the fully connected layer, and output the pose estimation result of the current iteration. This method guides the extraction of joint point features in the current iteration through the pose estimation results of the previous iteration, and finally stitches the extracted joint point features to obtain the pose estimation results of the current iteration, which can make the pose estimation of the current iteration The result is more accurate than the pose estimation result of the previous iteration.

Based on any of the above embodiments, a method for estimating the pose of a human hand is provided, using the feature optimization layer to extract the region features corresponding to each joint point from the feature map according to the pose estimation result of the last iteration, specifically: for the last iteration For any joint point in the pose estimation result, the feature optimization layer is used to obtain the projection point corresponding to the joint point in the feature map, and the area feature of the preset size is extracted with the projection point as the center, and the area feature corresponding to the joint point is obtained.

Specifically, after inputting the feature map and the pose estimation result of the last iteration into the feature optimization layer of the first preset neural network, the feature optimization layer can obtain the three-dimensional coordinates of each joint point in the pose estimation result of the last iteration. On this basis, for any joint point in the pose estimation result of the last iteration, the projection point corresponding to the joint point is obtained in the feature map according to the three-dimensional coordinates of the joint point. That is, the pose estimation result of the last iteration can be projected into the feature map, so as to obtain the projection points corresponding to each joint point in the pose estimation result of the last iteration in the feature map.

Through the above method steps, the projection points corresponding to all relevant nodes in the pose estimation result of the last iteration can be obtained. On this basis, for any projected point, with the projected point as the center, the region feature of the preset size is extracted from the feature map, and the region feature corresponding to the projected point can be obtained, that is, the region corresponding to the projected point The region features corresponding to the joint points. In this way, the regional features corresponding to all relevant nodes can be obtained and used as the regional features corresponding to all relevant nodes in the current iteration. Among them, the preset size can be set to 7×7, that is, the region feature of 7×7 size is extracted centering on the projection point. In practical applications, the preset size can be set according to actual needs, which is not specifically limited here.

In the method for estimating the posture of a human hand provided by the present invention, for any joint point in the posture estimation result of the last iteration, the feature optimization layer is used to obtain the projection point corresponding to the joint point in the feature map, and the projection point is extracted as the center. The area feature of the preset size is used to obtain the area feature corresponding to the joint point. This method guides the extraction of joint point features in the current iteration through the attitude estimation results of the previous iteration, and can extract more optimized joint point features, so that the attitude estimation results of the current iteration are better than those of the previous iteration. The result is more accurate.

Based on any of the above-mentioned embodiments, a method for estimating the pose of a human hand is provided. The fully-connected layer includes a first fully-connected layer, a second fully-connected layer, and a third fully-connected layer; To the fully connected layer, output the pose estimation result of the current iteration, specifically: input the regional features corresponding to all related nodes to the first fully connected layer, and use the first fully connected layer to perform the regional features corresponding to the joint points belonging to the same finger Connect in series to obtain the local features corresponding to each finger; input the local features corresponding to all fingers to the second fully connected layer, and use the second fully connected layer to concatenate the local features corresponding to all fingers to obtain the corresponding The overall feature; the overall feature is input to the third fully connected layer, and the pose estimation result of the current iteration is output.

Specifically, in this embodiment, the fully connected layer of the first preset neural network includes a first fully connected layer, a second fully connected layer, and a third fully connected layer. On this basis, after inputting the regional features corresponding to all relevant nodes into the fully connected layer of the first preset neural network, the regional features corresponding to all relevant nodes will first be input into the first fully connected layer, using the first fully connected The layer concatenates the regional features corresponding to the joint points belonging to the same finger to obtain the local features corresponding to each finger. That is, among the regional features corresponding to all related nodes, the regional features corresponding to all related nodes belonging to the thumb are concatenated to obtain the local features corresponding to the thumb; the regional features corresponding to all related nodes belonging to the index finger are concatenated , to obtain the local features corresponding to the index finger; concatenate the regional features corresponding to all related nodes belonging to the middle finger to obtain the local features corresponding to the middle finger; concatenate the regional features corresponding to all related nodes belonging to the ring finger to obtain the corresponding local features; concatenate the regional features corresponding to all related nodes belonging to the little finger to obtain the local features corresponding to the little finger.

Further, input the local features corresponding to all fingers to the second fully connected layer, and use the second fully connected layer to concatenate the local features corresponding to all fingers to obtain the overall features corresponding to the target hand. Finally, the overall feature corresponding to the target hand is input to the third fully connected layer, and the third fully connected layer can output the pose estimation result of the current iteration after performing a regression operation according to the overall feature.

The method for estimating the posture of a human hand provided by the present invention is to input the regional features corresponding to all related nodes into the first fully connected layer, and use the first fully connected layer to concatenate the regional features corresponding to the joint points belonging to the same finger to obtain The local features corresponding to each finger; input the local features corresponding to all fingers to the second fully connected layer, and use the second fully connected layer to concatenate the local features corresponding to all fingers to obtain the overall features corresponding to the target hand; The overall feature is input to the third fully connected layer, and the pose estimation result of the current iteration is output. When concatenating the regional features corresponding to all relevant nodes, this method first concatenates the regional features corresponding to the joint points belonging to the same finger to obtain the local features corresponding to each finger, and then concatenates the local features corresponding to all fingers. The overall features of the hand are obtained by receiving the overall features, so that the finally obtained overall features can effectively satisfy the constraint relationship between each joint point, which is conducive to improving the accuracy of the pose estimation results.

Based on any of the above embodiments, a method for estimating the pose of a human hand is provided, the feature map and the pose estimation result of the last iteration are input to the decision-making layer of the first preset neural network, and the step of obtaining the initial pose estimation result is also included before, Specifically, the depth image is input to the second preset neural network, and an initial pose estimation result is obtained according to an output result of the second preset neural network.

Specifically, during each iteration of the first preset neural network, the decision-making layer therein needs to simultaneously input a feature map and a pose estimation result. For the current iteration, the pose estimation result input to the decision-making layer is the pose estimation result of the previous iteration. It can be seen from this that the steps of the above method start from the second iteration by default. Therefore, for the first iteration, a pose estimation result needs to be initialized, so that the first preset neural network inputs the initialized pose estimation result to the decision-making layer during the first iteration.

In this embodiment, before performing iterations, the depth image of the target hand is first input to the second preset neural network, and the posture of the target hand is initially estimated through the second preset neural network, and the second preset neural network The output of the network is used as the initial pose estimation result of the target hand. Wherein the second preset neural network may be a convolutional neural network, and the second preset neural network is pre-trained. In practical applications, the initialized attitude estimation result can also be obtained by other attitude estimation methods, and the specific acquisition method can be set according to actual needs, which is not specifically limited here.

The present invention provides a method for estimating the posture of a human hand. Before inputting the feature map and the posture estimation result of the last iteration into the decision-making layer of the first preset neural network, the depth image is input into the second preset neural network. According to the first 2. Preset the output result of the neural network to obtain the initial pose estimation result. The method obtains the initial pose estimation result of the target hand through the second preset neural network, which is beneficial for the first preset neural network to perform the first iteration according to the initial pose estimation result.

Based on any of the above-mentioned embodiments, a method for estimating the pose of a human hand is provided, and the depth image is input to the second preset neural network. Before that, it also includes: acquiring a plurality of depth image samples of hands, and marking each depth image sample A preset number of joint points is generated, and each marked depth image sample is used as a training sample; all training samples are used to train the second preset neural network.

Specifically, before inputting the depth image into the second preset neural network, the second preset neural network needs to be trained. The specific training process is as follows:

First obtain the depth image samples of the hand, and mark a preset number of joint points in each depth image sample. In this embodiment, a total of 14 joint points are marked for each depth image sample, that is, in each depth image A total of 14 joint points are marked for all finger parts and palm parts of the sample. On this basis, each marked depth image is used as a training sample, and finally all the training samples are input into the second preset neural network to train the second preset neural network. In the process of training, a target error can be set in advance, when the error between the attitude estimation result of the output of the second preset neural network and the actual attitude estimation result is less than the target error, then the second preset neural network training is completed .

It can be understood that if 14 joint points are marked in the training samples used in the training process of the second preset neural network, then the depth image of the target hand is input to the initial pose estimation output by the second preset neural network The result also contains the three-dimensional coordinates of 14 joint points. Correspondingly, since the first iteration of the first preset neural network is based on the initial pose estimation result, feature extraction is performed, and the pose estimation result of the first iteration is finally obtained. , and in each subsequent iteration, the feature extraction of the current iteration is carried out according to the attitude result of the previous iteration. Therefore, the number of joint points in the attitude estimation result output by each iteration of the first preset neural network is the same as The number of joint points in the initial pose estimation result is the same, that is, the number of joint points in the pose estimation result output by each iteration of the first preset neural network is also 14.

In the method for estimating the posture of a human hand provided by the present invention, before inputting the depth image into the second preset neural network, a preset number of joint points are marked in each depth image sample, and each depth image sample after marking is As training samples; using all the training samples to train the second preset neural network. In this method, by training the second preset neural network, it is beneficial to use the second preset neural network to obtain the initial pose estimation result of the target hand after the second preset neural network is trained, which in turn is beneficial to the first preset neural network. The neural network performs the first iteration based on the initial pose estimation results.

In order to facilitate the understanding of the above iterative steps of the first preset neural network, the following examples are used for specific description:

Fig. 2 is a schematic diagram of the iterative process of the first preset neural network according to the embodiment of the present invention. As shown in Fig. 2, first, the depth image of the target hand is input to the first preset neural network, wherein the first preset neural network is The CNN convolutional neural network obtains the feature map corresponding to the target hand after the convolution processing of the first preset neural network.

For the tth iteration, the pose estimation result pose _t-1 and the feature map of the t-1th iteration are input to the feature optimization layer of the first preset neural network (the first preset neural network is not shown in the figure Each layer structure of the network), in the feature optimization layer, the pose estimation result of the t-1th iteration is projected into the feature map, so that each joint in the pose estimation result of the t-1th iteration is obtained in the feature map Points corresponding to the projection points, and finally take each projection point as the center to extract the regional features of 7×7 size, so that the regional features corresponding to each joint point in the t-th iteration can be obtained. As shown in FIG. 2 , the regional features corresponding to each joint point in the t-th iteration include the regional features corresponding to the 14 joint points where the fingertips, heels, and palm points of each finger are located.

Then input the regional features corresponding to all related nodes to the first fully connected layer of the first preset neural network, and concatenate the regional features belonging to the same finger in the first fully connected layer to obtain the local features corresponding to each finger; Then input the local features corresponding to all fingers to the second fully connected layer of the first preset neural network, and concatenate the local features corresponding to all fingers in the second fully connected layer to obtain the overall features of the target hand; finally, Input the overall features of the target hand into the third fully connected layer of the first preset neural network, and perform regression calculation according to the overall features in the third fully connected layer to obtain the pose estimation result pose _t of the t-th iteration.

For the t+1th iteration, the pose estimation result pose _t and the feature map of the tth iteration are input to the feature optimization layer of the first preset neural network to perform the t+1th iteration.

FIG. 3 is a schematic diagram of the overall structure of a human hand posture estimation system according to an embodiment of the present invention. As shown in FIG. 3 , based on any of the above-mentioned embodiments, a human hand posture estimation system is provided, including:

The feature extraction module 1 is used to obtain the depth image of the target hand, input the depth image to the convolutional layer of the first preset neural network, and output the feature map of the target hand;

The result iteration module 2 is used to input the feature map and the attitude estimation result of the last iteration to the decision-making layer of the first preset neural network, and output the attitude estimation result of the current iteration;

The result determination module 3 is configured to use the pose estimation result of the current iteration as the final pose estimation result of the target hand if the deviation between the pose estimation result of the current iteration and the pose estimation result of the previous iteration is less than a preset threshold.

Specifically, the present invention provides a human hand pose estimation system, which includes a feature extraction module 1, a result iteration module 2, and a result determination module 3, and realizes the method steps in any of the above method embodiments through cooperation between the modules, and specifically realizes For the process, reference may be made to the foregoing method embodiments, and details are not repeated here.

A system for estimating the posture of a human hand provided by the present invention obtains a depth image of the target hand, inputs the depth image to the convolutional layer of the first preset neural network, and outputs a feature map of the target hand; combines the feature map with the previous The iterative attitude estimation result is input to the decision-making layer of the first preset neural network, and the attitude estimation result of the current iteration is output; if the deviation between the attitude estimation result of the current iteration and the attitude estimation result of the previous iteration is less than the preset threshold, the The pose estimation result of the current iteration is used as the final pose estimation result of the target hand. The system guides the feature extraction of the current iteration through the pose estimation result of the previous iteration, and further performs feature extraction on the basis of the feature map obtained by convolution to obtain more optimized features, and finally performs pose based on the more optimized features. Estimation, the attitude estimation result of the current iteration is obtained, which can make the attitude estimation result of the current iteration more accurate than the estimation result of the previous iteration, and finally when the iteration process basically converges, the attitude estimation result of the current iteration is used as the manual The final pose estimation result of the part can maximize the accuracy of the pose estimation result and solve the problem of inaccurate pose estimation results due to the self-similarity between fingers.

Fig. 4 shows a structural block diagram of an electronic device according to an embodiment of the present invention. Referring to Fig. 4, described electronic equipment comprises: processor (processor) 41, memory (memory) 42 and bus 43; Wherein, described processor 41 and memory 42 complete mutual communication through described bus 43; The processor 41 is used to call the program instructions in the memory 42 to execute the method provided by any of the above method embodiments, for example, including: acquiring the depth image of the target hand, and inputting the depth image to the first preset neural network The convolutional layer of the target hand outputs the feature map of the target hand; the feature map and the pose estimation result of the previous iteration are input to the decision-making layer of the first preset neural network, and the pose estimation result of the current iteration is output; if the pose estimation result of the current iteration If the deviation between the result and the pose estimation result of the previous iteration is less than the preset threshold, the pose estimation result of the current iteration is taken as the final pose estimation result of the target hand.

This embodiment discloses a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by the computer, the computer The method provided by any of the above method embodiments can be executed, for example, including: acquiring a depth image of the target hand, inputting the depth image to the convolutional layer of the first preset neural network, and outputting a feature map of the target hand; The graph and the attitude estimation result of the last iteration are input to the decision-making layer of the first preset neural network, and the attitude estimation result of the current iteration is output; if the deviation between the attitude estimation result of the current iteration and the attitude estimation result of the last iteration is less than the preset threshold, the pose estimation result of the current iteration is taken as the final pose estimation result of the target hand.

This embodiment provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the method provided in any of the above method embodiments, for example Including: obtaining the depth image of the target hand, inputting the depth image to the convolutional layer of the first preset neural network, and outputting the feature map of the target hand; inputting the feature map and the attitude estimation result of the last iteration to the first preset The decision-making layer of the neural network is set to output the pose estimation result of the current iteration; if the deviation between the pose estimation result of the current iteration and the pose estimation result of the previous iteration is less than the preset threshold, the pose estimation result of the current iteration is used as the target hand The final pose estimation result.

Those of ordinary skill in the art can understand that all or part of the steps for realizing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the It includes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other various media that can store program codes.

The above-described embodiments such as electronic equipment are only illustrative, and the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, It can be located in one place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative efforts.

Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

Finally, the method of the present application is only a preferred embodiment, and is not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. a kind of estimation method of manpower posture characterized by comprising

The depth image, is input to the convolutional layer of the first default neural network by the depth image for obtaining target hand, is exported The characteristic pattern of target hand；

The Attitude estimation result of the characteristic pattern and last iteration is input to the decision-making level of the first default neural network, output The Attitude estimation result of current iteration；

If the deviation between the Attitude estimation result of current iteration and the Attitude estimation result of last iteration is less than preset threshold, Using the Attitude estimation result of current iteration as the final carriage estimated result of target hand.

2. the method according to claim 1, wherein the Attitude estimation of the output current iteration is as a result, later Further include:

If the deviation between the Attitude estimation result of current iteration and the Attitude estimation result of last iteration is greater than preset threshold, The decision-making level is input to using the Attitude estimation result of current iteration as the Attitude estimation result of last iteration.

3. the method according to claim 1, wherein the decision-making level includes characteristic optimization layer and full articulamentum；

Correspondingly, the Attitude estimation result by the characteristic pattern and last iteration is input to the first default neural network Decision-making level, export the Attitude estimation of current iteration as a result, specifically:

The Attitude estimation result of the characteristic pattern and last iteration is input to the characteristic optimization layer, it is excellent using the feature Change layer and the corresponding provincial characteristics of each artis is extracted from the characteristic pattern according to the Attitude estimation result of last iteration；

The corresponding provincial characteristics of all artis is input to the full articulamentum, exports the Attitude estimation result of current iteration.

4. according to the method described in claim 3, it is characterized in that, described utilize the characteristic optimization layer according to last iteration Attitude estimation result the corresponding provincial characteristics of each artis is extracted from the characteristic pattern, specifically:

For any one artis in the Attitude estimation result of last iteration, using the characteristic optimization layer in the spy The corresponding subpoint of the artis is obtained in sign figure, and the provincial characteristics of default size is extracted centered on the subpoint, is obtained The corresponding provincial characteristics of the artis.

5. according to the method described in claim 3, it is characterized in that, the full articulamentum include the first full articulamentum, it is second complete Articulamentum and the full articulamentum of third；

Correspondingly, the corresponding provincial characteristics of all artis is input to the full articulamentum, the posture for exporting current iteration is estimated Meter as a result, specifically:

The corresponding provincial characteristics of all artis is input to the described first full articulamentum, will be belonged to using the described first full articulamentum It is concatenated in the corresponding provincial characteristics of the artis of same finger, obtains the corresponding local feature of each finger；

The corresponding local feature of all fingers is input to the described second full articulamentum, will be owned using the described second full articulamentum The corresponding local feature of finger is concatenated, and the corresponding global feature of the target hand is obtained；

The global feature is input to the full articulamentum of the third, exports the Attitude estimation result of current iteration.

6. the method according to claim 1, wherein the posture by the characteristic pattern and last iteration is estimated Meter result is input to the decision-making level of the first default neural network, further includes the steps that obtaining initial attitude estimated result, tool before Body are as follows:

The depth image is input to the second default neural network, according to the output of the described second default neural network as a result, Obtain initial attitude estimated result.

7. according to the method described in claim 6, it is characterized in that, described be input to the second default nerve for the depth image Network, before further include:

The depth image sample for obtaining multiple hands marks the joint of preset quantity in each depth image sample Point, using depth image sample described each of after label as training sample；

The described second default neural network is trained using all training samples.

8. a kind of estimating system of manpower posture characterized by comprising

The depth image is input to the first default nerve for obtaining the depth image of target hand by characteristic extracting module The convolutional layer of network exports the characteristic pattern of target hand；

As a result iteration module, for the Attitude estimation result of the characteristic pattern and last iteration to be input to the first default nerve The decision-making level of network exports the Attitude estimation result of current iteration；

As a result determining module, if for inclined between the Attitude estimation result of current iteration and the Attitude estimation result of last iteration Difference is less than preset threshold, then using the Attitude estimation result of current iteration as the final carriage estimated result of target hand.

9. a kind of electronic equipment characterized by comprising

At least one processor；And

At least one processor being connect with the processor communication, in which:

The memory is stored with the program instruction that can be executed by the processor, and the processor calls described program to instruct energy Enough methods executed as described in claim 1 to 7 is any.

10. a kind of non-transient computer readable storage medium, which is characterized in that the non-transient computer readable storage medium is deposited Computer instruction is stored up, the computer instruction makes the computer execute the method as described in claim 1 to 7 is any.