JP7464512B2 - 3D human posture estimation device, method and program - Google Patents

Description

The present invention relates to a three-dimensional human posture estimation device, method, and program, and in particular to a three-dimensional human posture estimation device, method, and program that estimates the posture of a three-dimensional human by taking into account the amount of twisting rotation of the limbs connecting joint points.

3D human pose estimation technology is a technology that estimates a person's pose in three-dimensional space. In general, the body posture in three-dimensional space is expressed for each major joint by three-dimensional coordinates relative to a reference point, or three-dimensional rotation relative to a reference posture.

3D coordinate-based posture representation is often not suitable for expressing detailed postures; for example, arm twists cannot be expressed using only the 3D coordinates of the elbow and wrist. In contrast, 3D rotation-based posture representation makes it possible to express detailed postures, including limb twists, although it requires a reference body model. There are many prior art technologies for estimating posture parameters from human images.

Non-Patent Document 1 discloses a technology that estimates the two-dimensional coordinates of each joint point from an RGB image of a person, obtains the initial two-dimensional coordinates of each joint point of a body model by projecting the three-dimensional coordinates of each joint point with the initial values of the posture parameters applied onto the image, and obtains the desired posture parameters by optimizing the camera parameters and posture parameters so that the difference between these two two-dimensional coordinates for each joint point is minimized.

Non-Patent Document 2 discloses a technology for estimating detailed posture of the entire body by considering not only the joint points considered in Non-Patent Document 1, but also two-dimensional coordinates of facial key points such as the eyes and nose, and fine joint points of the fingers. Non-Patent Document 2 further aims to improve estimation accuracy by introducing constraints to avoid impossible postures and sudden changes in the time direction when optimizing posture parameters.

Non-Patent Document 3 proposes a regression model that finds posture parameters directly from a rectangular region around a person in an RGB image. In Non-Patent Document 3, not only does the regression model learn from a large number of pairs of person images and posture parameters, but it also acquires the distribution of a large amount of posture parameter data representing actual human movements obtained from a motion sensor or the like as prior knowledge and uses this knowledge for learning.

Non-Patent Document 4 discloses a technology that estimates the two-dimensional coordinates of a person's joint points from an RGB image while efficiently extracting three-dimensional information such as depth to estimate the three-dimensional coordinates of the joint points. In Non-Patent Document 4, posture parameters are optimized so that a body model matches the three-dimensional coordinates of the joint points. Constraints are introduced in the optimization to avoid impossible postures and sudden changes in the time direction.

Patent document 1 discloses a technology that predicts posture parameters of a person from an RGBD image and a body model of the person, evaluates the consistency between the 3D coordinates of the joint points of the body model at that time projected onto a 2D image and the change in the time direction, and obtains the final posture parameter estimation results that reflect the evaluation value.

JP 2007-333690 A

Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., & Black, M. J. (2016). Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image. ECCV, 561-578. Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Os-man, A. A., Tzionas, D., & Black, M. J. (2019). Ex-pressive Body Capture: 3D Hands, Face, and Body From a Single Image. CVPR, 10967-10977. Kanazawa, A., Black, M. J., Jacobs, D. W., & Malik, J. (2018). End-to-end recovery of human shape and pose. CVPR, 7122-7131. Mehta, D., Sotnychenko, O., Mueller, F., Xu, W., Elgha-rib, M., Fua, P., Seidel, H.-P., Rhodin, H., Pons-Moll, G., & Theobalt, C. (2020). XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Cam-era. TOG, 39(4), 82:1-82:17.

In Non-Patent Document 1, only two-dimensional information of joint points is extracted from RGB images, and this is combined with prior knowledge of the human body shape held by the body model to interpolate depth information and estimate three-dimensional posture, so the depth information remains ambiguous and it is difficult to correctly estimate detailed postures, including twisting of the limbs.

In Non-Patent Document 2, the ambiguity of depth information is reduced by using two-dimensional information of more joint points and key points on the body, as well as prior knowledge of possible postures of the human body, enabling more accurate estimation of detailed postures, including twisting of the limbs. However, the increase in reference information increases the amount of calculation required for optimization processing.

In Non-Patent Document 3, posture parameters are estimated directly from RGB images of a person, but because posture parameters are constructed by three-dimensional rotation, they take discontinuous and complex values. This makes regression learning difficult and accuracy tends to be relatively low. In addition, because the regression learning mainly aims to minimize errors in the three-dimensional coordinates of joint points, it is difficult to correctly estimate detailed postures, including limb twists.

In Non-Patent Document 4, when pose parameters are calculated by fitting a body model to the 3D coordinates of a person's joint points estimated from an RGB image, restrictions are placed on the rotation angles of the joints to prevent the body model from assuming an impossible pose, but it is difficult to estimate the twisting of the limbs.

Patent document 1 focuses on the two-dimensional and three-dimensional coordinates of joint points and their changes over time from RGBD images, but does not take into account detailed postures, including twisting of limbs.

As such, most of the prior art techniques are limited to estimating the positions of major joint points correctly and under the constraint of not resulting in impossible postures, and it is difficult to correctly estimate postures including limb twists. As a result, sufficient accuracy cannot be achieved in posture estimation in scenes where limb twists are important, such as gestures and sports forms. In addition, Non-Patent Document 2 also has the problem of a large amount of calculation.

The object of the present invention is to provide a 3D human posture estimation device, method, and program that can solve the above technical problems and accurately estimate a person's posture, including the twisting of the limbs, with a small amount of calculations.

To achieve the above object, the present invention provides a 3D human pose estimation device that estimates the 3D pose of a subject human, and is characterized by having the following configuration.

(1) The device includes a means for estimating three-dimensional coordinates for each joint point based on an image of the subject person, a means for estimating the amount of twisting rotation of the limbs for each joint point based on the image of the subject person, and a means for estimating posture parameters of the subject person based on the estimated three-dimensional coordinates and amount of twisting rotation.

(2) The method for estimating the amount of twisting rotation is to apply the image of the subject person to a regression model that has learned the relationship between an arbitrary person image and the amount of twisting rotation of its limbs, and estimate the amount of twisting rotation of each limb.

(3) The means for estimating the amount of twist rotation is to estimate the amount of twist rotation of each limb by applying the estimated results of the image of the subject person and the three-dimensional coordinates of each joint point, or a heat map of each joint point generated based on the estimated results, to a regression model that has learned the relationship between an arbitrary human image and the amount of twist rotation of the limbs at each joint point.

(1) The limbs of an actual person whose posture is to be estimated may be twisted. In this invention, the three-dimensional posture of the subject person is estimated by taking into account the amount of twisting rotation of the limbs, which makes it possible to accurately estimate the posture of a person whose limbs are twisted.

(2) A regression model is prepared in advance by learning the relationship between any human image and the amount of twisting rotation of the limbs. The amount of twisting rotation of the limbs is estimated by applying the subject image to the regression model, making it possible to estimate posture taking into account the amount of twisting rotation of the limbs with a small amount of calculation.

(3) The amount of twist rotation for each joint point is estimated using the 3D coordinates of each joint point estimated based on the subject image or a heat map of the joint points generated based on the estimation results, so the search range for each joint point can be limited. This enables more accurate pose estimation with less computational effort.

1 is a functional block diagram showing the configuration of a main part of a 3D human pose estimation device according to a first embodiment of the present invention. FIG. 2 is a diagram showing an example of a body model. FIG. 2 is a diagram showing the relationship between joint points and limbs. FIG. 2 is a diagram showing the relationship between an object's joint points, swing rotation, twist rotation, and twist rotation axis. 1 is a flowchart showing a procedure for estimating a three-dimensional human posture. FIG. 11 is a functional block diagram showing the configuration of the main parts of a 3D human pose estimation device according to a second embodiment of the present invention. FIG. 11 is a functional block diagram showing the configuration of the main parts of a 3D human pose estimation device according to a third embodiment of the present invention.

Embodiments of the present invention will now be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a 3D human pose estimation device 1 according to a first embodiment of the present invention, and configuration that is not necessary for explaining the present invention is omitted from the illustration. The present invention is directed to pose estimation based on 3D rotation.

Such a 3D human pose estimation device 1 can be configured by implementing applications (programs) that realize each function on at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

The image input unit 10 acquires a camera image that shows a person whose pose is to be estimated as a subject. The camera image may be acquired directly from the camera, or may be acquired by reading a camera image file that has been captured in advance and stored on a hard disk drive or the like. In this embodiment, the camera image is acquired in RGB format.

In the body model storage unit 40, representative joint points of a person are stored as a body model, as shown in FIG. 2, together with definitions of the limbs (ribs) that connect each joint point, and constraint conditions, as shown in FIG. 3.

The joint point three-dimensional coordinate estimation unit 20 estimates the three-dimensional coordinates of each joint point of the subject person from the camera image of the subject. In this embodiment, in accordance with the definition and constraints of each joint point, each joint point is detected, distinguishing between the right wrist and the right elbow, and its three-dimensional coordinates are estimated. Any method can be used for such joint point detection and coordinate estimation, and for example, a method similar to that described in Non-Patent Document 4 can be used.

Alternatively, a method may be used in which a CNN-based trained model is used to detect locations on an image where a specific joint point is likely to be present, and then information regarding the depth of the location may be extracted to estimate the 3D coordinates. The origin of the estimated 3D coordinates may be the center point of the waist, for example. The scale of the 3D coordinates may be set so that the sum of all the ribs connecting each joint point is a constant value.

The twist rotation amount estimation unit 30 estimates the twist rotation amount of each rib from the image area of the subject person before, after, or in parallel with the estimation of the three-dimensional coordinates. The twist rotation amount is a value defined on the body model, and can be calculated from posture parameters (three-dimensional rotation _Rj of each joint point: j is a joint point identifier) also defined on the body model.

In this embodiment, a regression model is prepared that is constructed by learning the relationship between a large number of person images and the amount of twisting rotation of the limbs, and the acquired image of the subject person is applied to the regression model to estimate the amount of twisting rotation of each limb for each joint point.

The three-dimensional rotation _Rj of each joint point is a value for rotating the child joint point and the rib between the joint point and the target joint point as the center according to the tree structure of each joint point of the body model, as shown in Figure 4. The twist rotation _Tj is an element of the three-dimensional rotation of the joint point that has the rib as the rotation axis, and the other rotation elements are called swing rotation _Sj . The twist rotation _Tj , swing rotation _Sj , and twist rotation angle are derived as follows.

That is, the three-dimensional rotation _Rj of each joint point _Jj can be expressed by the following equation (1) by using quaternion representation as a rotation representation for the swing rotation _Sj and twist rotation _Tj obtained by decomposing it.

If the direction vector from each joint point _Jj of a body model taking a certain posture (usually all posture parameters are 0) to its child joint point is _vj , the result _wj of rotating this by a three-dimensional rotation _Rj is equal to the result of rotating it by a swing rotation _Sj as shown in the following equation (2).

From this, as shown in the following formula (3), the twist rotation Sj can be defined as a rotation with the axis being the perpendicular vector of the plane formed by the direction vectors _vj and _wj , and the angle formed by the direction vectors _{vj and wj} being the rotation angle. Here, Φ represents the conversion formula from the rotation vector representation to the quaternion representation. The twist rotation _Tj can be calculated from the following formula (4).

Here, the twist rotation angle θ _j is obtained by converting the twist rotation T _j from a quaternion representation to a rotation vector representation (ψ) using the following equation (5).

In addition, since the twist rotation angle _θj is expected to be limited to a certain range due to the structure of the human body, it can be defined as a continuous value in the range from 0 to 1 by normalizing it within that range. In this embodiment, this is called the twist rotation amount.

The amount of twist rotation can be expected to be estimated from the appearance of the ribs and joint points. For example, the amount of twist rotation of all ribs can be estimated directly from the image region of a person using a CNN-based trained regression model.

The posture parameter estimation unit 50 uses the estimated results of the three-dimensional coordinates of each joint point output by the joint point three-dimensional coordinate estimation unit 20, the estimated results of the twist rotation amount of each rib output by the twist rotation amount estimation unit 30, and the body model stored in the body model storage unit 40 to estimate the posture parameters of the subject person by deforming the body model so as to maximize the similarity between the body model and the subject person using a model fitting-based optimization method.

In this embodiment, the posture parameters of the subject person are estimated by repeatedly updating the posture parameters of the body model so as to minimize a loss function consisting of the difference between the three-dimensional coordinates of each joint point of the body model and the amount of twist rotation of the limbs, and the three-dimensional coordinates estimated by the joint point three-dimensional coordinate estimation unit 20 and the amount of twist rotation estimated by the twist rotation amount estimation unit 30. Any nonlinear optimization method can be used for the optimization, such as the gradient descent method or the Levenberg-Marquardt method.

The processes from image input to posture parameter estimation are repeated until image input stops. If multiple people are extracted from an image, the processes from joint point 3D coordinate estimation to posture parameter estimation are performed in parallel. The posture parameters obtained can be output in any format, such as a file or by drawing the figure applied to a body model.

Figure 5 is a flowchart showing the procedure for estimating three-dimensional human pose according to this embodiment. In step S1, the image input unit 10 acquires camera images at a predetermined cycle, for example, frame by frame. In step S2, a human region is extracted from the acquired camera images. In step S3, the joint point three-dimensional coordinate estimation unit 20 detects each joint point for each human region, and further estimates three-dimensional coordinates for each joint point.

Before, after, or in parallel with this, in step S4, the twist rotation amount estimation unit 30 estimates the twist rotation amount of each limb of the subject person by applying the image of each person area to a regression model that has been constructed in advance. In this embodiment, the regression model is constructed in advance by learning a large amount of relationships between any person image and the twist rotation amount of the limbs.

In step S5, the posture parameter estimation unit 50 calculates the three-dimensional coordinates of each joint point constituting the body model and the amount of twist rotation of each limb based on the posture parameters reflecting the current posture of the body model.

In step S6, a loss function f(Δd) is calculated, which is the difference Δd between the three-dimensional coordinates of each joint point and the amount of twisting rotation of each limb of the body model and the three-dimensional coordinates of each joint point and the amount of twisting rotation of each limb estimated by the joint point three-dimensional coordinate estimation unit 20 and the twisting rotation amount estimation unit 30, respectively.

In step S7, it is determined whether the loss function f(Δd) is a sufficiently small value that satisfies a specified convergence condition. If the value does not satisfy the convergence condition, the process proceeds to step S8, where the posture parameters of the body model are updated (posture deformation) and the process returns to step S5. After that, each of the above processes is repeated until it is determined in step S7 that the loss function f(Δd) satisfies the specified convergence condition.

When the loss function f(Δd) satisfies a predetermined convergence condition, the process proceeds to step S9, where the current posture parameters of the body model are estimated to be the posture parameters of the subject person. In step S10, it is determined whether there is a next image, and the process returns to step S1 and the above processes are repeated until posture estimation is completed for all images.

Thus, the limbs of an actual person who is the subject of posture estimation may be twisted, and according to this embodiment, the three-dimensional posture of the subject person is estimated taking into account the amount of twisting rotation of the limbs, making it possible to accurately estimate the posture of a person whose limbs are particularly twisted.

In addition, according to this embodiment, a regression model is prepared in advance that is constructed by learning the relationship between an arbitrary person image and the amount of twisting rotation of the limbs, and the amount of twisting rotation of the limbs is estimated by applying the subject image to the regression model, making it possible to estimate posture taking into account the amount of twisting rotation of the limbs with a small amount of calculation.

Figure 6 is a functional block diagram showing the configuration of the main parts of a 3D human pose estimation device 1 according to a second embodiment of the present invention. The same reference symbols as those in the first embodiment represent the same or equivalent parts, and therefore their explanation will be omitted.

In the first embodiment, the twist rotation amount estimation unit 30 was described as directly estimating the twist rotation amount of each limb of the subject person from only the camera image. However, since each joint point of the person has a predetermined positional relationship, either relative or absolute, if the position of each joint point can be limited in advance on the camera image, it is expected that the estimation accuracy of the twist rotation amount of each limb can be improved.

In this embodiment, therefore, the joint point three-dimensional coordinate estimation unit 20 is provided with a heat map generation unit 21 that generates a heat map of joint point coordinates based on the estimation results of the three-dimensional coordinates of each joint point, and outputs the heat map to the twist rotation amount estimation unit 30. In the heat map, the position of each joint point can be expressed as a two-dimensional normal distribution centered on the estimated position. Note that the heat map may also be generated based on two-dimensional information obtained in the process of estimating the three-dimensional coordinates.

In addition, in this embodiment, when constructing a regression model in advance, the regression model is constructed by adding the coordinate position of each joint point to the relationship between the person image and the amount of twist rotation of the limbs for each joint. Then, during estimation, the estimated position of each joint point based on the subject image and the heat map is applied to the regression model to estimate the amount of twist rotation of each limb for each joint point.

According to this embodiment, the amount of twist rotation for each joint point is estimated using a heat map of the joint points generated based on the subject image, so the search range for each joint point can be limited. This enables more accurate pose estimation with less computational effort.

In the second embodiment, the estimated position of each joint point is provided to the twist rotation amount estimation unit 30 in the form of a heat map. However, the present invention is not limited to this. As shown in FIG. 7, the estimated results of the three-dimensional coordinates of each joint point output by the joint point three-dimensional coordinate estimation unit 20 may be provided to the rotation amount estimation unit 30 instead of the heat map.

Furthermore, in each of the above embodiments, the posture parameter estimation unit 50 has been described as estimating the posture parameters of the photographed person using a fitting-based optimization method. However, the present invention is not limited to this, and the posture parameters may be estimated from the estimation results of the three-dimensional coordinates and the amount of twist rotation using an arbitrary regression model that has been trained in advance, for example, on a neural network basis, to determine the relationship between the three-dimensional coordinates and amount of twist rotation of each joint point and each posture parameter.

10...Image input unit, 20...Joint point 3D coordinate estimation unit, 30...Twist rotation amount estimation unit, 31...Heat map generation unit, 40...Body model storage unit, 50...Posture parameter estimation unit

Claims (12)

A three-dimensional human posture estimation device for estimating a three-dimensional posture of a subject person, comprising:
A means for estimating three-dimensional coordinates for each joint point based on an image of a subject person;
A means for estimating a twist rotation amount of a limb for each joint point based on an image of a subject person;
and estimating a posture parameter of the photographed person based on the estimated three-dimensional coordinates and the estimated amount of twist and rotation.
The pose parameter estimation means estimates the pose parameters of the subject person by applying the estimated results of the three-dimensional coordinates and the amount of twist rotation to a regression model that has learned the relationship between the three-dimensional coordinates of each joint point of an arbitrary human image and the amount of twist rotation of the limbs and the pose parameters . A three-dimensional human posture estimation device for estimating a three-dimensional posture of a subject person, comprising:
A means for estimating three-dimensional coordinates for each joint point based on an image of a subject person;
A means for estimating a twist rotation amount of a limb for each joint point based on an image of a subject person;
and estimating a posture parameter of the photographed person based on the estimated three-dimensional coordinates and the estimated amount of twist and rotation.
The means for estimating the amount of twist rotation estimates the amount of twist rotation of each of the four limbs by applying the image of the subject person to a regression model that has learned the relationship between an arbitrary person image and the amount of twist rotation of the limbs. A three-dimensional human posture estimation device for estimating a three-dimensional posture of a subject person, comprising:
A means for estimating three-dimensional coordinates for each joint point based on an image of a subject person;
A means for estimating a twist rotation amount of a limb for each joint point based on an image of a subject person;
and estimating a posture parameter of the photographed person based on the estimated three-dimensional coordinates and the estimated amount of twist and rotation.
a means for generating a heat map of joint points based on the three-dimensional coordinates estimated by the means for estimating the three-dimensional coordinates or two-dimensional information obtained in the process of estimating the three-dimensional coordinates;
The three-dimensional human pose estimation device is characterized in that the means for estimating the amount of twist rotation estimates the amount of twist rotation of each limb by applying the image of the subject person and a heat map to a regression model that has learned the relationship between an arbitrary human image and the amount of twist rotation of the limbs at each joint point of the human image. A three-dimensional human posture estimation device for estimating a three-dimensional posture of a subject person, comprising:
A means for estimating three-dimensional coordinates for each joint point based on an image of a subject person;
A means for estimating a twist rotation amount of a limb for each joint point based on an image of a subject person;
and estimating a posture parameter of the photographed person based on the estimated three-dimensional coordinates and the estimated amount of twist and rotation.
The means for estimating the amount of twist rotation estimates the amount of twist rotation of each limb by applying the image of the subject person and the estimated results of the three-dimensional coordinates to a regression model that has learned the relationship between an arbitrary human image and the amount of twist rotation of the limbs for each joint point of the human image, thereby estimating the amount of twist rotation of each limb. 5. The 3D human posture estimation device according to claim 2, wherein the posture parameter estimation means estimates the posture parameters of the subject human by deforming the body model so as to minimize differences between the three-dimensional coordinates of each joint point of the body model and the amount of twisting rotation of the limbs and the estimated results of the three-dimensional coordinates and the amount of twisting rotation. 5. The 3D human pose estimation device according to claim 2, wherein the pose parameter estimation means estimates the pose parameters of the subject human by applying each of the estimated results of the 3D coordinates and the twist rotation amount to a regression model which has learned the relationship between the pose parameters and the 3D coordinates of each joint point of an arbitrary human image and the twist rotation amount of each limb. A three-dimensional human posture estimation method for estimating a three-dimensional posture of a subject person by a computer, comprising:
The three-dimensional coordinates of each joint point are estimated based on the image of the subject person;
The amount of twist and rotation of the limbs is estimated for each joint based on the image of the subject.
estimating posture parameters of the photographed person based on the estimated three-dimensional coordinates and the amount of twist and rotation;
A three-dimensional human posture estimation method comprising : applying an image of a subject person to a regression model that has learned the relationship between an arbitrary human image and the amount of twisting rotation of the limbs to estimate the amount of twisting rotation of each limb . A three-dimensional human posture estimation method for estimating a three-dimensional posture of a subject person by a computer, comprising:
The three-dimensional coordinates of each joint point are estimated based on the image of the subject person;
The amount of twist and rotation of the limbs is estimated for each joint based on the image of the subject.
estimating posture parameters of the photographed person based on the estimated three-dimensional coordinates and the amount of twist and rotation;
generating a heat map of the joint points based on the estimation result of the three-dimensional coordinates of the joint points;
A three-dimensional human pose estimation method characterized by estimating the amount of twist rotation of each limb by applying an image and a heat map of the subject person to a regression model that has learned the relationship between an arbitrary human image and the amount of twist rotation of the limbs at each joint point. A three-dimensional human posture estimation method for estimating a three-dimensional posture of a subject person by a computer, comprising:
The three-dimensional coordinates of each joint point are estimated based on the image of the subject person;
The amount of twist and rotation of the limbs is estimated for each joint based on the image of the subject.
estimating posture parameters of the photographed person based on the estimated three-dimensional coordinates and the amount of twist and rotation;
A three-dimensional human posture estimation method, comprising: applying an image of a subject person and the estimated results of the three-dimensional coordinates to a regression model that has learned the relationship between an arbitrary human image and the amount of twisting rotation of each limb at each joint point, thereby estimating the amount of twisting rotation of each limb. In a three-dimensional human posture estimation program for estimating a three-dimensional posture of a subject person,
A step of estimating three-dimensional coordinates for each joint point based on an image of a subject person;
A step of estimating a twist rotation amount of a limb for each joint point based on an image of a subject person;
and estimating posture parameters of the photographed person based on the estimated three-dimensional coordinates and the estimated amount of twist rotation.
The three-dimensional human pose estimation program is characterized in that the step of estimating the amount of twist rotation includes applying an image of a subject person to a regression model that has learned the relationship between an arbitrary human image and the amount of twist rotation of its limbs to estimate the amount of twist rotation of each limb . In a three-dimensional human posture estimation program for estimating a three-dimensional posture of a subject person,
A step of estimating three-dimensional coordinates for each joint point based on an image of a subject person;
A step of estimating a twist rotation amount of a limb for each joint point based on an image of a subject person;
and estimating posture parameters of the photographed person based on the estimated three-dimensional coordinates and the amount of twist rotation.
generating a heat map of joint points based on the estimated three-dimensional coordinates by the means for estimating the three-dimensional coordinates;
The three-dimensional human pose estimation program is characterized in that the step of estimating the amount of twist rotation includes applying an image and a heat map of the subject person to a regression model that has learned the relationship between an arbitrary human image and the amount of twist rotation of each limb at each joint point to estimate the amount of twist rotation of each limb. In a three-dimensional human posture estimation program for estimating a three-dimensional posture of a subject person,
A step of estimating three-dimensional coordinates for each joint point based on an image of a subject person;
A step of estimating a twist rotation amount of a limb for each joint point based on an image of a subject person;
and estimating posture parameters of the photographed person based on the estimated three-dimensional coordinates and the estimated amount of twist rotation.
The step of estimating the amount of twist rotation of each limb includes applying the image of the subject person and the estimated results of the three-dimensional coordinates to a regression model that has learned the relationship between an arbitrary human image and the amount of twist rotation of the limbs at each joint point of the human image, thereby estimating the amount of twist rotation of each limb.

Images