JP7598042B2 - Height estimation method, height estimation device and program - Google Patents

Description

The present invention relates to a height estimation method, a height estimation device, and a program.

Conventionally, there are known methods for estimating the positions of a person's bones and the like in an input image from an input image in which the person is captured. For example, the estimation method described in Non-Patent Document 1 can estimate the position coordinates of the eyes, ears, nose, neck, shoulders, elbows, wrists, hips, knees, and ankles of a person in an input image.

In addition, methods have been known for extracting a rectangle (hereinafter referred to as a "circumscribing rectangle") that circumscribes the contour of a person or object in an input image from an input image in which the person or object is captured. For example, Mask R-CNN (Region-Convolutional Neural Network) described in Non-Patent Document 2 is an object detection algorithm that performs object detection and segmentation, and can extract the position coordinates of the circumscribing rectangle of a person or object in an input image.

By using the above two techniques, it is possible to extract the position coordinates of the skeleton of the person and the circumscribing rectangle of the person and the object shown in Fig. 15 from an input image in which the person and object shown in Fig. 14 are captured. Fig. 14 shows an input image 9 in which a person 91 standing upright and facing forward is captured, and an object 92 existing in the vicinity of the person 91. Fig. 15 shows the skeleton 93 of the person 91 estimated from the input image 9, and the circumscribing rectangle 94 of the person 91 and the circumscribing rectangle 95 of the object 92 extracted from the input image 9.

Zhe Cao, Tomas Simon, Shih-En Wei and Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", Computer Vision and Pattern Recognition (CVPR), pp.7291-7299, 2017. Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick, "Mask R-CNN", The IEEE International Conference on Computer Vision (ICCV), pp.2961-2969, 2017.

When an input image is an image of a person standing upright and an object, if the actual size (height) of the object is known, the actual height of the person can be easily estimated by comparing the height of the person in the input image with the height of the object in the input image. However, the posture of the person in the input image is not necessarily an upright posture. Conventionally, when a person in the input image is not standing upright but is, for example, in a posture in motion, there was a problem that it was not easy to estimate the actual height of the person from the input image even if an object whose actual height was known was captured together with the person.

In view of the above circumstances, the present invention aims to provide a height estimation method, height estimation device, and program that can more accurately estimate a person's height from an input image, even if the person in the input image is not standing upright.

One aspect of the present invention is a height estimation method having an image acquisition step of acquiring an image of a person and an object, a skeleton extraction step of extracting the position of the person's skeleton in the image, a determination step of determining a posture pattern of the person based on the positional relationship of each part of the skeleton, and an estimation step of estimating the actual height of the person based on a correction value obtained by curve approximation or linear approximation that is used depending on the characteristics of the bending of the skeleton corresponding to the pattern, and the height of the person in the image.

One aspect of the present invention is a height estimation device that includes an image acquisition unit that acquires an image of a person and an object, a skeleton extraction unit that extracts the position of the person's skeleton in the image, a determination unit that determines a posture pattern of the person based on the positional relationship of each part of the skeleton, and an estimation unit that estimates the actual height of the person based on a correction value obtained by curve approximation or linear approximation that is used depending on the characteristics of the bending of the skeleton corresponding to the pattern, and the height of the person in the image.

One aspect of the present invention is a program for causing a computer to execute the above height estimation method.

The present invention makes it possible to more accurately estimate a person's height from an input image, even if the person in the input image is not standing upright.

1 is a block diagram showing a functional configuration of a height estimation device 1 according to an embodiment of the present invention. 11A and 11B are diagrams for explaining differences in the Y-axis width of a person frame due to differences in a person's posture. FIG. 13 is a diagram for explaining correction of an estimated height value in the case of pattern 2. FIG. 13 is a diagram for explaining correction of an estimated height value in the case of pattern 3. FIG. 13 is a diagram for explaining correction of an estimated height value in the case of pattern 4. 5 is a flowchart showing an operation of a state determination unit 14 according to the embodiment of the present invention. 11A and 11B are diagrams illustrating an example of the contents of a determination condition and a pattern that is identified when the determination condition is satisfied. 4 is a flowchart showing an operation of a correction unit 15 in the embodiment of the present invention. 11A and 11B are diagrams illustrating an example of a correction process performed by the correction unit 15 for each pattern. 5 is a flowchart showing an operation of a height calculation unit 18 according to the embodiment of the present invention. 11 is a diagram showing an example of a calculation process of an estimated height value by the height calculation unit 18 for each pattern. FIG. 5 is a flowchart showing an operation of an object matching unit 17 in the embodiment of the present invention. FIG. 2 is a diagram showing an example of a configuration of object data 110 according to the embodiment of the present invention. FIG. 2 is a diagram showing an example of an input image in which a person and an object are captured. 11A and 11B are diagrams illustrating an example of the position coordinates of a skeleton of a person extracted from an input image and the position coordinates of circumscribing rectangles of a person and an object.

(Embodiment)
Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

In the following explanation, the circumscribing rectangle of a person in the input image is referred to as the "person frame," and the circumscribing rectangle of an object in the input image is referred to as the "object frame." A circumscribing rectangle is a rectangle that circumscribs the outline of a person or object in the input image.

[Configuration of Height Estimation Device]
The following describes the configuration of the height estimation device 1 according to the embodiment of the present invention. Fig. 1 is a block diagram showing the functional configuration of the height estimation device 1 according to the embodiment of the present invention.

The height estimation device 1 is an information processing device such as a general-purpose computer. As shown in FIG. 1, the height estimation device 1 includes a control unit 10, a memory unit 11, an image input unit 12, a skeleton extraction unit 13, a state determination unit 14, a correction unit 15, a person/object extraction unit 16, an object matching unit 17, a height calculation unit 18, and an output unit 19.

The control unit 10 is configured to include a processor such as a CPU (Central Processing Unit). The storage unit 11 is configured to include a storage medium such as a RAM (Random Access Memory), a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), and a HDD (Hard Disk Drive), or any combination of these storage media.

The control unit 10 reads and executes a computer program recorded in a storage medium such as the storage unit 11, thereby causing the height estimation device 1 to function as a device having each of the functional units described above. The control unit 10 controls the operation of each of the functional units described above.

The memory unit 11 pre-stores object data 110. Details of the object data 110 will be described later.

The image input unit 12 is an input interface that accepts input of an input image from an external device or storage medium. The input image input to the image input unit 12 is an image of a person and an object existing in the vicinity of the person. The image input unit 12 outputs the input image to the skeleton extraction unit 13 and the person/object extraction unit 16.

The skeleton extraction unit 13 acquires the input image output from the image input unit 12. The skeleton extraction unit 13 extracts the image area of the skeleton of the person appearing in the input image by image analysis, and identifies the position of each skeleton in the input image. Note that the image analysis method used here may be the method described in Non-Patent Document 1. The skeleton extraction unit 13 outputs information indicating the type of skeleton (hereinafter referred to as "skeleton information") and information indicating the XY coordinates of each skeleton in the input image to the state determination unit 14.

Note that, here, all parts of the human body, including the eyes and ears, are collectively referred to as "skeleton." In this embodiment, the following will be described as examples of skeletons to be extracted: both eyes, both ears, nose, both shoulders, both hips, both knees, and both ankles.

The state determination unit 14 acquires the skeletal information and information indicating the XY coordinates of each skeleton output from the skeleton extraction unit 13. The state determination unit 14 identifies a pattern to which the posture of the person in the input image corresponds based on the skeletal information, the XY coordinates of each skeleton, and a predetermined threshold value. Details of the threshold value will be described later. The state determination unit 14 outputs information indicating the identified pattern and information indicating the skeletal information and the XY coordinates of each skeleton to the correction unit 15.

In this embodiment, the state determination unit 14 identifies a pattern that corresponds to the posture of the person in the input image from among five predetermined patterns, Pattern 1 to Pattern 5. Pattern 1 is a posture in which the person stands upright, Pattern 2 is a posture in which the person spreads both legs to the left and right, Pattern 3 is a posture in which the person spreads both legs forward and backward, Pattern 4 is a posture in which the person bends sideways, and Pattern 5 is a posture that does not correspond to any of Patterns 1 to 4 above.

The correction unit 15 acquires information indicating the pattern, skeletal information, and information indicating the XY coordinates of each skeleton output from the state determination unit 14. The correction unit 15 corrects the value of the length between skeletons based on the length between skeletons calculated from the XY coordinates of each skeleton, the pattern, and a correction formula predetermined for each pattern. Hereinafter, the corrected value of the length between skeletons is referred to as the "correction value." The correction unit 15 outputs the correction value and information indicating the pattern to the height calculation unit 18. Details of the correction formula will be described later.

The person/object extraction unit 16 acquires the input image output from the image input unit 12. The skeleton extraction unit 13 detects image areas in which people and objects appear in the input image by image analysis, and identifies circumscribing rectangles of the detected people and objects. The skeleton extraction unit 13 identifies the XY coordinates of the person frame and the XY coordinates of the object frame from the identified circumscribing rectangle. Note that the image analysis method used here may be the method described in Non-Patent Document 2.

Furthermore, the person/object extraction unit 16 identifies the name of the detected object (hereinafter referred to as the "object name") from the image of the object. Note that the person/object extraction unit 16 identifies the object name of the detected object, for example, by performing pattern matching between the image of the detected object and a sample image of the object to which an object name has been previously associated. The person/object extraction unit 16 outputs information indicating the object name, the XY coordinates of the object frame, and the XY coordinates of the person frame to the object matching unit 17.

The object matching unit 17 acquires information indicating the object name output from the person/object extraction unit 16, the XY coordinates of the object frame, and the XY coordinates of the person frame. The object matching unit 17 refers to object data 110 pre-stored in the memory unit 11. The object data 110 is a collection of data in which the object name is associated with the value of the object's size (height).

The object matching unit 17 detects an object name that matches an object name based on the acquired information from the object data 110, and acquires a height value (hereinafter referred to as "height information") associated with the detected object name. The object matching unit 17 outputs the XY coordinates of the object frame, the height information, and the XY coordinates of the person frame to the height calculation unit 18.

The height calculation unit 18 acquires the correction value and information indicating the pattern output from the correction unit 15. The height calculation unit 18 also acquires the XY coordinates of the object frame, height information, and XY coordinates of the person frame output from the object matching unit 17. The height calculation unit 18 calculates an estimated height value of the person in the input image based on the pattern, the height information, the Y-axis width of the object frame, the Y-axis width of the person frame, the correction value, and a predetermined calculation formula for the estimated height. Details of the calculation formula for the estimated height will be described later.

The height calculation unit 18 outputs information indicating the estimation result to the output unit 19. That is, the height calculation unit 18 outputs information indicating the calculated estimated height value to the output unit 19. Note that, if the height calculation unit 18 is unable to calculate the estimated height value, it does not output information to the output unit 19. Alternatively, if the height calculation unit 18 is unable to calculate the estimated height value, it may output information indicating that estimation is impossible to the output unit 19.

The output unit 19 acquires information indicating the estimation result output from the height calculation unit 18. The output unit 19 outputs the information indicating the estimation result to an external device. The output unit 19 is an output interface that outputs the information indicating the estimation result. Note that the output unit 19 may be a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. In this case, the output unit 19 displays the information indicating the estimation result.

Below, we will explain the height estimation method using the height estimation device 1 of this embodiment using a specific example.

Figure 2 is a diagram to explain the difference in the Y-axis width of the person frame due to the difference in the person's posture. Figure 2 shows the image areas within four types of person frames. Each of the four images is an image cut out from the input image.

Image A10 is an image of a person standing upright, i.e., the aforementioned pattern 1. Image A20 is an image of a person with their legs spread apart, i.e., the aforementioned pattern 2. Image A30 is an image of a person with their legs spread apart, i.e., the aforementioned pattern 3. Image A40 is an image of a person with their body bent sideways, i.e., the aforementioned pattern 4.

As shown in image A10, when a person is standing upright, the positions of the eyes, ears, shoulders, hips, and ankles are horizontal. Note that "horizontal" here means that the inclination of the line connecting these two parts is within a specified range.

As shown in image A20, when a person has their legs spread apart, the Y-axis width of the person frame is shorter than in image A10. In other words, when a person has their legs spread apart, it is thought that an error may occur in which the height is calculated to be lower than when the person is standing upright.

As shown in image A30, when a person has their legs spread apart, the Y-axis width of the person frame is slightly shorter than in image A10. In other words, when a person has their legs spread apart, it is thought that an error may occur in which the height is calculated to be slightly lower than when the person is standing upright.

As shown in image A40, when a person is bent sideways, the Y-axis width of the person frame is shorter than in image A10. In other words, when a person is bent sideways, it is thought that an error may occur in which the height is calculated to be lower than when the person is standing upright.

In this embodiment, the height estimation device 1 corrects the error in the estimated height value as described above using the correction unit 15. Below, specific examples of the correction of the estimated height value by the height estimation device 1 will be explained for each pattern. Note that if the person is standing upright, that is, in the case of the above-mentioned pattern 1, correction of the estimated height value is not necessary.

Figure 3 is a diagram for explaining the correction of the estimated height value in the case of pattern 2. Image A20 shown in Figure 3 is the same image as image A20 shown in Figure 2. As shown in image A20, when a person is in a posture with both legs spread apart, the positions of both eyes, both ears, both shoulders, both hips, and both ankles are horizontal.

When the state determination unit 14 identifies the posture of the person in the input image as pattern 2, the correction unit 15 compares the lengths of the right and left legs of the person in image A20. That is, the correction unit 15 calculates the length of the right leg by obtaining the Euclidean distance between the XY coordinates of the right hip and the XY coordinates of the right ankle, calculates the length of the left leg by obtaining the Euclidean distance between the XY coordinates of the left hip and the XY coordinates of the left ankle, and compares the two lengths.

Through the above comparison, the correction unit 15 determines the length of the longer leg. In image A21 of Figure 3, the dashed line indicates that the right leg is longer than the left leg for the person in the input image. However, because pattern 2 is a case in which the person has their legs spread apart, it is likely that in reality the lengths of both legs of the person in the input image are roughly equal in many cases.

The correction unit 15 corrects the estimated height value by correcting the right foot, which is tilted as represented by the dashed line in image A21, to be vertical (i.e., parallel to the Y axis of the person frame) as in image A22.

Figure 4 is a diagram for explaining the correction of the estimated height value in the case of pattern 3. Image A30 shown in Figure 4 is the same image as image A30 shown in Figure 2. As shown in image A30, when a person has their legs spread apart, the eyes, ears, shoulders, and hips are positioned horizontally. Also, as shown in image A30, when a person has their legs spread apart, the front leg (the right leg in image A30) is longer than the back leg (the left leg in image A30), so the ankles are not horizontal.

When the state determination unit 14 identifies the posture of the person in the input image as pattern 3, the correction unit 15 compares the lengths of the right and left legs of the person in image A30. That is, the correction unit 15 calculates the length of the right leg by obtaining the Euclidean distance between the XY coordinates of the right hip and the XY coordinates of the right ankle, calculates the length of the left leg by obtaining the Euclidean distance between the XY coordinates of the left hip and the XY coordinates of the left ankle, and compares the two lengths.

The correction unit 15 uses the above comparison to determine the length of the longer leg. In image A31 of Figure 4, the dashed line indicates that the right leg of the person in the input image is longer than the left leg.

The correction unit 15 corrects the estimated height by applying the length of the longer right leg as shown by the dashed line in image A31 to the left leg as shown in image A32. Note that in this embodiment, the length of the longer leg is used to estimate height, but this is not limited to this, and the average value of the lengths of both legs may be used to estimate height.

Figure 5 is a diagram for explaining the correction of the estimated height value in the case of pattern 4. Image A40 shown in Figure 5 is the same image as image A40 shown in Figure 2. As shown in image A40, when a person is bending their body sideways, the positions of the eyes, ears, shoulders, and hips are not horizontal. Also, as shown in image A40, when a person is bending their body sideways, the positions of the ankles are horizontal.

When the state determination unit 14 identifies the posture of the person in the input image as pattern 4, the correction unit 15 compares the lengths of the right leg and the left leg of the person in image A40. That is, the correction unit 15 identifies the length from the position of the nose to the position of the center between the waist, passing through the position below the neck (i.e., the center position between the shoulders), using a quadratic curve approximation.

In this embodiment, a configuration in which quadratic curve approximation is used is described, but the present invention is not limited to this configuration. For example, other approximation methods, such as cubic curve approximation, may be used.

The correction unit 15 corrects the estimated height value by correcting the quadratic curve represented by the dotted curve in image A41 to become a vertical straight line (i.e., parallel to the Y axis of the person frame) represented by the dotted line in image A42 without changing the length.

In addition, the correction unit 15 calculates the length of the right leg by calculating the Euclidean distance between the XY coordinates of the right hip and the XY coordinates of the right ankle, and calculates the length of the left leg by calculating the Euclidean distance between the XY coordinates of the left hip and the XY coordinates of the left ankle, and compares the lengths of the two. The correction unit 15 identifies the length of the longer leg through the above comparison. In image A41 in Figure 5, the dashed line indicates that the left leg is longer than the right leg for the person in the input image.

The correction unit 15 corrects the estimated height value by correcting the length of the longer left leg, as represented by the dashed line in image A41, so that it also applies to the left leg, as shown in image A32.

[Status Determination Process]
The condition determination process by the state determination unit 14 will be described below. In this embodiment, the state determination unit 14 defines the following six thresholds and six coefficients, and performs condition determination of the posture of the person in the input image by six magnitude determinations based on these thresholds and coefficients. The values of each threshold and each coefficient are calculated based on the length between each skeleton.

The thresholds A1 to A6 are defined as follows: The thresholds A1 to A6 are numerical values used to determine the positional relationship of each part of the skeleton in the input image, as follows:

Threshold A1: The value of the inclination between the eyes (or between the ears).
Threshold A2: The value of the slope between the shoulders.
Threshold A3: The value of the inclination between both hips.
Threshold A4: The value of the inclination between the ankles.
Threshold A5: The length from the nose to the center between the ankles divided by the length between the ankles.
Threshold A6: A value obtained by dividing the difference between (the sum of the length between the right hip and right knee and the length between the right knee and right ankle) and (the sum of the length between the left hip and left knee and the length between the left knee and left ankle) by the sum of (the sum of the length between the right hip and right knee and the length between the right knee and right ankle) and (the sum of the length between the left hip and left knee and the length between the left knee and left ankle).

The coefficients K1 to K6 are defined as follows: The coefficients K1 to K6 are numerical values based on the length between each part of the human skeleton in the input image, and are used to compare the magnitude with the above thresholds A1 to A6, respectively.

Coefficient K1={(right eye Y-axis value−left eye Y-axis value)/(right eye X-axis value−left eye X-axis value)+(right ear Y-axis value−left ear Y-axis value)/(right ear X-axis value−left ear X-axis value)}/2
Coefficient K2 = (Right shoulder Y-axis value - Left shoulder Y-axis value) / (Right shoulder X-axis value - Left shoulder X-axis value)
Coefficient K3 = (right hip Y-axis value - left hip Y-axis value) / (right hip X-axis value - left hip X-axis value)
Coefficient K4 = (Y-axis value of right ankle - Y-axis value of left ankle) / (X-axis value of right ankle - X-axis value of left ankle)

Coefficient K5 = (length between both ankles) / (length from nose to center of both ankles)
Here, the value of the length between the ankles and the value of the length from the nose to the center of the ankles are calculated as follows.
(Length of both ankles) = √{(Y-axis value of right ankle - Y-axis value of left ankle) ² + (X-axis value of right ankle - X-axis value of left ankle) ² }
(Length from nose to center of both ankles) = √[{Y-axis value of nose - (Y-axis value of right ankle + Y-axis value of left ankle) / 2} ² + {X-axis value of nose - (X-axis value of right ankle + X-axis value of left ankle) / 2} ² ]

Coefficient K6 = {(Total of the length from the right hip to the right knee and the length from the right knee to the right ankle) - (Total of the length from the left hip to the left knee and the length from the left knee to the left ankle)} / {(Total of the length from the right hip to the right knee and the length from the right knee to the right ankle) + (Total of the length from the left hip to the left knee and the length from the left knee to the left ankle)}
Here, the values of the length from the right hip to the right knee, the length from the right knee to the right ankle, the length from the left hip to the left knee, and the length from the left knee to the left ankle are calculated as follows.
Length from right hip to right knee = √{(right hip Y-axis value - right knee Y-axis value) ² + (right hip X-axis value - right knee X-axis value) ² }
Length from right knee to right ankle = √{(Y-axis value of right knee - Y-axis value of right ankle) ² + (X-axis value of right knee - X-axis value of right ankle) ² }
Length from left hip to left knee = √{(left hip Y-axis value - left knee Y-axis value) ² + (left hip X-axis value - left knee X-axis value) ² }
Length from left knee to left ankle = √{(Y-axis value of left knee - Y-axis value of left ankle) ² + (X-axis value of left knee - X-axis value of left ankle) ² }

The state determination unit 14 performs the following threshold determination using the thresholds A1 to A6 and coefficients K1 to K6 calculated based on the above definitions. The following threshold determinations 1 to 6 are determination formulas that determine the posture pattern of a person in the input image by comparing the magnitude of the above thresholds A1 to A6 with the above coefficients K1 to K6.

Threshold decision 1: Whether or not (absolute value of coefficient K1)≦threshold A1.
Threshold decision 2: Whether or not (absolute value of coefficient K2)≦threshold A2.
Threshold decision 3: Whether or not (absolute value of coefficient K3)≦threshold A3.
Threshold decision 4: Whether (absolute value of coefficient K4)≦threshold A4 or not.
Threshold decision 5: Whether or not (absolute value of coefficient K5)≦threshold A5.
Threshold decision 6: Whether or not (the absolute value of coefficient K6) is less than or equal to threshold A6.

[Operation of state determination unit]
In this embodiment, the state determination unit 14 determines the posture pattern of the person in the input image using the results of the six threshold determinations and the four determination conditions. The pattern determination process by the state determination unit 14 will be described below. Fig. 6 is a flowchart showing the operation of the state determination unit 14 in this embodiment of the present invention.

First, the state determination unit 14 acquires skeleton information and information indicating the XY coordinates of each skeleton output from the skeleton extraction unit 13 (step S1401).

Next, the state determination unit 14 calculates the aforementioned threshold values A1 to A6 and coefficients K1 to K6 using the skeleton information and information indicating the XY coordinates of each skeleton. The state determination unit 14 performs the aforementioned threshold determinations 1 to 6 using the calculated values of the threshold values A1 to A6 and coefficients K1 to K6.

The state determination unit 14 determines whether or not determination condition 1 is satisfied based on the results of threshold determinations 1 to 6 (step S1402). Here, determination condition 1 refers to the condition that all threshold determinations 1 to 6 are satisfied. If it is determined that determination condition 1 is satisfied (step S1402: YES), the state determination unit 14 identifies the posture of the person in the input image as pattern 1 (i.e., the person is standing upright) (step S1403).

If it is determined that judgment condition 1 is not satisfied (step S1402, NO), the state judgment unit 14 judges whether or not judgment condition 2 is satisfied based on the judgment results of threshold judgments 1 to 6 (step S1404). Here, judgment condition 2 refers to a condition in which threshold judgments 1 to 4 and 6 are satisfied and threshold judgment 5 is not satisfied. If it is determined that judgment condition 2 is satisfied (step S1404, YES), the state judgment unit 14 identifies the posture of the person in the input image as pattern 2 (i.e., the posture in which the person has both legs spread apart) (step S1405).

If it is determined that judgment condition 2 is not satisfied (step S1404, NO), the state judgment unit 14 judges whether or not judgment condition 3 is satisfied based on the judgment results of threshold judgments 1 to 6 (step S1406). Here, judgment condition 3 refers to a condition in which threshold judgments 1 to 3 and 5 are satisfied, and threshold judgments 4 and 6 are not satisfied. If it is determined that judgment condition 3 is satisfied (step S1406, YES), the state judgment unit 14 identifies the posture of the person in the input image as pattern 3 (i.e., a posture in which the person has both legs spread out in front and behind) (step S1407).

If it is determined that judgment condition 3 is not satisfied (step S1406, NO), the state judgment unit 14 judges whether or not judgment condition 4 is satisfied based on the judgment results of threshold judgments 1 to 6 (step S1408). Here, judgment condition 4 refers to the following condition: threshold judgments 3 to 6 are satisfied, and threshold judgments 1 and 2 are not satisfied. If it is determined that judgment condition 4 is satisfied (step S1408, YES), the state judgment unit 14 identifies the posture of the person in the input image as pattern 4 (i.e., the posture of the person bending the body sideways) (step S1409).

If it is determined that judgment condition 4 is not satisfied (step S1408, NO), i.e., if none of judgment conditions 1 to 4 are satisfied, the state determination unit 14 determines that the posture of the person in the input image is pattern 5 (i.e., it does not fall into any of patterns 1 to 4, and the posture cannot be determined) (step S1410).

Next, the state determination unit 14 outputs information indicating the identified pattern, the skeleton information, and information indicating the XY coordinates of each skeleton to the correction unit 15 (step S1411). This completes the operation of the state determination unit 14 shown in FIG. 6.

Figure 7 summarizes the contents of each of the above judgment conditions 1 to 4 and the patterns that are identified when judgment conditions 1 to 4 are met.

[Correction Value Calculation Process]
The calculation process of the correction value by the correction unit 15 will be described below. In this embodiment, the correction unit 15 defines a calculation formula for each pattern shown below and calculates the calculation values B1 to B2. The correction unit 15 uses the calculation values B1 to B2 to calculate the correction value B3, which is the correction value of the estimated height value.

The calculated values B1 to B2 and the correction value B3 for each pattern are defined as follows. Note that the correction unit 15 does not perform correction processing for patterns 1 and 5. This is because in the case of pattern 1, the person's posture is upright, so there is no need to correct the estimated height value. In the case of pattern 5, the person's posture does not fit into any of patterns 1 to 4, making it difficult to identify the posture, and therefore the estimated height value cannot be corrected.

In the case of pattern 2, the calculated values B1 to B2 and the correction value B3 are defined as follows.
Calculated value B1 = (the larger of the Y-axis width values between the nose and each ankle)
Calculated value B2 = (length from nose to bottom of neck) + (length from bottom of neck to center between both waists) + (the greater value of the sum of the length from right waist to right knee and the length from right knee to right ankle on the XY axis, and the sum of the length from left waist to left knee and the length from left knee to left ankle on the XY axis)
Correction value B3=B2/B1

In the case of pattern 3, the calculated values B1 and B2 and the correction value B3 are defined as follows.
Calculated value B1 = (the larger of the Y-axis width values between the nose and each ankle)
Calculated value B2 = (length from nose to bottom of neck) + (length from bottom of neck to center between both waists) + (the greater value of the sum of the length from right waist to right knee and the length from right knee to right ankle on the XY axis, and the sum of the length from left waist to left knee and the length from left knee to left ankle on the XY axis)
Correction value B3=B2/B1

In the case of pattern 4, the calculated values B1 and B2 and the correction value B3 are defined as follows.
Calculated value B1 = (the larger of the Y-axis width values between the nose and each ankle)
Calculated value B2 = (Length of the quadratic curve from the nose to the center between both hips) + (the greater value of the sum of the length from the right hip to the right knee and the length from the right knee to the right ankle on the XY axis and the sum of the length from the left hip to the left knee and the length from the left knee to the left ankle on the XY axis)
Here, the length of the quadratic curve from the nose to the center between the two waists is calculated by calculating the quadratic curve connecting the nose, the neck, and the midpoint between the two waists, as described above (corresponding to the dotted line portion of image A41 in Figure 5).
Correction value B3=B2/B1

In this way, the correction value B3 is obtained by curve approximation or linear approximation, which is used depending on the characteristics of the curvature of the skeleton corresponding to the pattern.
Different approximation methods may be used for the upper body and the lower body. For example, curve approximation may be used to calculate the length of the upper body (here, from the nose to the center between the two waists), and linear approximation may be used to calculate the length of the lower body (here, from the waist to the ankles).

[Operation of correction unit]
The following describes the operation of the correction unit 15. Fig. 8 is a flowchart showing the operation of the correction unit 15 in the embodiment of the present invention.

First, the correction unit 15 acquires information indicating the pattern, skeletal information, and information indicating the XY coordinates of each skeleton output from the state determination unit 14 (step S1501).

Next, the correction unit 15 calculates the above correction value B3 according to the pattern based on the acquired information (step S1502). As described above, the correction unit 15 does not perform correction processing in the case of pattern 1 and pattern 5.

Next, the correction unit 15 outputs the correction value, B3, and information indicating the pattern to the height calculation unit 18 (step S1503). This completes the operation of the correction unit 15 shown in the flowchart of FIG. 8.

Figure 9 is a diagram summarizing the correction processing by the correction unit 15 for each of the above patterns (methods of calculating calculated values B1-2 and correction value B3, etc.).

[Calculation process of estimated height value]
The calculation process of the estimated height value by the height calculation unit 18 will be described below. In this embodiment, a calculation formula for the estimated height value for each pattern shown below is defined, and the estimated height value is calculated. In addition, in the case of patterns 2 to 4, the height calculation unit 18 defines a calculation formula using the above correction value B3 to calculate the calculated value B4, and defines a calculation formula for the estimated height value using the calculated value B4.

The estimated height value for each pattern and the calculated value B4 are defined as follows. Note that the height calculation unit 18 does not perform the calculation process for the estimated height value in the case of pattern 5. This is because in the case of pattern 5, the person's posture does not fit into any of patterns 1 to 4, making it difficult to identify the posture and therefore making it impossible to calculate the estimated height value.

The estimated height value in the case of pattern 1 is defined as follows.
Estimated height value = (height of object) x (width of person frame on Y axis / width of object frame on Y axis)

The calculated value B4 and the estimated height value in the case of pattern 2 are defined as follows.
Calculated value B4 = (Y-axis width of person frame) x (correction value B3 of pattern 2)
Estimated height value = (height of object) x (calculated value B4 of pattern 2 / Y-axis width of object frame)

The calculated value B4 and the estimated height value in the case of pattern 3 are defined as follows.
Calculated value B4 = (Y-axis width of person frame) x (correction value B3 of pattern 3)
Estimated height value = (height of object) x (calculated value B4 of pattern 3 / Y-axis width of object frame)

The calculated value B4 and the estimated height value in the case of pattern 4 are defined as follows.
Calculated value B4 = (Y-axis width of person frame) × (correction value B3 of pattern 4)
Estimated height value = (height of object) x (calculated value B4 of pattern 4 / Y-axis width of object frame)

As mentioned above, in the case of pattern 5, the height calculation unit 18 does not perform the calculation process of the estimated height value and does not output the estimated height value. The case of pattern 5 is, for example, when the position of each skeleton cannot be correctly recognized, such as when the person is sitting or when the body is extremely bent. Note that in the case where there is no object near the person in the input image, when there is an object near the person but the size (height) of the object is unknown, or when there is an object near the person but the object name of the object cannot be identified, the height calculation unit 18 cannot perform the calculation process of the estimated height value and does not output the estimated height value.

[Height calculation unit operation]
The following describes the operation of the height calculation unit 18. Fig. 10 is a flowchart showing the operation of the height calculation unit 18 in the embodiment of the present invention.

First, the height calculation unit 18 acquires the correction value and the information indicating the pattern output from the correction unit 15. The height calculation unit 18 also acquires the XY coordinates of the object frame, the height information, and the XY coordinates of the person frame output from the object matching unit 17 (step S1801).

Next, the height calculation unit 18 calculates the above-mentioned estimated height value according to the pattern based on the acquired information (step S1802). As described above, the correction unit 15 does not perform the calculation process of the estimated height value when the pattern is 5.

Next, the height calculation unit 18 outputs information indicating the estimation result (e.g., the calculated estimated height value) to the output unit 19 (step S1803). As described above, in the case of pattern 5, the height calculation unit 18 does not output information to the output unit 19. Alternatively, in the case of pattern 5, the height calculation unit 18 may output information indicating that estimation is impossible to the output unit 19. This completes the operation of the correction unit 15 shown in the flowchart of FIG. 10.

Figure 11 is a diagram summarizing the calculation process of estimated height values by the height calculation unit 18 for each of the above patterns.

[Operation of Object Matching Unit]
The following describes the operation of the object matching unit 17. Fig. 12 is a flowchart showing the operation of the object matching unit 17 in the embodiment of the present invention.

First, the object matching unit 17 acquires information indicating the object name output from the person/object extraction unit 16, the XY coordinates of the object frame, and the XY coordinates of the person frame (step S1701).

The object matching unit 17 refers to the object data 110 pre-stored in the memory unit 11. The object matching unit 17 matches the object name based on the acquired information with the object name included in the object data 110 (step S1702).

The object matching unit 17 obtains the height value (height information) of the object detected by matching and associated with the object name contained in the object data 110 (step S1703).

The object matching unit 17 outputs the XY coordinates of the object frame, the height information, and the XY coordinates of the person frame to the height calculation unit 18 (step S1704). This completes the operation of the object matching unit 17 shown in the flowchart of FIG. 12.

Figure 13 is a diagram showing an example of the configuration of object data 110 recorded in memory unit 11. As shown in Figure 13, object data 110 is a collection of data in which object names are associated with values of the object's size (height).

As described above, the height estimation device 1 in this embodiment extracts the position of a person's skeleton from an image in which a person and an object are captured, and determines the person's posture pattern based on the positional relationship of each part of the skeleton. The height estimation device 1 estimates the person's actual height based on a correction value calculated based on a predetermined arithmetic formula for each pattern, the actual height of the object, and the height of the object and the height of the person in the image. With this configuration, the height estimation device 1 in this embodiment can more accurately estimate the height of a person from an input image even if the person in the input image is not standing upright.

According to the above-described embodiment, the height estimation device includes an image acquisition unit, a skeleton extraction unit, a determination unit, and an estimation unit. For example, the image acquisition unit is the image input unit 12 in the embodiment, the determination unit is the state determination unit 14 in the embodiment, and the estimation unit is the height calculation unit 18 in the embodiment. The image acquisition unit acquires an image of a person and an object. The skeleton extraction unit extracts the position of the person's skeleton in the image. The determination unit determines the person's posture pattern based on the positional relationship of each part of the skeleton. The estimation unit estimates the person's actual height based on the height of the person in the image and a correction value obtained by curve approximation or linear approximation that is used depending on the characteristics of the bending of the skeleton corresponding to the pattern.

The height estimation device may further include a height information acquisition unit. For example, the height information acquisition unit is the object matching unit 17 in the embodiment. The height information acquisition unit acquires information indicating the actual height of the object. In this case, the estimation unit estimates the height based on the actual height of the object and the height of the object in the image.

The height estimation device may further include a correction unit that calculates a correction value using the length of the longer of the person's two legs in the image.

The correction unit may also calculate a quadratic curve based on the position of the nose, the position of the neck, and the position of the center of the waist of a person in the image, and calculate a correction value using the length of the quadratic curve from the position of the nose to the position of the center of the waist.

The determination unit may determine that the person's posture is one of an upright posture pattern, a posture pattern with legs spread apart, a posture pattern with legs spread apart, a posture pattern with the body bent sideways, or another posture pattern.

The height estimation device may further include an identification unit. For example, the identification unit is the object matching unit 17 in the embodiment. The identification unit detects an object appearing in the image and identifies the name of the object. In this case, the height information acquisition unit acquires the height of the object appearing in the image based on object information in which the name of the object is associated with information indicating the height of the object, and the name of the object appearing in the image. For example, the object information is object data 110 in the embodiment.

The height estimation device 1 in each of the above-mentioned embodiments may be realized in part or in whole by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment and also includes designs that do not deviate from the gist of the present invention.

1...height estimation device, 10...control unit, 11...memory unit, 12...image input unit, 13...skeleton extraction unit, 14...state determination unit, 15...correction unit, 16...person/object extraction unit, 17...object matching unit, 18...height calculation unit, 19...output unit, 110...object data

Claims (8)

1. A computer implemented method for estimating height, comprising:
An image acquisition step of acquiring an image of a person and an object;
a skeleton extraction step of extracting a position of the skeleton of the person in the image;
a determining step of determining a posture pattern of the person based on at least one of an inclination between the eyes, between the ears, between the shoulders, between the hips, and between the ankles, which are calculated from the positional relationship of each part of the skeleton, a ratio of the length between the ankles to the length from the nose to the center of the ankles, and a ratio of the difference in the lengths of the feet to the sum of the lengths of the feet ;
an estimation step of estimating an actual height of the person based on a correction value obtained by curve approximation or linear approximation, which is used depending on the characteristics of the bending of the skeleton corresponding to the pattern, and a height of the person in the image;
The height estimation method has the following features. acquiring information indicating an actual height of the object;
The height estimation method according to claim 1 , wherein the estimating step estimates the height based on an actual height of the object and a height of the object in the image. The height estimation method according to claim 1 or 2, further comprising: a correction step of calculating the correction value using a length of a longer one of both legs of the person in the image. 4. The height estimation method according to claim 1, further comprising: a correction step of calculating a quadratic curve based on a position of a nose, a position under the neck, and a position of a center of a waist of the person in the image, and calculating the correction value using a length of the quadratic curve from the position of the nose to the position of the center of the waist. 5. The height estimation method according to claim 1, wherein the determining step determines that the person's posture is one of an upright posture pattern, a posture pattern with legs spread apart, a posture pattern with legs spread apart, a posture pattern with the body bent sideways, and other posture patterns. The method further includes a step of detecting the object in the image and identifying the name of the object,
6. The height estimation method according to claim 1, wherein the height information acquisition step acquires a height of an object appearing in the image based on object information in which a name of the object is associated with information indicating a height of the object, and a name of the object appearing in the image. an image acquisition unit that acquires an image of a person and an object;
a skeleton extraction unit that extracts a position of the skeleton of the person in the image;
a determination unit which determines a pattern of the person's posture based on at least one of an inclination between the eyes, between the ears, between the shoulders, between the hips, and between the ankles, which are calculated from the positional relationship of each part of the skeleton, a ratio of the length between the ankles to the length from the nose to the center of the ankles, and a ratio of the difference in the lengths of the feet to the sum of the lengths of the feet ;
an estimation unit that estimates an actual height of the person based on a correction value obtained by curve approximation or linear approximation that is used depending on the characteristics of the bending of the skeleton corresponding to the pattern and the height of the person in the image;
A height estimation device comprising: A program for causing a computer to execute the height estimation method according to any one of claims 1 to 6.

Images