JP2020194454A - Image processing device and image processing method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate attribute information of an object in an image.
An image processing device includes a depth generating means that generates depth information indicating a distance distribution in the depth direction of the subject from an image captured by capturing the subject, and an object that detects a region of a specific object from the captured image. A detection means, a posture estimation means for estimating the posture of the specific object, a posture conversion means for converting the posture of the specific object in the captured image and the depth information, and a captured image and a depth obtained by converting the posture. It has an attribute information estimating means for estimating the attribute information of the specific object from the information and the image shooting conditions.
[Selection diagram] Fig. 1

Description

The present invention relates to a technique for estimating attribute information of an object in an image.

In wildlife observation and livestock growth management, it may be difficult to directly measure the size (volume) and mass of an object, and it is required to acquire animal attribute information in a non-contact manner.

As a method for measuring the shape of an object surface in a non-contact manner, a pattern projection method, a multi-eye imaging method, a TOF (Time of Flight) method, and the like are known. Patent Document 1 has been proposed as a method of estimating a volume from a surface shape measured in a non-contact manner. In Patent Document 1, in addition to surface shape measurement using the optical cutting method as a method for estimating the volume of a crop, the volume is corrected by a correction coefficient after the shadow surface of the crop is set to the ground plane, and the volume is approximately estimated. A method has been proposed.

Animals move with respect to stationary objects such as crops, and it is difficult to control their movements. It is necessary to know the posture of the animal in order to acquire the attribute information such as the size of the animal. Patent Document 2 proposes a method of estimating the posture of a target object by extracting feature points from an image taken by an imaging device and comparing them with feature quantities obtained by learning in advance.

Japanese Patent No. 5686058 Japanese Patent No. 4449410

In Patent Document 1, the back shape of the object to be measured is not measured or estimated, and it is possible to estimate the approximate volume, but the volume cannot be obtained because the three-dimensional shape of the object cannot be obtained. And the estimation accuracy of the mass will decrease.

In wildlife observation and livestock growth management, it is difficult to control the posture of the target object when targeting animals, and when estimating the total length of the target animal from captured images, the estimation results differ depending on the different postures. .. In Patent Document 2, the posture of the target object is estimated by comparison with the feature points learned in advance to recognize the target object, and the target object is recognized. However, the attribute information (dimensions, shape, volume, and size) of the target object are recognized. Mass, etc.) is not estimated.

The present invention has been made in view of the above problems, and an object of the present invention is to realize a technique capable of estimating attribute information of an object in an image.

In order to solve the above problems and achieve the object, the image processing apparatus of the present invention includes a depth generation means for generating depth information indicating a distance distribution in the depth direction of the subject from an captured image of the subject, and the above-mentioned. An object detecting means for detecting a region of a specific object from a captured image, a posture estimating means for estimating the posture of the specific object, and a posture changing means for converting the posture of the specific object in the captured image and the depth information. It also has an attribute information estimating means for estimating the attribute information of the specific object from the captured image in which the posture is changed, the depth information, and the shooting conditions of the image.

According to the present invention, it is possible to estimate the attribute information of an object in an image.

A block diagram (a) showing the apparatus configuration of the first embodiment, a diagram (b) showing the pixel arrangement of the image sensor, and a schematic diagram (c) showing the cross-sectional structure of the image sensor. The schematic diagram explaining the relationship between the image sensor of Embodiment 1, an optical system, and an image. The flowchart which shows the attribute information estimation processing of Embodiment 1. The figure which illustrates the target object selection screen of Embodiment 1. FIG. The figure explaining an example of the posture estimation process of Embodiment 1. FIG. The figure which shows an example of the posture change process and the dimension measurement position of Embodiment 1. FIG. The figure explaining an example of the input method of the dimension measurement position of Embodiment 1. FIG. The flowchart which shows the attribute information estimation processing of Embodiment 2. The flowchart which shows the posture estimation / conversion process of Embodiment 2.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted.

[Embodiment 1] Hereinafter, the first embodiment will be described.
Hereinafter, an example of an embodiment in which the present invention is applied to a digital camera capable of acquiring depth information indicating a distance distribution of a subject as an example of an image processing device will be described. However, the present invention is applied to any device capable of estimating the attribute information of an object (dimension, shape, volume, mass, etc.) based on the captured image, the depth information corresponding to the captured image, and the shooting conditions of the image. It is possible.

<Configuration of Digital Camera> First, the configuration and functions of the digital camera 100 of the present embodiment will be described with reference to FIG.

The image pickup optical system 10 is a photographing lens included in the digital camera 100, and forms an optical image of a subject on the image pickup element 11. The image pickup optical system 10 is composed of a plurality of lenses (not shown) arranged on the optical axis 102, and has an exit pupil 101 at a position separated from the image pickup element 11 by a predetermined distance. In the present specification, the direction parallel to the optical axis 102 is the z direction or the depth direction, the direction orthogonal to the optical axis 102 and parallel to the horizontal direction of the image pickup element 11 is the x direction, and the direction parallel to the image pickup element 11 is the vertical direction. The parallel direction is defined as the y direction, or an axis is provided.

The image pickup device 11 is, for example, a CCD (charge-coupled device) or a CMOS sensor (complementary metal oxide semiconductor). The image sensor 11 photoelectrically converts a subject image formed on the image pickup surface via the image pickup optical system 10 and outputs an image signal related to the subject image. Further, in the present embodiment, the image sensor 11 has a distance measuring function of the imaging surface phase difference ranging method as described later, and in addition to the captured image, the distance from the imaging device to the subject (subject distance) is measured. It is possible to generate and output the indicated distance information.

The control unit 12 is a control device such as a CPU or a microprocessor, and controls the operation of each block included in the digital camera 100. The control unit 12 is, for example, autofocus (AF: automatic focusing) at the time of imaging, change of focus position, change of F value (aperture), image capture, storage unit 14, input unit 15, display unit 16, communication. The unit 17 is controlled.

The image processing device 13 is a block that realizes various image processing possessed by the digital camera 100. As shown in the figure, the image processing device 13 includes an image processing block of an image generation unit 130, a depth generation unit 131, an object detection unit 132, a posture estimation unit 133, a posture conversion unit 134, and an attribute information estimation unit 135, and image processing. It has a memory 136 used as a work area. The image processing device 13 can be configured by using a logic circuit. Further, as another form, it may be composed of a central processing unit (CPU) and a memory for storing an arithmetic processing program.

The image generation unit 130 performs various signal processing such as noise removal, demosaiking, brightness signal conversion, aberration correction, white balance adjustment, and color correction of the image signal output from the image sensor 11. The image data (captured image) output from the image generation unit 130 is stored in the memory 136 and used by the object detection unit 132 and the display unit 16.

The depth generation unit 131 generates a depth image showing the distribution of depth information based on the signal obtained by the distance measuring pixel included in the image sensor 11 described later. Here, the depth image is two-dimensional information in which the value stored in each pixel is the subject distance of the subject existing in the region of the captured image corresponding to the pixel.

The object detection unit 132 detects an object to be measured in advance included in the captured image by using the captured image generated by the image generation unit 130, and specifies the position and size in the captured image. When the type of the object to be measured is not specified in advance, the object detection unit 132 specifies the type. In this embodiment, the target object is an animal other than a human being.

The posture estimation unit 133 learns and acquires the posture of the target object in advance in the object region detected by the object detection unit 132, and estimates using the information stored in the storage unit 14.

The posture conversion unit 134 converts the posture of the target object in the viewing image and the depth image into a specific posture suitable for estimating the attribute information of the target object estimated by the posture estimation unit 133. The specific posture differs depending on the target object, and is determined from the posture information previously stored in the storage unit 14 and / or the memory 136 based on the type of the object specified in advance or the object information specified by the object detection unit 132.

The attribute information estimation unit 135 uses at least one of dimensions, shape, volume, and mass as attribute information of the target object from the viewing image and the depth image in which the posture of the target object is converted into a specific posture by the posture conversion unit 134. presume. In dimensional estimation, the position where dimensions are measured differs depending on the target object. Therefore, estimation is performed using the information for dimension measurement stored in the storage unit 14 and / or the memory 136 in advance based on the type of the object specified in advance or the type of the object specified by the object detection unit 132. In the shape estimation, the surface shape is acquired from the attitude-converted depth image, and the three-dimensional shape of the target object stored in the storage unit 14 and / or the memory 136 in advance according to the type of the object specified by the object detection unit 132. Estimate with reference to the data. In volume estimation, estimation is performed from the three-dimensional shape of the target object obtained by shape estimation and imaging parameters. In mass estimation, estimation is performed using the volume obtained by volume estimation and the density information according to the target object. The density information is measured in advance for each object and stored in the storage unit 14.

The storage unit 14 is a non-volatile recording medium that records captured image data, intermediate data generated in the process of operation of each block, parameters referred to in the operation of the image processing device 13 and the digital camera 100, and the like. is there. The storage unit 14 may be any recording medium that can read and write at high speed and has a large capacity as long as the processing performance allowed for the realization of processing is guaranteed. Flash memory or the like is desirable.

The input unit 15 is a user interface that detects information input and setting change operation input made to the digital camera 100, such as dials, buttons, switches, and touch panels. When the input unit 15 detects the operation input made, it outputs the corresponding control signal to the control unit 12.

The display unit 16 is, for example, a display device such as a liquid crystal display or an organic EL. The display unit 16 is used for confirming the composition at the time of shooting by displaying the captured image through, and for notifying various setting screens and message information. In the present embodiment, the display unit 16 also displays an object detection result, an estimation result such as a shape, a volume, and a mass.

The communication unit 17 is a communication interface included in the digital camera 100 that realizes information transmission / reception with the outside. The communication unit 17 may be configured so that the obtained captured image, depth information, estimation result of subject attribute information (dimensions, shape, volume, mass), and the like can be sent to another device.

<Structure of Image Sensor> Next, the detailed configuration of the image sensor 11 of the present embodiment will be described with reference to FIGS. 1 (b) and 1 (c).

As shown in FIG. 1B, the image pickup device 11 is configured by connecting and arranging a plurality of pixel groups 110 of 2 rows × 2 columns to which different color filters are applied. As shown in the enlarged view, red (R), green (G), and blue (B) color filters are arranged in the pixel group 110, and R, G, and B are arranged from each pixel (photoelectric conversion element). An image signal showing any of the color information of is output. In the present embodiment, as an example, the color filter is described as having a distribution as shown in the figure, but it can be easily understood that the implementation of the present invention is not limited to this.

In the image pickup device 11 of the present embodiment, one pixel (photoelectric conversion element) relates to the horizontal direction of the image pickup device 11 in order to realize the distance measurement function of the image pickup surface phase difference distance measurement method, as shown in FIG. 1 (b). In the I-I'cross section, a plurality of photoelectric conversion units are arranged side by side. More specifically, as shown in FIG. 1 (c), each pixel includes a light guide layer 113 including a microlens 111 and a color filter 112, and a first photoelectric conversion unit 115 and a second photoelectric conversion unit 116. Including, it is composed of.

In the light guide layer 113, the microlens 111 is configured to efficiently guide the light flux incident on the pixel to the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116. Further, the color filter 112 passes light in a predetermined wavelength band, passes only light in any of the wavelength bands R, G, and B described above, and passes through the first photoelectric conversion unit 115 and the first photoelectric conversion unit 115 in the subsequent stage. It leads to the photoelectric conversion unit 116 of 2.

The light receiving layer 114 is provided with two photoelectric conversion units (first photoelectric conversion unit 115 and second photoelectric conversion unit 116) that convert the received light into an analog image signal, and these two photoelectric conversion units are provided. Two types of signals output from are used for distance measurement. That is, each pixel of the image sensor 11 also has two photoelectric conversion units arranged in the horizontal direction, and is an image composed of signals output from the first photoelectric conversion unit 115 of all the pixels. An image signal composed of a signal and a signal output from the second photoelectric conversion unit 116 is used. In other words, the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 partially receive the light flux that enters the pixel through the microlens 111, respectively. Therefore, the two types of image signals finally obtained are a pupil-divided image group related to the light flux passing through different regions of the exit pupil of the imaging optical system 10. Here, the image signal obtained by synthesizing the image signal photoelectrically converted by the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 in each pixel is described in an embodiment in which only one photoelectric conversion unit is provided in the pixel. It is equivalent to an image signal (for viewing) output from one photoelectric conversion unit.

By having such a structure, the image sensor 11 of the present embodiment can output an image signal for viewing and an image signal for distance measurement (two types of pupil-divided images). In the present embodiment, it will be described that all the pixels of the image sensor 11 are provided with two photoelectric conversion units and are configured to be capable of outputting high-density depth information. However, this is the embodiment of the present invention. It is not limited to.

<Distance measurement principle of imaging surface phase difference distance measurement method>
Here, the principle of deriving the subject distance based on the pupil division image group output from the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116, which is performed by the digital camera 100 of the present embodiment, is illustrated. This will be described with reference to 2.

FIG. 2A is a schematic view showing the light flux received by the exit pupil 101 of the image pickup optical system 10 and the first photoelectric conversion unit 115 of the pixels in the image pickup element 11. FIG. 2B is a schematic view showing the luminous flux received by the second photoelectric conversion unit 116 in the same manner.

The microlens 111 shown in FIGS. 2A and 2B is arranged so that the exit pupil 101 and the light receiving layer 114 are in an optically conjugate relationship. The luminous flux that has passed through the exit pupil 101 of the imaging optical system 10 is focused by the microlens 111 and guided to the first photoelectric conversion unit 115 or the second photoelectric conversion unit 116. At this time, as shown in FIGS. 2 (a) and 2 (b), the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 mainly receive light fluxes that have passed through different pupil regions. The first photoelectric conversion unit 115 has a luminous flux that has passed through the first pupil region 210, and the second photoelectric conversion unit 116 has a luminous flux that has passed through the second pupil region 220.

The plurality of first photoelectric conversion units 115 included in the image sensor 11 mainly receive the light flux that has passed through the first pupil region 210, and output the first image signal. At the same time, the plurality of second photoelectric conversion units 116 included in the image sensor 11 mainly receive the light flux passing through the second pupil region 220 and output the second image signal. It is possible to obtain the intensity distribution of the image formed on the image pickup device 11 by the luminous flux passing through the first pupil region 210 from the first image signal. Further, the intensity distribution of the image formed on the image pickup device 11 by the luminous flux passing through the second pupil region 220 from the second image signal can be obtained.

The relative positional deviation amount (so-called parallax amount) between the first image signal and the second image signal is a value corresponding to the defocus amount. The relationship between the parallax amount and the defocus amount will be described with reference to FIGS. 2 (c), (d), and (e). 2 (c), 2 (d), and 2 (e) are schematic views illustrating the image pickup device 11 and the image pickup optical system 10 of the present embodiment. Reference numeral 211 in the figure indicates a first luminous flux passing through the first pupil region 210, and reference numeral 221 indicates a second luminous flux passing through the second pupil region 220.

FIG. 2C shows the state at the time of focusing, and the first luminous flux 211 and the second luminous flux 221 are converged on the image sensor 11. At this time, the amount of parallax between the first image signal formed by the first luminous flux 211 and the second image signal formed by the second luminous flux 221 becomes zero. FIG. 2D shows a state in which the image side is defocused in the negative direction of the z-axis. At this time, the amount of parallax between the first image signal formed by the first light flux and the second image signal formed by the second signal does not become 0 and has a negative value. FIG. 2E shows a state in which the image side is defocused in the positive direction of the z-axis. At this time, the amount of parallax between the first image signal formed by the first light flux and the second image signal formed by the second light flux has a positive value. From the comparison between FIGS. 2 (d) and 2 (e), it can be seen that the directions of the positional deviations are switched according to the positive and negative of the defocus amount. Further, it can be seen that the positional deviation occurs according to the imaging relationship (geometric relationship) of the imaging optical system according to the defocus amount. The amount of parallax, which is the positional deviation between the first image signal and the second image signal, can be detected by a region-based matching method described later.

<Attribute Information Estimating Process> Next, a process of estimating the attribute information of the target object from the captured image executed by the digital camera 100 of the present embodiment will be described with reference to the flowchart of FIG. 3A. The process corresponding to the flowchart of FIG. 3A is digitally executed by the control unit 12 reading, for example, the corresponding processing program stored in the storage unit 14, expanding it into a volatile memory (not shown), and executing the process. This can be achieved by controlling each part of the camera 100. The same applies to FIGS. 8 and 9 described later.

In S301, the control unit 12 selects an object to be measured. A list of objects to be measured is displayed on the display unit 16 at the time of shooting, and the input unit 15 allows the user to select a desired object. As a result, the estimation of the type of the object can be omitted, and erroneous recognition can be prevented. Here, the input unit 15 and the display unit 16 are separately configured, but the display unit 16 may be configured to have the function of the input unit 15 by a touch panel or the like. FIG. 4 shows a display example of a list of objects to be measured. In FIG. 4, animal classifications and detailed animal types for each classification are displayed in a selectable manner, and the user may select one of the displayed options. When the object to be photographed does not correspond to the type of the object prepared in advance, for example, as shown in FIG. 4, the "other" option may be displayed and the attribute information may be estimated using general parameters.

In S302, the control unit 12 processes to perform imaging with imaging settings such as a set focal position, aperture, and exposure time. More specifically, the control unit 12 causes the image pickup device 11 to take an image, transmits the obtained captured image to the image processing device 13, and controls the image to be stored in the memory 136. Here, the captured image is composed of an image signal S1 composed of signals output only from the first photoelectric conversion unit 115 included in the image sensor 11 and a signal output only from the second photoelectric conversion unit 116. It is assumed that there are two types of image signals S2.

In S303, the image processing device 13 generates a viewing image and a depth image from the obtained captured image. More specifically, the image generation unit 130 of the image processing device 13 first generates one Bayer array image by adding the pixel values of each pixel of the image signal S1 and the image signal S2. The image generation unit 130 performs demosing processing of images of each color of R, G, and B on the Bayer array image to generate an image for viewing. It goes without saying that the demosaiking process is performed according to the color filter arranged on the image sensor, and any method may be used for the demosaiking method. In addition, the image generation unit 130 performs processing such as noise removal, luminance signal conversion, aberration correction, white balance adjustment, and color correction to generate a final viewing image and store it in the memory 136.

<Depth image generation processing>
On the other hand, for the depth image, the depth generation unit 131 performs the processing related to the generation. Here, the process related to the depth image generation will be described with reference to the flowchart of FIG. 3 (b).

In S311 the depth generation unit 131 performs light amount correction processing on the image signal S1 and the image signal S2. Due to vignetting at the peripheral angle of view of the imaging optical system 10, the shapes of the first pupil region 210 and the second pupil region 220 are different, and the light amount balance is lost between the image signal S1 and the image signal S2. ing. Therefore, in this step, the depth generation unit 131 corrects the light amount of the image signal S1 and the image signal S2 by using, for example, the light amount correction value previously stored in the storage unit 14 and / or the memory 136.

In S312, the depth generation unit 131 performs a process of reducing noise generated during conversion in the image sensor 11. Specifically, the depth generation unit 131 realizes noise reduction by applying a filter process to the image signal S1 and the image signal S2. In general, the higher the spatial frequency, the lower the SN ratio and the relatively large amount of noise components. Therefore, the depth generation unit 131 performs a process of applying a low-pass filter in which the passing rate decreases as the spatial frequency increases. Since the light amount correction in S311 does not give a desirable result depending on the manufacturing error of the imaging optical system 10, the depth generation unit 131 blocks a DC component and uses a bandpass filter having a low passing rate of a high frequency component. It is desirable to apply.

In S313, the depth generation unit 131 derives the amount of parallax between these images based on the image signal S1 and the image signal S2. Specifically, the depth generation unit 131 sets a point of interest corresponding to the representative pixel information and a collation area centered on the point of interest in the image signal S1. The collation area may be, for example, a rectangular area such as a square area having a predetermined length on one side centered on the point of interest. Next, the depth generation unit 131 sets a reference point in the image signal S2, and sets a reference region centered on the reference point. The reference region has the same size and shape as the collation region described above. The depth generation unit 131 derives the degree of correlation between the image included in the collation area of the image signal S1 and the image included in the reference area of the image signal S2 while sequentially moving the reference points, and the degree of correlation is the highest. A high reference point is specified as a corresponding point corresponding to the point of interest in the image signal S2. The amount of relative positional deviation between the corresponding point and the point of interest identified in this way is the amount of parallax at the point of interest.

The depth generation unit 131 calculates the parallax amount while sequentially changing the points of interest according to the representative pixel information, thereby deriving the parallax amount at a plurality of pixel positions determined by the representative pixel information. For simplicity in this embodiment, the pixel positions (pixel groups included in the representative pixel information) for calculating the parallax amount are set to be the same as the viewing image in order to obtain depth information at the same resolution as the viewing image. It is assumed that it has been done. As a method for deriving the degree of correlation, a method such as NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), or SAD (Sum of Absolute Difference) may be used.

Further, the derived parallax amount can be converted into a defocus amount which is the distance from the image sensor 11 to the focal point of the image pickup optical system 10 by using a predetermined conversion coefficient. Here, assuming that the predetermined conversion coefficient K and the defocus amount are ΔL, the parallax amount d can be converted into the defocus amount by the following equation 1.

(Equation 1)
ΔL = K × d
Further, the defocus amount ΔL can be converted into the subject distance by using the following equation 2 which is the formula of the lens in geometrical optics.
(Equation 2)
1 / A + 1 / B = 1 / F
Here, A is the distance from the object surface to the principal point of the imaging optical system 10 (subject distance), B is the distance from the principal point of the imaging optical system 10 to the image plane, and F is the focal length of the imaging optical system 10. It shall be. That is, in the lens formula, since the value of B can be derived from the defocus amount ΔL, the distance A from the subject to the object surface can be derived based on the setting of the focal length at the time of imaging.

The depth generation unit 131 configures two-dimensional information having the subject distance derived in this way as a pixel value, and stores it in the memory 136 as a depth image.

On the other hand, in S304, the object detection unit 132 detects the target object region. Based on the type of the target object selected in S301, the object detection unit 132 identifies the target object area by using the information learned and acquired in advance and stored in the storage unit 14, and forms the contour of the specified area. The target object is extracted along the line. In this case, it is also possible to assist the extraction of the target object area by using the depth image. A specific or constant value is set except for the extracted target object area, and an object extraction image in which only the target object area is left is generated. Similarly, in the depth image, the objects other than the target object area are replaced with specific or constant values, and an object extraction depth image having a valid value only in the target object area is generated. The object extraction image and the object extraction depth image are stored in the memory 136 and used for the subsequent processing. Various machine learning such as Deep Learning can be used as the learning method for extracting the target object region, but the learning method is not limited to a specific method, and any method may be used.

In S305, the posture estimation unit 133 estimates the posture of the target object in the object extracted image. The posture estimation unit 133 extracts feature points in the region in the object extraction image from the detection result of the target object region in S304, learns in advance, acquires the feature points, and stores the three-dimensional shape in the storage unit 14. The posture is estimated using the feature point data. Furthermore, it is possible to estimate a more detailed attitude change by using the object extraction depth image. In the posture estimation, information such as where the head, torso, legs, and tail of the animal as the target object are located and which direction the head is facing is estimated mainly by using the object extraction image. In addition, the orientation of the entire target object as viewed from the photographing direction can be determined by the orientation of the body. An object extraction depth image is used to determine the orientation of the fuselage. The normal direction of each pixel is calculated from the object extraction depth image, and the normal direction of the body portion is acquired by using the information on the body position. Since the body of an animal is curved, the normal direction is not constant. Therefore, the principal component analysis is performed to calculate the main normal direction. The plane perpendicular to the main normal direction is estimated as a plane representing the direction of the target object. For example, as shown in FIG. 5, a plane Pv that vertically cuts from the head to the tail through the center of the body of the target animal and a plane Ph that cuts horizontally are estimated and used for the posture change described later.

In the present embodiment, the target object area detection and the posture estimation are performed separately, but the target object area detection and the posture estimation of the object may be performed at the same time by using machine learning.

In S306, the posture conversion unit 134 uses the posture of the target object estimated in S305 to perform posture conversion in the object extraction image of the target object and posture conversion in the object extraction depth image. For example, when estimating the size of an animal as attribute information of an object from an object extracted image, the animal has a specific posture that is easy to measure depending on the type. For example, in the case of mammals, the total length La and head and torso length Lb can be obtained by converting the image (FIG. 6 (a)) taken as shown in FIG. 6 into an image taken from the side (FIG. 6 (b)). -It becomes easy to measure dimensions such as body height Lc.

Birds, fish, etc. should also be converted so that they are captured images from the side. In the case of birds, dimensions such as total length La and wingspan Ld are measured as shown in FIG. 6 (c). However, in the case of a bird with its wingspan as shown in FIG. 6D, it is desirable to convert the image so that the image is viewed from above, and the wingspan Le is measured. It is also desirable to have a bird's-eye view of arthropods such as reptiles, amphibians, and insects. However, in any animal, it is desirable to appropriately convert the image from the side surface or the image from above according to the posture of the target object in the captured image.

Ｐは変換前の画像上の位置（ｘ、ｙ、ｚ）を意味し、Ｐ’は変換後の画像上の位置である。変換により欠落した画素位置の情報は周辺の画素の情報から補間することで欠落のない変換画像を生成する。 The posture conversion here basically converts the image by geometric transformation. Taking the case of mammals as an example, the vertical cutting plane Pv indicating the posture of the animal estimated in the above posture estimation is used, and the rotation angle is calculated so that the normal of this plane is perpendicular to the imaging device 1. .. A rotation matrix R is generated from the obtained rotation angle, and the object extraction image and the object extraction depth image are rotationally transformed as shown in Equation 3 below to obtain the posture of the target animal shown in FIG. 6 (a). Convert to the posture as in b).
(Equation 3)

P means a position (x, y, z) on the image before conversion, and P'is a position on the image after conversion. The information on the pixel position missing due to the conversion is interpolated from the information on the surrounding pixels to generate a converted image without any loss.

In this way, the posture-transformed object extraction image and the object extraction depth image are stored in the memory 136 and used for the subsequent processing. In the present embodiment, the posture conversion using the rotation matrix has been described as an example, but it is also possible to use the transformation in which parallel movement, enlargement / reduction, etc. are added in addition to rotation. There is also an advantage that the data to be measured and held in advance for estimating the attribute information described later can be reduced by performing the posture change.

In S307, the attribute information estimation unit 135 estimates the attribute information of the target object. The attribute information represents the size, shape, volume, and mass of the target object, and at least one of these is estimated in the attribute information estimation.

First, dimension estimation, which is one of the attribute information, will be described. In the dimensional estimation, the posture-transformed object extraction image is displayed on the display unit 16, and the user specifies a desired measurement position in the animal in the image displayed by the input unit 15. As a specification method, a method of designating two places (Fig. 7 (a)), specifying three or more places and connecting them linearly (Fig. 7 (b)), or connecting using a polynomial. The method of tracing the part that the user wants to measure is used (FIG. 7 (c)). FIG. 7A shows a case where two points P1 and P2 are designated and the horizontal length between the two points is measured. In addition, it is desirable to be able to specify the case of measuring the length in the vertical direction and the case of measuring the Euclidean distance between two points. FIG. 7B shows an example in which four points P1 to P4 are designated and the points are connected by a straight line. Alternatively, the length may be measured by interpolating between each point using a spline curve or the like. FIG. 7C is an example of measuring the length of the curve traced by the user from P1 to P2 in the figure. In this way, there are various methods for designating measurement positions and methods for specifying designated sections, but while specifying only two points makes it possible to measure only straight lines, simple measurement is possible, but by specifying or tracing three or more points, a curve The length can also be measured, increasing the degree of freedom in measurement. In particular, the measurement using a curve is effective for the measurement when the posture cannot be converted into a desired posture. For example, as shown in FIG. 6E, it is effective for measuring an animal that is difficult to extend in a straight line, such as measuring the total length La of a snake.

The length after designating the measurement point is measured in pixel units or sub-pixel units in the image by counting the number of pixels between the designated measurement positions. The measured value in this case is the length in the image space. In order to convert the measured pixel unit length to the length in the actual object space, first, the size of one pixel size of the image sensor 11 is converted to the length of the International System of Units in the image space. Next, the shooting magnification M is obtained using the shooting parameters, and the actual length in the object space is calculated by taking the product of the measured length in the image space and the shooting magnification M.

The photographing magnification M can be calculated by the following equation 4 using the focal length F of the imaging optical system 10 and the target object distance Z, which are parameters at the time of photographing.
(Equation 4)
M = Z / F
For the target object distance Z, the focus distance corresponding to the position of the focus lens included in the imaging optical system 10 is measured in advance, the focus lens position at the time of shooting is detected, and the corresponding focus distance is acquired as the target object distance Z. To do.

Next, shape estimation, which is one of the attribute information, will be described. The shape estimation estimates the three-dimensional shape of the target object. This is performed using the information on the type of the object selected in S301, the posture-transformed object extraction image and the object extraction depth image generated in S306. Since the posture-converted object extraction depth image has a value that depends on the distance from the digital camera 100 to the target object, the depth image of the target object surface seen from the digital camera 100 is obtained by subtracting the distance to the target object. (= Shape) is calculated. In the following description, it is assumed that a specific surface of the measured target object is the surface. Only a specific one-sided shape of the target object can be measured in one shooting, and the opposite side that cannot be seen from the shooting direction cannot be measured. In estimating the opposite surface of the target object, the three-dimensional shapes of a plurality of objects to be measured are measured in advance, and the reference three-dimensional shapes are stored in the storage unit 14 and / or the memory 136. The reference three-dimensional shape to be stored may be one average three-dimensional shape for each object, but it is desirable to retain a plurality of three-dimensional shapes in order to improve the estimation accuracy of the opposite surface. The reference three-dimensional shape may be either voxel unit data or data expressed in the International System of Units, but the following conversion process is changed depending on the reference three-dimensional shape data unit. Here, voxel means a three-dimensional pixel size in which one pixel is extended in the xyz direction. In addition, the data of the International System of Units means that the size of the target object on the object side is measured by the International System of Units. In the calculated surface shape information of the target object, the depth information (Z direction) is in the International System of Units, but the size of the target object in the XY direction is in pixel units. The depth information is changed to voxel units according to the data unit of the reference three-dimensional shape, or the size of the target object in the XY direction is changed to the international system of units. In the conversion of the voxel unit and the international system of units, the photographing magnification M is obtained by using the photographing parameter as described above, and the conversion is performed by using the equation 4.

Next, the reference three-dimensional shape is converted so that the size of the detected target object is the same as that of the reference three-dimensional shape. After that, the position matching process is performed between the surface shape of the target object and the size-converted reference three-dimensional shape. It is determined which surface the surface shape measured by the matching process corresponds to in the reference three-dimensional shape. At the same time, the opposite surface of the unmeasured object is identified in the reference three-dimensional shape. The three-dimensional shape of the target object is estimated by synthesizing the opposite surface in the specified reference three-dimensional shape with the measured surface shape of the target object. When a plurality of different reference 3D shapes are stored, the opposite surface is estimated from the reference 3D shape that most closely matches the measurement shape.

In order to further improve the estimation accuracy of the 3D shape, the difference in shape between the measured surface shape and the matching surface of the reference 3D shape is calculated, and the calculated difference is added to or subtracted from the opposite surface of the reference 3D shape. Correct the shape to make it the estimated opposite surface. Alternatively, the amount of the above difference with respect to the thickness of the shape may be calculated, and the correction amount may be changed according to the thickness of the shape on the opposite surface.

When the size of one pixel in the object space is larger than the size of the target object, the estimated three-dimensional shape becomes an inaccurate shape with a step. Therefore, it is desirable that the number of pixels of the image sensor 11 is large, and it is desirable to shoot so that the object occupies as much as possible with respect to the screen at the time of shooting. In order to reduce the step difference in pixel units, the shape may be changed to a smoother shape by applying interpolation processing, and further, polygon data may be used.

Next, volume estimation, which is one of the attribute information, will be described. In the volume estimation, the volume is calculated using the three-dimensional shape estimated by the shape estimation. When the estimated volume is voxel unit data, the number of voxels in the estimated three-dimensional shape is counted, and the length of one side of the voxel is estimated by using Equation 4 to estimate the volume in the object space. If the estimated volume is already expressed in the International System of Units in object space, the volume is estimated by integrating within the estimated three-dimensional shape. The estimated volume may be derived based on the voxel standard, which is a unit volume element (normal lattice unit) that is the base in image processing, or is derived based on the actual size dimensional standard in the real world. You may.

Next, mass estimation, which is one of the attribute information, will be described. In the mass estimation, the mass of the target object is estimated by multiplying the volume of the target object derived by the volume estimation by the density information of the target object stored in the storage unit 14 and / or the memory 136. The density information may be uniform with respect to the target object, but it is also possible to hold and use different information for each part in order to estimate the mass with higher accuracy. Each part is divided into, for example, the head, torso, arms, legs, etc. by using skeleton analysis of the target object, and mass estimation is performed using different density information for each part. In the present embodiment, the mass is calculated using the density information of the target object, but the mass is not limited to this, and the specific weight information of the target object is stored in advance in the storage unit 14 and / or the memory 136 for estimation. The weight may be estimated by multiplying the volume of the three-dimensional shape of the object.

In S308, the control unit 12 displays the attribute information estimated in S307 on the display unit 16 and stores it in the storage unit 14. It is desirable that the attribute information estimated in S307 is recorded in association with the depth image as the metadata of the viewing image generated in S303.

As described above, according to the present embodiment, it is possible to estimate the attribute information of the object in the image from the depth image generated from the captured image and the conditions under which the image was taken. Specifically, the attribute information of the target object is estimated by using the depth image, the information learned and acquired in advance for area detection, attitude estimation, and attitude conversion of the object, the shape measured in advance, the shooting parameters, and the mass ratio. it can.

[Embodiment 2] Next, the second embodiment will be described.

In the first embodiment, the user selects the measurement target object. On the other hand, the second embodiment is different from the first embodiment in that the user does not have a selection input of the measurement target object. In the second embodiment, the configuration and functions of the digital camera 100 are the same as those in FIGS. 1 and 3 of the first embodiment, and the differences from the attribute information estimation process of the first embodiment will be mainly described.

FIG. 8 shows the attribute information estimation process of the second embodiment, and shows the same process as the process of FIG. 3 of the first embodiment with the same step number.

In S801, similarly to S302 in FIG. 3, the control unit 12 processes so as to perform imaging with the imaging settings such as the set focal position, aperture, and exposure time.

In S802, similarly to S303 in FIG. 3, the image generation unit 130 generates an viewing image and a depth image.

In S803, the object detection unit 132 recognizes the subject. Subject recognition / region detection generates an object extraction image by identifying an object in an image and extracting a position / contour based on information on the classification / type of the object acquired in advance by machine learning. The extracted position / contour information is also applied to the depth image, and the depth generation unit 131 generates the object extraction depth image. Machine learning is not limited to a specific method, and any method may be used.

In S804, the posture estimation unit 133 and the posture conversion unit 134 perform posture estimation and posture conversion of the object identified and extracted in S803. Here, the details of the processing of S804 performed by the posture estimation unit 133 and the posture conversion unit 134 of the image processing device 13 will be described with reference to the flowchart of FIG.

In S8041, the posture estimation unit 133 acquires the shape of the surface of the target object as seen from the shooting direction by subtracting the distance from the digital camera 100 to the reference position of the target object from the object extraction depth image generated in S803. .. The reference position may be set at the closest position from the digital camera 100 to the target object, or may be set at the farthest position, and is not particularly limited.

In S8042, the posture estimation unit 133 compares the surface shape of the target object obtained in S8041 with the three-dimensional shape of the target object stored in the storage unit 14 in advance, and either of them has the same size. Change the size of one. Considering the subsequent estimation of attribute information, it is desirable to match the size of the three-dimensional shape stored in the storage unit 14 and / or the memory 136 with the size of the surface shape of the target object in advance, and the conversion coefficient is stored in advance. It is desirable to store in the unit 14 and / or the memory 136.

In S8043, the posture estimation unit 133 performs matching processing between the surface shape of the target object acquired in S8041 and the three-dimensional shape previously stored in the storage unit 14 and / or the memory 136, and photographs the target object. Identify the direction you are in. This method is effective when the posture of the photographed object is similar to the posture in which the attribute information stored in the storage unit 14 and / or the memory 136 is easily acquired, and the imaging directions are different. On the other hand, when the posture of the target object is significantly different from the posture stored in the storage unit 14 and / or the memory 136 in advance, the evaluation value (matching score) in the matching process is lowered. Therefore, in S8044, the matching score is compared with the threshold value. When the matching score is higher than the threshold value in S8044, the photographing surface representing the orientation of the target object is specified in S8045. If the matching score is lower than the threshold value, the joint site and skeleton of the target object are specified in S8046. This identification is also performed by using the information acquired by learning in advance.

In S8047, the posture changing unit 134 calculates the difference from the skeleton position in the three-dimensional shape prepared in advance, rotates the portion in the tip direction direction from the joint position with the joint position as a fulcrum, and resembles the reference posture. The surface shape is changed so as to. For example, when a cow is photographed while sitting, the joint positions, lengths, and angles of rotation of the legs such as the thigh, hock, and front knee are estimated, and the joints are rotated as fulcrums of rotation to stand up. Generate an estimated image of. The converted surface shape is input to the matching process of S8043 again, and the photographing surface representing the direction of the target object is specified again.

In S8048, the posture conversion unit 134 performs geometric transformation on the object extracted image and the surface shape based on the imaged surface information specified in S8045, as if the target object was photographed from the side surface, as in S306 of FIG. Use to change posture. The converted object extracted image and surface shape are stored in the memory 136 and used for the subsequent processing.

Returning to the description of FIG. 8, in S805, the attribute information estimation unit 135 estimates the attribute information of the target object in the same manner as in S307 of FIG. The attribute information estimated in S805 is displayed on the display unit 16 and stored in the storage unit 14. It is desirable that the estimated attribute information is stored together with the depth image as the metadata of the viewing image generated in S802.

Here, the difference between the attribute information estimation of S805 and S307 will be described.

Regarding the dimension estimation which is one of the attribute information, in the first embodiment, the user specifies the measurement position and measures the dimension at the designated position. On the other hand, in the second embodiment, the image processing device 13 identifies the type of the object based on the object identification result obtained in S803, and determines the dimensions from the necessary dimensional information (total length, head and body length, body height, etc.). Determine the measurement position. Then, similarly to S8046, the skeleton recognition is performed using the information acquired in advance by learning, and the measurement position is specified by using the contour information obtained by the subject recognition in S803.

Regarding shape estimation, which is one of the attribute information, conversion for matching the size of the surface shape generated by the posture estimation / conversion of S804 with the size of the three-dimensional shape stored in the storage unit 14 and / or the memory 136 in advance. Use the coefficient. The size of the three-dimensional shape stored in the storage unit 14 is converted using the conversion coefficient, and the back surface portion other than the surface shape of the object to be measured is acquired from the three-dimensional shape whose size has been converted. The three-dimensional shape of the measurement target object is generated by synthesizing the surface shape of the measurement target object and the back surface shape obtained from the three-dimensional shape. In synthesizing, smoothing is performed so that the connected portion is smooth.

Regarding the estimation of volume and mass, which is one of the attribute information, the estimation is performed in consideration of the hair of the target object. In the first embodiment, the body hair is not taken into consideration, and the mass of the body hair is calculated with the same density. Therefore, the error between the actual mass and the estimated mass may become large. In addition, in wool production and the like, it may be necessary to estimate only the body hair volume.

As described above, according to the present embodiment, in addition to the treatment of the first embodiment, the amount of hair is estimated and corrected. To estimate the amount of hair, first perform a matching process using a joint bilateral filter, a guided filter, etc., and calculate the alpha matte. The thickness of the hair region is calculated with the region where the alpha mat is 1 or less as the hair region, and the volume obtained by excluding the hair region from the estimated three-dimensional shape is calculated. Similarly, for mass estimation, the mass excluding the hair region is estimated by multiplying the density information stored for each type of object by using the volume excluding the hair region. Alternatively, the volume of only the body hair region is calculated from the volume including the body hair and the volume excluding the body hair, the body hair mass is estimated using the density information of the body hair, and the sum is taken with the mass excluding the body hair. Estimate the mass considering the difference in hair density.

[Other Embodiments]
In the present embodiment, it has been described that the image pickup device 11 has an image pickup surface phase difference ranging type photoelectric conversion element and can acquire a viewing image and a depth image. However, in the embodiment of the present invention, the depth information is acquired. Is not limited to this. The depth information may be acquired by a stereo ranging method based on, for example, a plurality of captured images obtained from a binocular imaging device or a plurality of different imaging devices. Alternatively, for example, it may be acquired by using a stereo ranging method using a light irradiation unit and an image pickup device, a method using a combination of a TOF (Time of Flight) method and an image pickup device, or the like.

The attribute information estimation processing of the first embodiment and the second embodiment is not limited to the respective embodiments, and can be realized by exchanging the processes using the same information.

Further, the image processing device applicable as the present embodiment includes a digital still camera, a digital video camera, an in-vehicle camera, a mobile phone, a smartphone, and the like.

The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device execute the program. It can also be realized by the process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention is a non-contact attribute information estimation of an object using an imaging device. For example, a simple livestock growth record, animal health management in a zoo, acquisition of attributes of wild animals from a distance, etc. It is useful in situations where measurement with a mass meter is difficult.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

100 ... Digital camera, 12 ... Control unit, 13 ... Image processing device, 130 ... Image generation unit, 131 ... Depth generation unit, 132 ... Object detection unit, 133 ... Attitude estimation unit, 134 ... Attitude conversion unit, 135 ... Attribute information Estimator

Claims (24)

Depth generation means for generating depth information showing the distance distribution in the depth direction of the subject from the captured image obtained by capturing the subject, and
An object detecting means for detecting a region of a specific object from the captured image,
A posture estimation means for estimating the posture of the specific object, and
A posture changing means for converting the posture of the specific object in the captured image and the depth information, and
An image processing apparatus comprising: an attribute information estimating means for estimating the attribute information of the specific object from the captured image in which the posture is changed, the depth information, and the shooting conditions of the image. Claim 1 is characterized in that the object detecting means detects a specific object type and region by extracting a region of the specific object using the object information acquired by learning in advance. The image processing apparatus according to. The image processing apparatus according to claim 1, wherein the posture estimating means estimates the posture of the specific object by using the information of the object acquired by learning in advance. The third aspect of the present invention, wherein the posture estimating means estimates the posture of the specific object by using the three-dimensional shape data of the object acquired in advance and the data of the part of the specific object. Image processing device. The posture changing means uses the posture of the specific object in the captured image and the depth information estimated by the posture estimating means and the posture of the specific object acquired in advance, and uses the captured image and the depth information. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is subjected to geometric conversion. The posture changing means uses the information on the part of the specific object estimated by the posture estimation means and the information on the posture and the part of the object acquired in advance, and the identification in the captured image and the depth information. The image processing apparatus according to claim 5, wherein geometric transformation is performed for each part of the object. The posture estimation means calculates the main normal direction of the body portion of the specific object, and obtains a plane perpendicular to the normal direction.
The image processing apparatus according to claim 6, wherein the posture changing means performs geometric transformation on the captured image and the depth information with reference to the plane. The image processing apparatus according to any one of claims 1 to 7, wherein the attribute information is at least one of dimensions, shape, volume, and mass. The claim is characterized in that the attribute information estimating means specifies a measurement position according to the type of the specific object, and estimates the dimension at the measurement position using the information of the object acquired by learning in advance. 8. The image processing apparatus according to 8. The attribute information estimation means estimates the shape of the part of the object that cannot be acquired from the captured image using the three-dimensional shape of the object acquired in advance, and synthesizes the shape of the part of the object obtained from the depth information of the object. The image processing apparatus according to claim 8, wherein the three-dimensional shape of the specific object is estimated. The eighth aspect of claim 8, wherein the attribute information estimating means estimates a volume from the three-dimensional shape of the specific object, and estimates the mass of the specific object using the volume and the density of the object. Image processing device. The image processing apparatus according to claim 11, wherein the attribute information estimation means estimates the mass of the specific object by using the portion of the object obtained from the captured image and the density of each portion of the object. .. When the specific object is an animal, the attribute information estimating means calculates the volume of the specific object excluding the hair region, and uses the volume excluding the hair region and the density of the object to obtain the specific object. The image processing apparatus according to claim 11, wherein the mass of the image processing apparatus is estimated. The image processing apparatus according to any one of claims 1 to 12, further comprising an input means for inputting information about the specific object. The object detecting means is characterized in that it determines the type of the specific object by using the information for identifying the type of the object acquired by learning in advance and the information about the object input by the input means. The image processing apparatus according to claim 14. Further having a designation means for designating the measurement position of the specific object,
The image processing apparatus according to claim 9, wherein the attribute information estimating means calculates the dimension of a measurement position of the specific object designated by the designated means. 16. The designation means is characterized in that the measurement position is designated according to an operation of designating two points for the specific object by a user, an operation of designating three or more points, or an operation of tracing. The image processing apparatus according to. The image processing apparatus according to any one of claims 1 to 17, wherein the specific object is an animal other than a human being. The image processing device is an image pickup device having an image pickup element for capturing an image.
The image processing apparatus according to any one of claims 1 to 17, wherein the depth generating means generates depth information from images having different parallax captured by the image pickup device. The image processing apparatus according to any one of claims 1 to 19, further comprising a recording means for recording the captured image, the depth information, and the attribute information in association with each other. The image processing apparatus according to claim 19, wherein the image pickup device has a plurality of photoelectric conversion units in one pixel. A step in which the depth generation means generates depth information indicating the distance distribution of the subject in the depth direction from the captured image of the subject.
A step in which the object detecting means detects a region of a specific object from the captured image,
A step in which the posture estimating means estimates the posture of the specific object,
A step in which the posture changing means changes the posture of the specific object in the captured image and the depth information.
An image processing method, characterized in that the attribute information estimating means includes a step of estimating the attribute information of the specific object from the captured image whose posture is changed, the depth information, and the shooting conditions of the image. A program for causing a computer to function as the image processing device according to any one of claims 1 to 21. A storage medium in which a program for operating a computer as an image processing device according to any one of claims 1 to 21 is stored.

Images