JP2013171494A - Image processing device, method, and program - Google Patents

Abstract

A user's viewpoint position is obtained more reliably and stably.
A detection processing unit detects a user's face and eyes from a photographed image taken with a user observing the display unit as a subject. When the user's face and eyes are detected from the captured image, the position calculation processing unit calculates the user's viewpoint position based on the eye position on the captured image and the user's statistical interocular distance. At this time, the position calculation processing unit also calculates relationship information indicating the relationship between the user's actual interocular distance and the face width from the obtained viewpoint position and face detection result. When only the user's face is detected from the captured image, the position calculation processing unit calculates the user's viewpoint position from the face detection result and the relationship information calculated in the past. The present technology can be applied to an image processing apparatus.
[Selection] Figure 3

Description

  The present technology relates to an image processing device and method, and a program, and more particularly, to an image processing device and method, and a program that can obtain a user's viewpoint position more reliably and stably.

  For example, there is known an autostereoscopic display technology for controlling the display of 3D video on a display based on the position of the user's eyes (viewpoint position) with respect to the display (see, for example, Patent Document 1). Such a technique requires the distance from the display to the user, that is, the user's viewpoint position in three dimensions.

  Therefore, a technique has been proposed in which the position of the user's eye is detected from the captured image, and the user's viewpoint position is obtained from the detection result.

JP 2011-139281 A

  However, with the above-described technique, it has been difficult to stably and reliably obtain the user's viewpoint position. For example, in the method of detecting the user's eye from the photographed image and obtaining the user's viewpoint position from the detection result, when the user's eye is not on the image, such as when the user crawls, the user's viewpoint position I couldn't get it.

  The present technology has been made in view of such a situation, and enables the user's viewpoint position to be obtained more reliably and stably.

  An image processing apparatus according to an aspect of the present technology detects a face of the user from the photographed image based on a photographed image photographed with the user as a subject, and detects the user's eye from the photographed image. When the user's eyes are detected, a position calculation unit that calculates the position of the user in real space is detected based on a detection result of the user's eyes, and the user's eyes are detected A relation information calculation unit that calculates relation information indicating a relation between the interocular distance of the user and a face width based on the calculated position of the user and the detection result of the user's face, When the user's face is detected and the user's eyes are not detected, the calculation unit calculates the position of the user based on the detection result of the user's face and the relationship information.

  When the user's eyes are detected, the position calculation unit can calculate the user's position based on an average user's interocular distance and a detection result of the user's eyes.

  The image processing apparatus further includes a determination unit that determines at least one of the sex, age, or race of the user based on the captured image, and the position calculation unit includes a determination result by the determination unit. The position of the user can be calculated based on the determined average distance between the eyes of the user and the detection result of the eyes of the user.

  The relationship information calculation unit can calculate an actual face width of the user as the relationship information.

The image processing apparatus further includes a face identification unit that extracts a feature amount from the user's face area detected from the captured image, and a relationship information holding unit that holds the feature amount and the relationship information in association with each other. The position calculating unit detects the user's face from the captured image immediately before the captured image to be processed and detects the user's face from the captured image to be processed; When the first eye is not detected, the position of the user can be calculated based on the relation information selected based on the feature amount and the detection result of the user's face.

  An image processing method or program according to one aspect of the present technology detects the user's face from the captured image based on a captured image captured using the user as a subject, detects the user's eyes from the captured image, When the user's eyes are detected, the position of the user in real space is calculated based on the detection result of the users' eyes, and when the user's eyes are detected, the calculated position of the user and Based on the detection result of the user's face, calculating relationship information indicating the relationship between the distance between the eyes of the user and the face width, when the user's face is detected, and the user's eyes are not detected, Calculating a position of the user based on a detection result of the user's face and the relationship information.

  In one aspect of the present technology, the user's face is detected from the captured image based on a captured image captured with the user as a subject, the user's eyes are detected from the captured image, and the user's eyes are If detected, the position of the user in real space is calculated based on the detection result of the user's eyes. If the user's eyes are detected, the calculated position of the user and the face of the user are detected. Based on the detection result, relationship information indicating the relationship between the user's interocular distance and the face width is calculated, and when the user's face is detected and the user's eyes are not detected, Based on the detection result and the relation information, the position of the user is calculated.

  According to one aspect of the present technology, the viewpoint position of the user can be obtained more reliably and stably.

It is a figure explaining calculation of a user's viewpoint position by a face width method. It is a figure explaining switching of a face width system and an eye width system. It is a figure which shows the structural example of an image processing system. It is a flowchart explaining a viewpoint position calculation process. It is a figure which shows the structural example of a computer.

  Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

<First Embodiment>
[About calculation of viewpoint position]
The present technology estimates the position (viewpoint position) of the user's eyes with respect to the display unit (display) using, for example, the results of face detection and eye detection, and based on the estimation result, 3D video (3D This is a technique related to autostereoscopic display of a head tracking system that controls display of (image). Note that the present technology can be applied to various types of control such as image display based on the user's viewpoint position. However, in the following description, the present technology is used as an example of stereoscopic image display control. to continue.

  In the present technology, as a method for obtaining a three-dimensional user's eye position (viewpoint position) with respect to the display unit, a method for obtaining a viewpoint position using the detection result of the user's face and a viewpoint using the detection result of the user's eye The method for obtaining the position is used while being appropriately switched.

  At this time, when the viewpoint position is obtained from the detection result of the user's eyes, the relationship information indicating the relationship between the user's actual interocular distance and the face size is obtained based on the viewpoint position. When obtaining the viewpoint position using the detection result of the user's face, not only the detection result of the user's face but also the related information is used to obtain the user's viewpoint position.

  Hereinafter, a method for obtaining a viewpoint position using a user's face detection result is also referred to as a face width method, and a method for obtaining a viewpoint position using a user's eye detection result is also referred to as a eye width method.

  The calculation of the user's viewpoint position according to the present technology will be described below.

  First, a method for obtaining a user's viewpoint position using face detection will be described.

  For example, as shown in FIG. 1, it is assumed that a display unit 11 on which an image is displayed is arranged in a three-dimensional space, and a user observes the display unit 11 from the upper side to the lower side in the drawing. Further, it is assumed that a photographing unit 12 that photographs the user is arranged at the center position of the display unit 11 and the photographed image PC11 with the user as a subject is photographed by the photographing unit 12. That is, it is assumed that the subject at which the captured image PC11 is illustrated is captured and the captured image PC11 is obtained.

  In the figure, the horizontal direction is the direction in which both eyes of the user observing the display unit 11 are lined up, and FIG. 1 is a view of the display unit 11 and the user looking down from directly above the user.

  In FIG. 1, it is assumed that the angle of view of the photographing unit 12 is AH, and the position of the intersection of the straight line DR11 parallel to the optical axis of the photographing unit 12 and the photographed image PC11, that is, the center in the horizontal direction in the figure of the photographed image PC11. Assume that the position is the position O. For example, in the drawing of the photographing unit 12, when the horizontal field angle is 63.24 degrees, the field angle AH is set to AH = 63.24 × π / 180 or the like.

  At this time, if the width in the horizontal direction is “1” in the figure of the photographed image PC11, the vertical distance IWD in the figure from the photographing unit 12 to the position O on the photographed image PC11 is expressed by the following equation (1). Desired.

  Further, it is assumed that the area FC on the photographed image PC11 is a user's face area, and the width in the horizontal direction in the figure of the area FC is width. Here, the area FC is an area between the position x and the position x ′ on the captured image PC11.

  Since the horizontal width of the captured image PC11 is “1”, if the position O is “0”, arbitrary positions in the horizontal direction in the figure of the captured image PC11, for example, the position x and the position x ′ are −0.5 to It can be expressed by a numerical value within the range of 0.5. The width width of the user's face area FC on the photographed image PC11 is a value between 0 and 1.

  Assume that the angle from the center of the photographing unit 12 to the left end in the figure of the user's face, that is, the angle formed by the straight line DR11 and the straight line DR12 is the angle φ. Further, it is assumed that an angle corresponding to the user's face width, that is, an angle formed by the straight line DR12 and the straight line DR13 is an angle θ.

  Here, the straight line DR12 is a straight line connecting the photographing unit 12 and the left end position x in the drawing of the region FC, and the straight line DR13 is connecting the photographing unit 12 and the right end position x ′ in the drawing of the region FC. It is a straight line.

  At this time, the angle φ is obtained by the following equation (2) from the position x of the user's face area FC and the above-described distance IWD.

  Therefore, the angle θ corresponding to the user's face width is obtained by using the angle φ obtained by the equation (2), the distance IWD, the position x of the user's face area FC, and the width width of the user's face area FC. It can obtain | require by following Formula (3).

  Thus, when the angle θ corresponding to the user's face width is obtained, the display unit 11 (photographing in real space (three-dimensional)) is obtained from the angle θ and the user's actual face width FW by the following equation (4). The distance R from the unit 12) to the user's viewpoint position (face) can be obtained. Here, the viewpoint position of the user is, for example, an intermediate position between the left and right eyes of the user.

  The actual face width FW of the user is, for example, about 12.5 cm if the user is a child, and about 17.0 cm if the user is an adult. The user's face width FW is a width in the horizontal direction of the user's face in real space, corresponding to the width width.

  As described above, in the calculation of the user's viewpoint position by the face width method, the angle of view AH of the photographing unit 12, the position x and width of the user's face on the photographed image PC11, and the actual face width FW of the user are separated. For example, the viewpoint position (distance R) of the user in three dimensions can be obtained. As will be described later, in the face width method, the value of the face width FW is obtained using relationship information indicating the relationship between the user's actual interocular distance and the face width.

  Next, calculation of the user's viewpoint position by the eye width method will be described.

  Also in the eye width method, the viewpoint position of the user can be calculated by the same calculation as in the face width method. Specifically, for example, it is assumed that the position x and the position x ′ in FIG. 1 are the position of the right eye and the position of the left eye of the user. In this case, if the equation (3) is calculated with the position x as the right eye position of the user and the width width as the width of the left and right eyes of the user, that is, the interocular distance, an angle corresponding to the user's eye width is set as the angle θ. Can be obtained.

  Therefore, the distance R from the photographing unit 12 to the user's viewpoint position can be obtained from the angle θ by the same calculation as in the equation (4). That is, if the angle corresponding to the user's eye width is θ ′ and the actual user's interocular distance is EW, the distance R to the viewpoint position can be obtained by calculating the following equation (5).

  From the above, in the calculation of the viewpoint position of the user by the eye width method, the angle of view AH of the imaging unit 12, the position and interocular distance of the user's eyes on the captured image PC11, and the actual interocular distance EW of the user. If known, the viewpoint position (distance R) of the user in three dimensions can be obtained.

  In the following description, the position of the user's eyes on the captured image PC11 (the user's right eye position in the example of FIG. 1) is xe, and the interocular distance of the user on the captured image PC11 is HE.

  By the way, as described above, the calculation of the user's viewpoint position by the face width method requires the user's actual face width FW, and the calculation of the user's viewpoint position by the eye width method requires the user's actual interocular distance EW. Necessary.

  Regarding the face width FW and the interocular distance EW used for these calculations, a method using values obtained by directly measuring the face width FW and the interocular distance EW for each individual, and a statistical average value of the face A method of using the width FW or the interocular distance EW as a value can be considered.

  For example, when a statistical average value is used as the face width FW or the interocular distance EW, the individual difference in the interocular distance EW is smaller than the individual difference in the face width FW. It is possible to determine the user's viewpoint position more accurately.

  Further, if the distance R to the user's viewpoint position is obtained by either the face width method or the eye width method, the relationship between the actual face width FW and the interocular distance EW can be obtained.

  Therefore, in the present technology, while the user's face and eyes are detected from the captured image, a statistical average value is used as the interocular distance EW, and the distance R is calculated by the eye width method. When the distance R is calculated by the eye width method, the relationship information indicating the relationship between the actual face width FW and the interocular distance EW is obtained from the calculated distance R, the interocular distance EW, and the face detection result. Retained. For example, the relationship information is the face width FW of each individual obtained from the distance R.

  On the other hand, when only the user's face is detected from the photographed image and the user's eyes are not detected, the face width FW obtained as the relationship information is used, and the distance R is calculated by the face width method. Is done.

  Specifically, for example, as shown in FIG. 2, the user's viewpoint position (distance R) is calculated by switching between the eye width method and the face width method. In FIG. 2, the horizontal direction indicates the time direction. Further, the hatched rectangle indicates a section where the user's face is detected from the captured image, and the rectangle without the hatched line indicates a section where the user's eyes are detected from the captured image. .

  In the example of FIG. 2, the user's face is continuously detected from the section Q1 to the section Q3. In addition, the user's eyes are continuously detected in the sections Q1 and Q3, but the user's eyes are not detected in the sections Q2.

  In the section where the user's face is continuously detected, tracking of the detected user's face and identification of the detected user's face are performed. For example, the face is identified by the feature amount extracted from the face area on the photographed image.

  In FIG. 2, when attention is paid to the section Q1, the user's face and eyes are detected in the section Q1. Therefore, in this section Q1, the distance R (viewpoint position) is calculated from the detection result of the user's eyes by the eye width method, and the face width FW as the related information is calculated and held.

  Here, the held face width FW is, for example, an average value of the face width FW obtained in each frame constituting the section Q1. Further, the face width FW as the obtained relation information is held in association with the feature amount extracted from the user's face area on the captured image. Thereby, the face width FW as the related information can be held for each individual user.

  In a section Q2 following the section Q1, the user's face is detected from the captured image, but the user's eyes are not detected. Therefore, in the section Q2, the distance R (viewpoint position) is calculated by the face width method using the face width FW as the relation information held for the detected face and the face detection result.

  For example, as in the section Q2, the user's face may be detected from the captured image depending on the conditions, such as when the user faces down, but the user's eyes may not be detected.

  In such a case, the viewpoint position was calculated by the eye width method while the user's eyes were detected, but when the eyes are no longer detected, the statistical face width is used and the face width method is used. If the viewpoint position is calculated by the above, the viewpoint positions arranged in time series become discontinuous or the viewpoint position error becomes large.

  Therefore, in the section where only the user's face is detected, such as the section Q2, the viewpoint position is calculated by the face width method using the relationship information obtained in advance, so that it is continuous and more accurate in the time direction. Can be obtained. That is, even if there is a section in which the user's eyes are not detected, the user's viewpoint position can be obtained more reliably and stably.

  Further, in the section Q3 after the section Q2, the user's face and eyes are detected again. Further, from the face tracking results, the faces detected in the sections Q1 to Q3 are the faces of the same user. Therefore, in the section Q3, the face width method is switched from the conventional face width method, the distance R (viewpoint position) is calculated, and the face width FW as the related information is calculated and held. That is, the related information is updated.

  Between the section Q3 and the section Q4, since the eyes of the user's face are not detected, the viewpoint position of the user is not calculated in this section.

  Thereafter, the user's face is detected again in the section Q4. In section Q4, since the user's face is detected, but the user's eyes are not detected, the relationship information is used and the distance R (viewpoint position) is calculated by the face width method.

  However, since the user's face is not detected in the section immediately before section Q4, the face detected in section Q4 and the face detected in the section before section Q4 are the same user's face. Is identified by the feature amount extracted from the face. When the faces detected in these sections are the same face, the relationship information about the faces is used for calculating the viewpoint position in the section Q4.

  As described above, even when the tracking of the face area on the photographed image is interrupted, the feature amount is compared to identify whether the user is the same user (face). The viewpoint position can be obtained.

  Further, since the user's face and eyes are detected in and after the section Q4, the viewpoint position of the user is calculated by the eye width method in this section, and the related information is also updated.

[Image processing system configuration example]
Next, specific embodiments to which the present technology is applied will be described. FIG. 3 is a diagram illustrating a configuration example of an embodiment of an image processing system to which the present technology is applied. In FIG. 3, the same reference numerals are given to the portions corresponding to those in FIG. 1, and the description thereof will be omitted as appropriate.

  The image processing system in FIG. 3 includes an imaging unit 12, an image processing device 41, a display control unit 42, and a display unit 11. This image processing system performs parallax control of a stereoscopic image displayed on the display unit 11 according to a user's viewpoint position, for example, and is incorporated in a television receiver, a personal computer, a mobile phone, or the like.

  The photographing unit 12 is provided in the upper center of the display unit 11, photographs a user who observes the display unit 11 from almost the front as a subject, and supplies a photographed image obtained as a result to the image processing device 41.

  The image processing apparatus 41 detects the user's face and eyes from the captured image supplied from the imaging unit 12, calculates the user's viewpoint position based on the detection result, and supplies the user's viewpoint position to the display control unit 42. The image processing device 41 includes a detection processing unit 51 and a position calculation processing unit 52.

  The detection processing unit 51 detects the user's face and eyes from the captured image supplied from the imaging unit 12, and supplies the detection result to the position calculation processing unit 52. The detection processing unit 51 includes a face detection unit 61, an eye detection unit 62, a determination unit 63, and a face identification unit 64.

  The face detection unit 61 detects the user's face from the photographed image from the photographing unit 12. The eye detection unit 62 detects the user's eyes from the captured image from the imaging unit 12. The determination unit 63 determines the gender, age, and race of the detected user based on the captured image from the imaging unit 12. The determination result such as the user's sex is used to specify the user's actual interocular distance EW.

  The face identifying unit 64 performs face identification by extracting a feature amount from a region of the user's face detected from the captured image.

  The face detection result obtained by the face detection unit 61 to the face identification unit 64, the eye detection result, the determination result such as the gender of the user, and the feature amount of the face region are appropriately transmitted from the detection processing unit 51 to the position calculation processing unit. 52.

  The position calculation processing unit 52 calculates the viewpoint position (distance R) of the user in three dimensions based on the face detection result supplied from the detection processing unit 51 and supplies the calculated position to the display control unit 42. The position calculation processing unit 52 includes a viewpoint position calculation unit 71, a relationship information calculation unit 72, and a relationship information holding unit 73.

  The viewpoint position calculation unit 71 calculates the viewpoint position of the user based on the detection result of the face and eyes supplied from the detection processing unit 51. At this time, the viewpoint position calculation unit 71 calculates the user's viewpoint position using the relationship information held in the relationship information holding unit 73 as necessary.

  The relationship information calculation unit 72 calculates the relationship information based on the face detection result from the detection processing unit 51 and the viewpoint position calculation result by the viewpoint position calculation unit 71 and causes the relationship information holding unit 73 to hold the relationship information. The relationship information holding unit 73 holds the relationship information calculated by the relationship information calculation unit 72 in association with the feature amount supplied from the detection processing unit 51.

  In addition, the display control unit 42 supplies a stereoscopic image acquired from a recording unit or the like (not shown) to the display unit 11 for stereoscopic display. At this time, the display control unit 42 performs parallax control of the stereoscopic image displayed on the display unit 11 according to the viewpoint position of the user supplied from the position calculation processing unit 52. Specifically, for example, the display control unit 42 assigns a right-eye image or a left-eye image constituting a stereoscopic image to each display area of the display unit 11 according to the user's viewpoint position. 3D images can be viewed.

  The display unit 11 includes a display that displays a stereoscopic image by the naked eye method, and displays the stereoscopic image supplied from the display control unit 42 under the control of the display control unit 42.

[Description of viewpoint position calculation processing]
By the way, when the display of a stereoscopic image is instructed to the image processing system of FIG. Then, the image processing device 41 performs viewpoint position calculation processing and outputs the user's viewpoint position to the display control unit 42. Then, the display control unit 42 performs parallax control according to the viewpoint position from the image processing device 41 and causes the display unit 11 to display a stereoscopic image.

  Hereinafter, the viewpoint position calculation process performed by the image processing apparatus 41 will be described with reference to the flowchart of FIG.

  In step S <b> 11, the face detection unit 61 performs face detection based on the photographed image supplied from the photographing unit 12 and detects a region of the user's face from the photographed image. The face detection method may be any method such as a method using a discriminator.

  In step S12, the face detection unit 61 performs tracking (tracking) of the face area on the captured image based on the face detection result in step S11 and the past face detection result. Thereby, it can be specified whether the face detected in the captured image of each time is a face of the same user.

  In step S <b> 13, the eye detection unit 62 performs eye detection based on the captured image supplied from the imaging unit 12, and detects the user's eye area from the captured image. At this time, the eye detection unit 62 uses the face detection result by the face detection unit 61 as necessary to detect the user's eye region from within the face region on the captured image. The eye detection method may be any method such as a method using a discriminator or a template.

  In step S <b> 14, the determination unit 63 determines the gender, age, and race of the user detected from the captured image based on the captured image supplied from the capturing unit 12 and the result of face detection by the face detection unit 61. Make a decision.

  In step S <b> 15, the face identifying unit 64 identifies the face detected from the captured image based on the captured image supplied from the capturing unit 12 and the result of face detection by the face detecting unit 61. That is, the face identification unit 64 extracts a feature amount from a face area on the captured image. This feature amount may be used for face tracking as necessary.

  In step S <b> 16, the detection processing unit 51 determines whether a face is detected from the captured image. For example, when the user's face is detected from the captured image in the process of step S11, it is determined that the face has been detected.

  If it is determined in step S16 that no face has been detected, the process returns to step S11 and the above-described process is repeated. That is, the captured image of the next frame is set as a processing target, and face detection or the like is performed on the captured image.

  If no face is detected from the photographed image, the processes in steps S12 to S15 described above are not substantially performed.

  On the other hand, if it is determined in step S16 that a face has been detected, the process proceeds to step S17. At this time, the detection processing unit 51 appropriately positions the face detection result, the face tracking result, the eye detection result, the gender determination result, and the facial feature amount obtained in the processes of steps S11 to S15. This is supplied to the calculation processing unit 52.

  In step S <b> 17, the position calculation processing unit 52 determines whether the user's eyes are detected from the captured image based on the eye detection result supplied from the detection processing unit 51.

  If it is determined in step S17 that an eye has been detected, in step S18, the viewpoint position calculation unit 71 determines a three-dimensional user based on the user's eye position on the captured image and the user's interocular distance EW. Is calculated, that is, the viewpoint position (distance R) of the user in the real space.

  For example, the viewpoint position calculation unit 71 associates gender, age, and race with an average (statistical) interocular distance EW of users having the same gender, age, and race. The eye width table is recorded in advance. The viewpoint position calculation unit 71 acquires the interocular distance EW specified from the determination result of the sex, age, and race of the user supplied from the detection processing unit 51 from the recorded eye width table. That is, the interocular distance EW specified by the sex, age, and race obtained as a result of the determination is read.

  The viewpoint position calculation unit 71 also includes the user's eye position xe and interocular distance HE on the captured image as a result of eye detection supplied from the detection processing unit 51, and the known angle of view AH of the imaging unit 12. Based on the above, the same calculations as in equations (1) to (3) are performed.

  Since the angle θ ′ corresponding to the user's eye width is obtained by this calculation, the viewpoint position calculation unit 71 calculates the above-described formula (θ) from the obtained angle θ ′ and the interocular distance EW read from the eye width table. 5) is calculated, and the distance R from the display unit 11 to the user is calculated as the viewpoint position. The user's viewpoint position may be the distance R or the coordinates of the user's viewpoint position on the three-dimensional coordinate space obtained from the distance R.

  As described above, by selecting and using the interocular distance EW that matches the sex, age, and race of the user on the captured image from among the statistical values for each sex, age, and race, The user's viewpoint position (distance R) can be obtained with higher accuracy.

  In addition, although it has been described that the interocular distance EW specified by the determination result of the user's sex, age, and race is used, at least one of the user's sex, age, and race is used. The inter-distance EW may be specified.

  Alternatively, the interocular distance EW of the user may be directly measured by a sensor or the like, and the measurement result may be recorded in the viewpoint position calculation unit 71 in advance, or the interocular distance EW directly input or selected by the user. May be recorded in the viewpoint position calculation unit 71 in advance.

  In step S <b> 19, the relationship information calculation unit 72 calculates relationship information based on the face detection result from the detection processing unit 51 and the viewpoint position calculation result by the viewpoint position calculation unit 71.

  Specifically, the relational information calculation unit 72 is a known photographing that includes the position x of the user's face on the photographed image as the face detection result from the detection processing unit 51 and the width width of the user's face on the photographed image. Expressions (1) to (3) are calculated based on the angle of view AH of the unit 12.

  Since the angle θ corresponding to the face width of the user is obtained by this calculation, the relationship information calculation unit 72 calculates the equation (4) from the obtained angle θ and the distance R calculated by the viewpoint position calculation unit 71. Further, the actual face width FW of the user is calculated as related information.

  In step S20, the relationship information holding unit 73 holds the relationship information calculated by the relationship information calculation unit 72 in the process of step S19.

  More specifically, the relationship information holding unit 73 includes the feature amount associated with the relationship information among the face width FW as the held relationship information, which is supplied from the detection processing unit 51. The relationship information that is most similar to is identified. For example, a feature having the shortest distance between feature amounts, such as a difference between feature amounts, is a similar feature amount. Further, when the face is continuously detected by the face tracking, the relationship information obtained for the face is specified.

  Then, the relationship information holding unit 73 updates the relationship information based on the face width FW as the specified relationship information and the face width FW as the relationship information calculated in step S19, and the updated relationship information is displayed. Hold.

  For example, in updating the relationship information, an average value of the face width FW newly calculated in step S19 and several face widths FW obtained in the past is obtained, and the obtained average value is the updated face value. The width is FW (related information).

  Note that, for example, the face width FW may not be updated after being updated a predetermined number of times, or the face width FW is updated only in a section where the face is continuously detected. If a new face is detected, new relationship information may be held.

  In step S <b> 21, the position calculation processing unit 52 outputs the user's viewpoint position (distance R) calculated by the viewpoint position calculation unit 71, that is, the three-dimensional eye position, to the display control unit 42. The processes performed in steps S18 to S21 are processes performed in, for example, the section Q1 and the section Q3 in FIG.

  In step S22, the position calculation processing unit 52 determines whether to end the process of calculating the viewpoint position. For example, when the user instructs to stop the reproduction of the stereoscopic image, it is determined that the process is to be ended.

  If it is determined in step S22 that the process is not terminated, the process returns to step S11, and the above-described process is repeated.

  If it is determined in step S22 that the process is to be terminated, the viewpoint position calculation process is terminated.

  Furthermore, when it is determined in step S17 described above that no eye has been detected, the process proceeds to step S23.

  In step S <b> 23, the viewpoint position calculation unit 71 determines the position of the three-dimensional user's eye based on the face detection result and the feature amount from the detection processing unit 51 and the relationship information held in the relationship information holding unit 73. That is, the viewpoint position (distance R) of the user in the real space is calculated.

  Specifically, it is assumed that the face tracking result supplied from the detection processing unit 51 is a result that the face is continuously detected from the section in which the user's face and eyes are detected from the captured image. In this case, the viewpoint position calculation unit 71 acquires the relationship information obtained for the continuously detected face from the relationship information holding unit 73. That is, when the frame to be processed is, for example, a frame in the section Q2 in FIG. 2, the relationship information of the face detected continuously is acquired. At this time, the feature amount extracted from the face area by the face identification unit 64 is used as necessary.

  Further, the face tracking result supplied from the detection processing unit 51 is a section in which no face is detected between the section in which the user's face and eyes are detected from the last captured image and the captured image to be processed. Suppose that there is a result. In this case, the viewpoint position calculation unit 71 acquires, from the relationship information holding unit 73, related information calculated for the same face as the face detected from the captured image that is the processing target.

  For example, among the captured images before the processing target captured image, the facial feature amount of the captured image in which the face and eyes are detected last is similar to the facial feature amount of the processing target captured image. If they are, they are assumed to be the same face, and related information about the faces is acquired. Further, for example, the relationship information associated with the feature amount having the highest similarity with the feature amount extracted from the face region on the captured image to be processed by the face identification unit 64 is acquired from the relationship information holding unit 73. You may make it do.

  In addition, when there is a section where no face is detected between the section in which the user's face and eyes were detected from the last captured image and the captured image to be processed, for example, the captured image to be processed is This is the case in the section Q4 of FIG. For example, in the section Q4 of FIG. 2, when it is determined that the feature amount extracted from the face in the section Q4 and the feature amount extracted from the face in the immediately preceding section Q3 are the same, the section The relationship information of the face detected in Q3 is acquired.

  Upon obtaining the face width FW as the relationship information from the relationship information holding unit 73, the viewpoint position calculation unit 71 calculates the angle θ corresponding to the user's face width.

  In other words, the viewpoint position calculation unit 71 includes the position x of the user's face on the captured image as the face detection result from the detection processing unit 51, the width width of the user's face on the captured image, and the known imaging unit 12. Equations (1) to (3) are calculated on the basis of the angle of view AH and the angle θ is obtained. Then, the viewpoint position calculation unit 71 calculates the user's viewpoint position (distance R) by calculating Expression (4) from the obtained angle θ and the face width FW as the relationship information.

  In step S23, when the viewpoint position of the user is calculated, the processes in steps S21 and S22 are then performed, and the viewpoint position calculation process ends.

  As described above, the image processing apparatus 41 calculates the user's viewpoint position (distance R) by switching between the eye width method and the face width method according to the results of face detection and eye detection. Further, when the viewpoint position is calculated by the eye width method, the image processing apparatus 41 obtains the relationship between the actual distance between the eyes of the user and the face width from the calculation result, and holds it as the relationship information.

  As described above, when the viewpoint position is calculated by the eye width method, the relationship information is obtained and held, and when the viewpoint position is calculated by the face width method, the relationship information is used, thereby more reliably. The viewpoint position of the user can be obtained stably. As a result, in the parallax control of the stereoscopic image by the display control unit 42, more stable and accurate parallax control can be performed, and the crosstalk of the stereoscopic image displayed on the display unit 11 can be reduced, or the reverse view can be performed. Can be suppressed.

  By the way, the above-described series of processing can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing a computer incorporated in dedicated hardware and various programs.

  FIG. 5 is a block diagram illustrating an example of a hardware configuration of a computer that executes the above-described series of processes using a program.

  In a computer, a central processing unit (CPU) 201, a read only memory (ROM) 202, and a random access memory (RAM) 203 are connected to each other by a bus 204.

  An input / output interface 205 is further connected to the bus 204. An input unit 206, an output unit 207, a recording unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

  The input unit 206 includes a keyboard, a mouse, a microphone, and the like. The output unit 207 includes a display, a speaker, and the like. The recording unit 208 includes a hard disk, a nonvolatile memory, and the like. The communication unit 209 includes a network interface and the like. The drive 210 drives a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  In the computer configured as described above, the CPU 201 loads, for example, the program recorded in the recording unit 208 to the RAM 203 via the input / output interface 205 and the bus 204, and executes the program. Is performed.

  The program executed by the computer (CPU 201) can be provided by being recorded on the removable medium 211 as a package medium or the like, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  In the computer, the program can be installed in the recording unit 208 via the input / output interface 205 by attaching the removable medium 211 to the drive 210. Further, the program can be received by the communication unit 209 via a wired or wireless transmission medium and installed in the recording unit 208. In addition, the program can be installed in the ROM 202 or the recording unit 208 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

  For example, the present technology can take a configuration of cloud computing in which one function is shared by a plurality of devices via a network and is jointly processed.

  In addition, each step described in the above flowchart can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

  Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

  Furthermore, this technique can also be set as the following structures.

[1]
A face detection unit that detects the user's face from the photographed image based on a photographed image photographed with the user as a subject;
An eye detector that detects the eyes of the user from the captured image;
A position calculating unit that calculates the position of the user in real space based on a detection result of the user's eye when the user's eye is detected;
When the user's eyes are detected, the relationship information that calculates the relationship information indicating the relationship between the user's interocular distance and the face width based on the calculated position of the user and the detection result of the user's face A calculation unit and
When the user's face is detected and the user's eyes are not detected, the position calculation unit calculates the user's position based on the detection result of the user's face and the relation information. .
[2]
The position calculation unit calculates the position of the user based on an average user's interocular distance and a detection result of the user's eyes when the user's eyes are detected. [1] Image processing apparatus.
[3]
A determination unit that determines at least one of gender, age, or race of the user based on the captured image;
The position calculation unit calculates the position of the user based on the average interocular distance of the user determined by the determination result by the determination unit and the detection result of the user's eyes. Image processing device.
[4]
The image processing apparatus according to any one of [1] to [3], wherein the relationship information calculation unit calculates an actual face width of the user as the relationship information.
[5]
A face identifying unit that extracts a feature amount from an area of the user's face detected from the captured image;
A relation information holding unit that holds the feature quantity and the relation information in association with each other; and
The position calculation unit does not detect the user's face from the captured image immediately before the captured image to be processed, detects the user's face from the captured image to be processed, and the user's eyes If not detected, the position of the user is calculated based on the relation information selected based on the feature amount and a detection result of the user's face. [1] to [4] Image processing device.

  41 image processing device 61 face detection unit 62 eye detection unit 63 determination unit 64 face identification unit 71 viewpoint position calculation unit 72 relationship information calculation unit 73 relationship information holding unit

Claims (7)

A face detection unit that detects the user's face from the photographed image based on a photographed image photographed with the user as a subject;
An eye detector that detects the eyes of the user from the captured image;
A position calculating unit that calculates the position of the user in real space based on a detection result of the user's eye when the user's eye is detected;
When the user's eyes are detected, the relationship information that calculates the relationship information indicating the relationship between the user's interocular distance and the face width based on the calculated position of the user and the detection result of the user's face A calculation unit and
When the user's face is detected and the user's eyes are not detected, the position calculation unit calculates the user's position based on the detection result of the user's face and the relation information. . The position calculation unit calculates the position of the user based on an average distance between the eyes of the user and a detection result of the eyes of the user when the user's eyes are detected. Image processing apparatus. A determination unit that determines at least one of gender, age, or race of the user based on the captured image;
The position calculation unit calculates the position of the user based on the average user's interocular distance determined by the determination result by the determination unit and the detection result of the user's eyes. Image processing device. The image processing apparatus according to claim 1, wherein the relationship information calculation unit calculates an actual face width of the user as the relationship information. A face identifying unit that extracts a feature amount from an area of the user's face detected from the captured image;
A relation information holding unit that holds the feature quantity and the relation information in association with each other; and
The position calculation unit does not detect the user's face from the captured image immediately before the captured image to be processed, detects the user's face from the captured image to be processed, and the user's eyes The image processing apparatus according to claim 1, wherein if not detected, the position of the user is calculated based on the relation information selected based on the feature amount and a detection result of the user's face. Based on a photographed image photographed with the user as a subject, the user's face is detected from the photographed image,
Detecting the user's eyes from the captured image;
When the user's eyes are detected, the position of the user in real space is calculated based on the detection results of the user's eyes,
When the user's eyes are detected, based on the calculated position of the user and the detection result of the user's face, calculate relationship information indicating the relationship between the user's interocular distance and the face width;
An image processing method including a step of calculating the position of the user based on a detection result of the user's face and the relationship information when the user's face is detected and the user's eyes are not detected. Based on a photographed image photographed with the user as a subject, the user's face is detected from the photographed image,
Detecting the user's eyes from the captured image;
When the user's eyes are detected, the position of the user in real space is calculated based on the detection results of the user's eyes,
When the user's eyes are detected, based on the calculated position of the user and the detection result of the user's face, calculate relationship information indicating the relationship between the user's interocular distance and the face width;
When the user's face is detected and the user's eyes are not detected, the computer executes a process including a step of calculating the user's position based on the detection result of the user's face and the relation information. program.

Images