JPH04225478A - line of sight detection device - Google Patents

Abstract

PURPOSE:To obtain the center position even if the whole image of an iris cannot be obtained due to a shadow and insert-photographing, etc., by constituting the device so that an edge detecting part executes an edge extraction of an eye area, and an iris position detecting part approximates an extracted edge to a circle and sets the center of the circle to the iris center. CONSTITUTION:This device is provided with an image input part 1 for inputting an image of the face, an edge detecting part 2 for binarizing this inputted image and detecting an edge, and an iris position detecting part for detecting the iris center from this detected edge. In such a state, an image of the face is inputted from the image input part 1 and the edge is detected by the edge detecting part 2, and the edge detected by the iris position detecting part 3 is approximated by a circle and the center of the circle is set as the iris center.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight detection device that detects line-of-sight by image processing.

0002

2. Description of the Related Art In recent years, computer systems have progressed rapidly, and systems with a variety of functions have become widespread in a wide range of fields. Along with this, there has been a desire to realize a human interface that allows even non-experts to easily handle complex systems. As one means of achieving this, a method has been proposed in which the line of sight is used as a pointing interface. As the saying goes, ``the eyes speak as much as the mouth,'' the gaze plays an important role in communication between people. Therefore, it is believed that if a machine can detect the line of sight of a human, it will enable smoother communication between machines and humans. Application examples of this include remote control, which detects the movement of a person's line of sight when they follow the movement of an object on an image, and moves the camera in accordance with that movement; and detection of the line of sight of a person on the receiving side in image communication. By transmitting information in high definition for the part being viewed, and compressing information for other parts, discomfort caused by image deterioration can be alleviated when transmitting images during video conferences and the like. It can also be used as a substitute for a mouse.

[0003] However, the conventional line of sight detection method generally involves the observer wearing glasses, a sensor, or the like. Image processing methods also detect line of sight by extracting marks placed on facial feature points. When using the line of sight for interface applications, adding detection means to the observer (operator) side is troublesome for the observer and should be avoided. For this purpose, it is effective to detect the line of sight by extracting facial feature points through image processing, and several methods have been proposed for this purpose. Typical methods include a method that uses corneal reflection images, the direction of the face, and the iris (for Japanese people, the iris)
There is a method of detection based on the position of However, the former method has problems, such as requiring a reference light to obtain a corneal reflection image and making it difficult to achieve high precision because the distance between feature points is very small. In addition, the latter method has the problem that it is difficult to extract stable feature points only by image processing because the eye area changes depending on the movement of the skin, shadows, etc. of the feature points. Therefore, both methods require image processing technology to accurately extract feature points. Moreover, these processes must be performed at high speed so that the movement of the line of sight can be followed.

FIG. 6 shows an example of a device that implements such a conventional line of sight detection method. In this method, the face is first photographed using a TV camera, and feature points (the corners of the eyes, the corners of the eyes,
The center of the lips and the iris of the eye) are extracted by image processing. The three feature points at the corners of both eyes and the center of the lips are projected onto the vertices of a triangular plane, and the direction and three-dimensional position of the plane (face) is determined as shown in FIG. Note that in FIG. 7, the image plane is an image captured by a camera, and f is the focal point of the camera. The eyeball center position is estimated by multiplying this by an appropriate conversion parameter (the conversion parameter is determined in advance by a calibration method). Then, the center position of the iris is determined, and a vector connecting the center of the eyeball and the center of the iris, that is, the direction of the line of sight is detected. Finally, the gaze point where the line-of-sight vector intersects with the display screen is converted into display coordinates. Among the above processes, extraction of each feature point is performed by edge extraction, pattern matching, etc.

[0005] As shown in FIG. 8, the conventional method for extracting the iris involves detecting the edge of the eye area cut out using some method, extracting a closed loop within the edge as the iris outline, and finding its center of gravity. . However, the edges rarely form a simple closed loop due to shadows from the eyelids and eyelashes, lighting reflected in the iris, etc. Further, the iris of the eye is often partially blocked by the eyelid, and the center of gravity of the iris on the image does not necessarily coincide with the center position of the iris. Therefore, in order to detect line of sight using this method,
Photographs must be taken with eyes wide open and under optimal lighting conditions. Therefore, when considering the application of line of sight detection to interfaces, an eye extraction technique that does not depend on lighting conditions is required.

[0006]

[Problem to be solved by the invention] In interface applications, the illumination conditions and background of input images are not necessarily optimal for line-of-sight detection, and under such conditions, the shadows and images that appear on the eyes are sufficient. In many cases, edges cannot be detected. For this reason, there is a problem in that if sufficient edges cannot be extracted, the iris cannot be extracted. Furthermore, in cases where the iris is partially hidden by the eyelid, the center of gravity of the visible part of the iris is set as the iris center, so there is a problem in that the true iris center position cannot be obtained.

The present invention has been made in view of the above-mentioned problems, and includes an edge extraction method that is less susceptible to effects such as shadows and reflections, and an edge extraction method that is less susceptible to effects such as shadows and reflections, and an edge extraction method for extracting the iris of the eye for line of sight detection. The aim is to detect the line of sight with high accuracy by introducing a method that can determine the center position of the iris even from a distance.

[0008]

[Means for solving the problem] FIG. 1 is a diagram of the principle of the present invention,
(a) shows a configuration diagram of the present invention, (b) is an explanatory diagram of a method for determining an edge from an original image, and (c) is a diagram for explaining a method for determining the center of an iris from an edge. First, the original image shown in (a) of (b) is input from the image input section 1, and then edges are detected by the edge detection section 2. The edges are obtained by binarizing the original image in (a) using a threshold value, and this threshold value is set as a threshold value that matches the eye boundary shown in (b) and the edge shown in (c) as much as possible. A threshold value is used so that the range that is an edge and a boundary shown in d) becomes large, that is, the boundary becomes an edge. To find the center of the iris from the edges thus obtained, a circular arc is generated from the radius of curvature of each edge line segment, and the point where this circular arc intersects the most is determined as the center of the iris. As shown in (c), the arcs generated in the iris part intersect at almost one point, but the arcs generated in other parts do not intersect at one point. Note that the edge direction refers to the direction of the radius of curvature of the edge line segment.

When the edge detection section 3 performs edge extraction,
A threshold value is determined using a threshold selection method using a minimum value filter, and edges are obtained by binarizing using this threshold value. The minimum value filter is a filter that uses the minimum value of pixels (for example, 3×3 pixels) within a mask as the value of the center pixel within the mask. In the case of a grayscale image, it has a graphic reduction effect because the maximum value part is reduced and the minimum value part is enlarged.

[0010] Furthermore, when selecting a threshold value using a minimum value filter, the evaluation function is (number of pixels that are an edge and a boundary)/(number of pixels that are an edge or a boundary). (number of pixels) / (number of pixels that are the boundary)
The threshold value is determined so that this evaluation function is maximized.

[0011]

[Operation] With the above configuration, even if a part of the iris is shaded or reflected by an eyelid and the edge only represents a part (partial arc) of the iris shape (circle), the center position of the iris can be determined. You can almost certainly ask for it. Also,
By determining the threshold value for binarization using a minimum value filter, isolated local maximum points and the like are removed and edges suitable for representing the shape of the iris are obtained. Furthermore, when determining a threshold value using a minimum value filter, edges with a high SN ratio can be extracted by selecting the threshold value such that the above-mentioned evaluation function is maximized.

0012

Embodiments Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 2 is a configuration diagram of an embodiment of the present invention. The part shown in this figure is an image processing device that detects the iris position (iris position), and is part of the line of sight detection device. The overall view of the line of sight detection device is the same as that shown in FIG. In FIG. 1, 10 is an image capturing section such as a TV camera, 11 is an image input section that processes images captured by the image capturing section, and 12 is this image input section 1.
An edge detecting section 13 performs edge detection by binarizing the original image outputted from the edge detecting section 12 using a threshold value, and a pupil position detecting section 13 detects the center position of the iris from the edge detected by the edge detecting section 12.

FIG. 3 shows the configuration of the image input section 11. The image input unit 11 cuts out an image of a small area including one eye from the face image to form an original image. The original image is matched by a template matching section 111, and an eye region is cut out from the face image by an eye region cutting section 112. Since line-of-sight detection deals with continuous images, a template for the eye area of the previous frame is extracted by the template unit 113 and used as a template for the next frame.

FIG. 4 shows the configuration of the edge detection section 12. a differential image creation unit 121 that creates a differential image from the original image of the eye area input from the image input unit 11; and a minimum value filter 12 that reduces the original image using a minimum filter to generate a reduced image.
2, and an edge histogram calculation unit 123 that creates an edge histogram, a boundary and edge histogram, and a boundary length histogram from the original image, differential image, and reduced image, respectively.
, a boundary edge histogram calculation unit 124 , a boundary length histogram calculation unit 125 , a threshold determination unit 126 that calculates an evaluation function for all combinations of thresholds using these three histograms, The edge determining section 127 scans three images (original image, differential image, and reduced image) and determines a threshold value to extract a point.

FIG. 5 shows the configuration of the iris position detection section 13. An edge direction calculation unit 131 calculates an edge direction, which is the direction of the radius of curvature of an edge line segment, from data from the image input unit 11 and the edge detection unit 12, and generates an edge arc in this edge direction with the radius of curvature of this edge line segment. Arc generator 13
2, and an iris position determination unit 133 that determines the center position of the iris by setting the point where the arcs generated by the arc generation unit 132 intersect most often as the center of the iris.

Next, the operation will be explained. Image input section 1
1, an image of a small region including one eye is cut out from a face image and output. In this embodiment, the eye area is cut out by template matching. Since line-of-sight detection deals with continuous images, it is possible to easily cut out the eye area by using the eye area of the previous frame as a template. Next, the edge detection unit 12 calculates an edge histogram, a boundary and edge histogram, and a boundary length histogram from the original image of the eye area input from the image input unit 11, its differential image, and the reduced image reduced by the minimum value filter. create. Then, in the threshold determining unit 126, 3
Using two histograms, evaluation functions are calculated for all combinations of thresholds, and thresholds are determined. Points that are edges and boundaries are extracted by determining a threshold value while scanning three images (original image, differential image, and reduced image) in the edge determination unit 127 .

Next, in the iris position detecting section 13, an edge direction calculating section 131 calculates the edge direction of the edge point detected by the edge detecting section 12 from the original image. An arc generator 132 generates an arc in the edge direction. The radius of the arc is within the range of ±α pixels of the radius r of the iris obtained in the previous frame (the size of the iris does not change much between successive frames). Generate an arc for each r in this range,
Accumulate the trajectory of the arc for each r (3-dimensional array (r, x
, y)). When arcs have been generated for all edge points, the iris position determination unit 133 determines the point where the arcs intersect the most, that is, the center position of the iris. Then, the radius r that gives the center position determines the radius range for generating the circular arc in the next frame.

Next, a method for selecting a threshold value will be explained. Binary ratio processing is one of the basic methods for extracting the area occupied by a target object from an image, but in order to accurately extract the target object under various conditions, it is necessary to Various methods have been proposed to appropriately set the binarization threshold depending on the characteristics of each image, such as the shape of the object. Most of them express the characteristics of an image using a density histogram, and select a threshold value based on the shape of the histogram. This method has the advantage of being able to determine the threshold with a relatively small amount of calculation, but because the density histogram does not include information about the shape of the object such as boundaries and edges, the area of the object and the background It is often not possible to accurately extract the object when there is a large difference in area or when the shadow of the object is noticeable.

In order to deal with this problem, attempts have been made to define an evaluation function that expresses properties related to the shape of a binarized image, and to select a threshold value to optimize the evaluation function. For example, a threshold value may be selected so that the image obtained by binarization has the simplest shape possible, or edges may be detected from the original image in advance, and the boundary points of the image obtained by binarization and the original image may be detected in advance. A method has been proposed to select the threshold to best match the edges. In the latter method, the degree of matching between edges and boundaries is evaluated based on the ratio of the number of pixels that are edges and boundaries to the number of pixels that are edges or boundaries. In this case, the evaluation function is expressed by the following equation.

[0020]

[Math 1]

[0021] Then, a threshold value is selected so that this evaluation function is maximized. The present invention converts this evaluation function into two dimensions (
s, t), and this is further improved so that edges and boundaries match and edges with a high SN ratio are extracted, and its evaluation function is expressed by the following equation.

[0022]

[Math 2]

The first term in equation (2) represents the proportion of coincidence between edges and boundaries, and usually only this term is used as the evaluation function. In this case, the threshold value is determined in such a way that the boundary and edge ridges match. However, the edge strength of the image processed by line-of-sight detection differs depending on the part even on the ridge line. Furthermore, edges are often blunt, and as the threshold value decreases, the number of edge points increases rapidly, and the degree of coincidence between edges and boundaries decreases. For this reason, in the evaluation function of equation (1), the threshold is determined so as to extract only the ridge line with the highest edge strength, and sufficient edge points cannot be obtained. Therefore (2)
As shown in the second term of the equation, by multiplying by an evaluation function that increases the proportion of boundaries that are both edges and boundaries,
The evaluation function is set so that the degree of coincidence between the boundary and the edge is high and the number of points that are both edges and boundaries is large. When the original image and the differential image are each binarized using the determined thresholds s and t, points that are edges and boundaries are extracted as edge points (FIG. 1(b) d). Here, the S/N ratio is improved by treating only edges and boundary points as edges instead of all edge points.

[0024]

As is clear from the above description, the present invention extracts the edges of the eye region from the original image, sets the iris part as a circle, and finds the point where the arcs generated by the radius of curvature of the edge line segment have the most intersections. Since the center of the iris is used, even if the iris is partially covered by the eyelids, the center can be detected correctly.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing the principle of the present invention.

FIG. 2 is a configuration diagram of an embodiment of the present invention.

FIG. 3 is a configuration diagram of an image input section.

FIG. 4 is a configuration diagram of an edge detection section.

FIG. 5 is a configuration diagram of an eye iris position detection section.

FIG. 6 is a configuration diagram of a line of sight detection device.

FIG. 7 is a diagram illustrating the direction and position of a face.

FIG. 8 is a diagram illustrating a conventional iris center extraction method.

[Explanation of symbols] 10 Image capturing unit 11 Image input unit 111 Template matching unit 112 Eye area extraction unit 113 Template unit 12 Edge detection unit 121 Differential image generation unit 122 Minimum value filter 123 Edge histogram calculation unit 124 Boundary/edge histogram calculation Section 125 Boundary length histogram calculation section 126 Threshold determination section 127 Edge determination section 13 Eye position detection section 131 Edge direction calculation section 132 Arc generation section 133 Eye position determination section

Claims (4)

[Claims] [Claim 1] An image input unit (1) for inputting a facial image.
), an edge detection unit (2) that binarizes this input image and detects edges, and an iris position detection unit (3) that detects the center of the iris from the detected edge, and the edge detection unit ( 2) A line-of-sight detection device characterized in that the edge extraction of the eye region is performed, and the iris position detection unit (3) approximates the extracted edge with a circle and sets the center of this circle as the iris center. 2. When approximating a circle from the edge extracted by the iris position detection unit (3), an arc is formed with the radius of curvature of the extracted edge line segment, and the point where this arc intersects the most is determined as the center of the iris. The line of sight detection device according to claim 1, characterized in that: 3. When the edge detection unit (2) performs edge extraction, it selects a threshold using a threshold selection method using a minimum value filter, and uses this threshold to
3. The visual line detection device according to claim 1, wherein the visual line detection device performs edge detection by performing value processing. 4. When selecting a threshold using the minimum value filter, the evaluation function is calculated as (number of pixels that are an edge and a boundary)/(number of pixels that are an edge or a boundary) (
4. The line of sight according to claim 3, wherein the threshold value is set such that the evaluation function is maximized by multiplying the number of pixels that are an edge and a boundary by the number of pixels that are an edge and a boundary. Detection device.