JP5995217B2 - A method to detect an ellipse that approximates the pupil - Google Patents

Description

  The present invention relates to a method for detecting an ellipse that approximates a pupil part, and more specifically, in an eye movement measurement system, an ellipse that approximates a pupil part is detected from an eyeball image captured by a camera using an image processing apparatus. Regarding the method.

  Conventionally, a gaze position detection system that can measure a gaze position is known (for example, refer to Patent Documents 1 and 2), and for the purpose of elucidating brain function, research using monkeys in anesthesia behavior as experimental animals, It is used for psychological research related to human vision, medical examination, and applied research to man-machine interface.

  FIG. 1 shows a conventional gaze position detection system. A chair 2 that can fix the head of the subject 1 and a display 3 as a visual stimulus presentation device are installed, and a CCD camera 4 is placed at a position where the pupil of the subject can be photographed. An image signal (NTSC signal) output from the CCD camera 4 is converted into a digital image signal by a video input device 5 and input to a computer 6 which is an image processing device. Further, in order to photograph the pupil darkly, illumination is performed by the infrared LED 7. With such a configuration, since the gaze position is measured by noninvasive measurement, the burden on the subject is small. In addition to the line-of-sight position, the pupil diameter used for psychological experiments can be measured. As long as the pupil can be imaged, the camera can be installed at an arbitrary position.

  As shown in FIG. 2, when the pupil is photographed by the CCD camera 11, the line-of-sight position can be measured under several preconditions (for example, see Non-Patent Document 1). The pupil 12 is considered as a thin disk existing in the space. This disk (pupil 12) is included in a surface (iris) obtained by cutting a part of a sphere (eyeball 13) that rotates around a certain point, and the center of the surface coincides with the center of the disk. Let the center coordinates of the disk be (Xc, Yc, Zc). The coordinates (rotation center coordinates) of the center of the sphere are (Xo, Yo, Zo). The distance from the center of the disk (Xc, Yc, Zc) to the center of the sphere (Xo, Yo, Zo) is R (rotation radius). The xy plane is parallel to the CCD camera surface. Here, the set coordinate system is a camera coordinate system. The normal vector (Vx, Vy, Vz) of the disk can be expressed as (Xc-Xo, Yc-Yo, Zc-Zo). This is a vector representing the line-of-sight direction.

  According to Non-Patent Document 1, in order to obtain the line-of-sight direction in the camera coordinate system, if the eyeball rotation center position (Xo, Yo), pupil center position (Xc, Yc), and pupil rotation radius (R) are obtained, A line-of-sight vector (Vx, Vy, Vz) indicating the line-of-sight direction can be obtained.

  Since the disk projected on the plane becomes an ellipse on the xy plane, the center of rotation projected on the plane exists on the extension of the short axis of the ellipse. Therefore, a plurality of pupil images are photographed, each pupil is approximated by an ellipse, and the position obtained by using the method of least squares as the intersection of the ellipse's short axis is defined as the center of eyeball rotation. Further, the ratio of the minor axis to the major axis of the ellipse changes as the eyeball rotates. If this relationship is expressed by using the angle θ formed by the xy plane and the line-of-sight vector, the pupil rotation radius can be obtained. Therefore, the average value obtained from a plurality of images is set as the pupil rotation radius. Further, the pupil is approximated by an ellipse by image processing, and the center coordinates of the ellipse are set as the pupil center position.

  In this way, the line-of-sight position of the subject can be calculated by converting the line-of-sight vector obtained in the camera coordinate system into the line-of-sight vector in the object coordinate system.

Japanese Patent No. 3726122 Japanese Patent No. 3834636

Junji Matsuda, Takeshi Nagami, Shigeru Yamane, "Development of Gaze Position Measurement System", IEICE Technical Report, Vol.100, No.47, TL2000-2, p9-p16, (2000)

  The conventional line-of-sight position detection system fetches the NTSC signal from the CCD camera 4 and has the following problems.

  1) Since the conventional video input is an interlace of 60 Hz, the exposure time of one screen was 16.7 ms at the maximum. Therefore, with a fast eye movement such as 700 deg / sec, it moves as much as 11.69 degrees during exposure, and as a result, a blurred image is recorded. Conventionally, even if the position of a stationary eyeball can be measured, it is impossible to measure the eye movement.

  2) Since the conventional video input is interlaced and there is a shift in the position of the pixels of the even and odd images, when all the images are handled, separate processing is performed for each of the even and odd images. There was a need to do.

  3) The conventional gaze position detection system has a problem that the aspect ratio is changed by the video input device 5 during A / D conversion. As described above, an elliptical shape is handled in image processing. However, if the aspect ratio changes, accurate line-of-sight detection cannot be performed, and thus processing for correcting the aspect ratio is necessary.

  Therefore, in the eye movement measurement system according to the present invention, a progressive digital image signal is directly input to a computer using a high-speed digital camera to perform digital image processing. An object of the present invention is to improve the detection accuracy of an ellipse that approximates a pupil part from an eyeball image captured by a high-speed digital camera in order to improve the accuracy of eye-gaze position detection in performing high-speed image processing. It is in.

  In order to achieve such an object, one embodiment of the present invention is a method for detecting an ellipse that approximates a pupil part from an eyeball image captured by a camera in an eye movement measurement system. A first step of inputting a digital image signal of an eyeball image captured by the camera and storing it as a luminance signal, and a histogram of the number of pixels for each luminance value from all the pixels of the image to be processed from the luminance signal. A second step of creating and a step of setting a threshold value from the luminance value of the histogram, wherein the pupil detected from the image one frame before the image to be processed is approximated by an ellipse and calculated from the equation of the ellipse Calculating the reference number of pixels which is the area of the ellipse, and accumulating the number of pixels from the luminance value 0 to the histogram of the image to be processed; A third step of setting a luminance value when the number of cels exceeds the threshold value as a threshold value, and a fourth step of extracting a pupil part from pixels having a luminance value equal to or lower than the threshold value and approximating the extracted pupil part with an ellipse. It is characterized by providing.

Another embodiment of the present invention relates to a method for detecting an ellipse that approximates a pupil part from an eyeball image captured by a camera in an eye movement measurement system in an eye movement measurement system, and digitally processing the eyeball image captured by the camera. A first step of inputting an image signal, creating a reduced image from the original image, and converting it to a binarized reduced image, extracting a pupil portion from the binarized reduced image, and extracting the pupil The second step of approximating the part with an ellipse and the ellipse approximated in the second step are superimposed on the original image, the vicinity of the edge of the approximated ellipse is probed, and the brightness of the pixel near the predetermined threshold A third step of recording coordinates with a point having the greatest change in pupil as a pupil edge, and a fourth step of calculating an ellipse equation using the coordinates of the pupil edge of the original image, the third step Do the processing To explore the neighborhood of the ellipse using the equation of the ellipse of the preceding frame image of the image, which is approximated by the first and second step only if it can not extract a pupil portion in the previous frame image ellipse The following formula is used.

  Another embodiment of the present invention relates to a method for detecting an ellipse that approximates a pupil part from an eyeball image captured by a camera in an eye movement measurement system in an eye movement measurement system, and digitally processing the eyeball image captured by the camera. A first step of inputting an image signal, binarizing and extracting a pupil part, approximating the extracted pupil part with an ellipse, and n line segments parallel to an arbitrary line segment intersecting with the ellipse Set, and for the midpoint of the coordinates of the two intersections with the extracted pupil edge on each line segment, the angle of the line segment connecting any two midpoints is calculated, and the total n * (n−1 ) / 2 The mode value of two angles is compared with the mode value of n-1 angles with respect to a specific midpoint. A second step of excluding the coordinates of the edge of the pupil on the line segment including the midpoint; Characterized by comprising a third step of performing elliptical approximation by using a plurality of the pupil edge coordinates left by 2 steps.

  According to the present invention, a pupil portion can be extracted regardless of noise by using a threshold value detected from an image one frame before an image to be processed.

  Further, since the vicinity of the ellipse is searched using the ellipse formula of the image one frame before the image to be processed, the edge of the pupil can be detected with high accuracy in a small search range.

  Further, since the coordinates of the edge of the pupil on the n line segments used when performing the ellipse approximation are set as a set of two points, one point is wasted when noise is included in one point. Since it becomes a point, among the data for performing elliptic approximation, the data discarded can be reduced.

It is a figure which shows the conventional gaze position detection system. It is a schematic diagram for demonstrating the principle which calculates a gaze position. It is a figure which shows the eye movement measurement system concerning one Embodiment of this invention. It is a figure which shows the method of the pupil extraction concerning one Embodiment of this invention. It is a figure which shows the 1st step in the extraction method of the edge of a pupil. It is a figure which shows the 2nd step in the extraction method of the edge of a pupil. It is a figure which shows the 2nd step in the extraction method of the edge of a pupil. It is a figure for demonstrating the method of elliptical approximation. It is a figure for demonstrating the method of elliptical approximation. It is a figure for demonstrating the method of removing noise in elliptical approximation.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. As described above, in order to obtain the pupil center position, the pupil portion is extracted from the acquired eyeball image, and the extracted pupil is approximated by an ellipse to obtain the center coordinate of the ellipse. In the present embodiment, a method for detecting a pupil accurately from an image acquired from a high-speed digital camera and approximating an ellipse will be described in detail.

  FIG. 3 shows an eye movement measurement system according to an embodiment of the present invention. A chair 12 that can fix the head of the subject 11 and a display 13 as a visual stimulus presentation device are installed, and a high-speed digital camera 14 is placed at a position where the pupil of the subject can be photographed. A digital image signal output from the high-speed digital camera 14 is input to a computer 16 that is an image processing apparatus.

  Further, by illuminating the subject with the infrared illumination device 17, it is possible to photograph the pupil black and the iris brightly. By attaching an infrared transmission filter or a visible light blocking filter to the high-speed digital camera 14, disturbance of external light can be suppressed and an image can be stably acquired.

  The high-speed digital camera 14 is a digital camera equipped with a semiconductor image sensor such as a high-speed CCD camera or CMOS camera. The high-speed digital camera 14 outputs a 334 Hz progressive digital image signal. The exposure time of one screen is 3 ms, and even when the eye movement is very fast (700 deg / sec), the eye movement during the exposure corresponds to a movement of 2.1 degrees, so that the eye movement can be measured. Further, since it is a progressive method, it is not necessary to perform a process for correcting the pixel position, and the data of each pixel of the image sensor is transferred to the computer 16 as it is, so that the aspect ratio does not vary.

  As the infrared illumination device 17, an infrared LED or a halogen projector equipped with an infrared transmission filter is used.

(Extraction of pupil part)
The computer 16 that is an image processing apparatus receives the digital image signal output from the high-speed digital camera 14 and creates a histogram of the number of pixels for each luminance value (0-255) from all the pixels of one image. The area of the pupil is obtained from the image one frame before, and the threshold is estimated by comparing the area with the histogram. The input image is binarized using the estimated threshold value, and a pupil portion is extracted from one image.

  FIG. 4 shows a pupil extraction method according to an embodiment of the present invention. FIG. 4A is a screen of an eyeball portion photographed by the high-speed digital camera 14, and a histogram created from this one screen is shown in FIG. 4B. The horizontal axis represents luminance (0-255), and the vertical axis represents the number of pixels. Since it is known that the pupil part is in the first peak part from the lower luminance value, the optimum threshold is the point A that becomes the minimum luminance value after exceeding the first peak. Further, since the area occupied by the entire pupil is equal to or less than a certain value, an optimum threshold can be set by setting the valley to the right of the peak in the range as the threshold.

  From the binarized image, a pixel having a luminance value equal to or lower than the threshold value A is extracted, and pixel labeling (numbering is performed for each pixel block) is performed. In the labeling, a pixel having a size of 1 pixel is ignored and connection is performed only in the vertical direction. Note the mass with the largest area. The image obtained by extracting the pupil part in this way is shown in FIG. 4C, and it can be seen that the pupil part shown in FIG. 4A is extracted.

  However, when photographing, a background image may be reflected in the pupil, or a histogram as shown in FIG. 4D may be obtained due to the influence of system noise. At this time, if the point B, which is the minimum value beyond the first peak, is set as a threshold value, the pupil cannot be accurately detected. In particular, in the high-speed digital camera 14, since the exposure time is short, the image tends to be dark, and it is difficult to determine the threshold value from the histogram.

Therefore, the accuracy of pupil part extraction is improved by the following method.
1) Reference is made to an ellipse equation detected from an image one frame before the image to be processed (FIG. 4B).
2) The area of the ellipse (reference pixel number) is calculated from the ellipse equation.
3) Regarding the histogram of the image to be processed (FIG. 4D), the number of pixels is accumulated from the luminance value 0, and a point C exceeding the area of the ellipse calculated in 2) is set as a threshold value.

  In the high-speed digital camera 14, the interval between the previous image and the next image is sufficiently shorter than the time for the pupil size to change in response to light, so detection is performed from the image one frame before the image to be processed. By using the area of the ellipse, the pupil portion can be extracted regardless of noise. Accordingly, when the brightness of the shooting location changes, even when the background image is reflected in the pupil, it is possible to reliably cut out only the pupil portion.

  In the conventional threshold determination method, an incorrect value does not appear only in a direction where the luminance is low, and it may appear in a direction where the luminance is high. When the ellipse cannot be approximated from the image one frame before and the reference pixel number cannot be obtained, 1) obtained by a conventional threshold determination method, 2) use a preset fixed value, and 3) perform processing. Three methods of using a threshold set in the most recent frame before the image can be considered. For example, for use in a laboratory where the brightness does not change greatly, the fixed value of 2) can be used.

  In addition, on the screen of the photographed eyeball part, a certain area such as a shadow of the edge of the eyelid may have the same luminance as the pupil. Therefore, when calculating the threshold value, the pupil portion can be detected with higher accuracy by using a value obtained by adding a preset constant value as the threshold value.

(Extraction of pupil edge)
In the above-described extraction of the pupil portion, if all the pixels on one screen are labeled, the processing load is large. Therefore, two-stage detection is performed for speeding up.

  FIG. 5 shows a first stage in the method for extracting the edge of the pupil. The first stage is the above-described extraction of the pupil portion. At this time, for example, a reduced image of 40 × 30 pixels is created from an original image of 320 × 240 pixels, and the reduced image is binarized. An ellipse equation is calculated from the binarized reduced image by the method of least squares, and a pupil portion is extracted.

  FIG. 6 shows a second stage in the pupil edge extraction method. The ellipse equation obtained above is superimposed on the original image of 320 × 240 pixels (FIG. 6A). The vicinity of the edge of the ellipse calculated by the ellipse equation is searched, and the coordinates are recorded with the point where the change in the brightness of the pixel is the maximum near the threshold obtained by extraction of the pupil part as the edge of the pupil (FIG. 6 ( b)). An ellipse equation is calculated using the coordinates of the edge of the pupil thus obtained. Such two-stage processing is suitable for high-speed processing because the memory in which images are stored can be measured with few accesses.

  However, since this method detects using a reduced image, there is a large error between the ellipse locus calculated by the ellipse equation and the pupil edge of the original image. For this reason, it is necessary to secure a certain amount of exploration range, and processing takes time. For example, as shown in FIG. 7A, when the shadow of the eyelid appears, if a reduced image is generated and binarized, the result is as shown in FIG. 7B. When the ellipse equation is obtained in the reduced image, an ellipse that protrudes greatly from the pupil is detected as shown in FIG. In the second stage, even if the vicinity of the edge of the ellipse calculated by the ellipse equation is searched, a phenomenon that the edge of the pupil cannot be found occurs (FIG. 7D).

  Therefore, in the second stage, when searching for the vicinity of the edge of the ellipse, the ellipse formula calculated from the image one frame before the image to be processed is used. In the high-speed digital camera 14, the shooting interval between the previous image and the next image is sufficiently smaller than the time when the pupil size changes in response to light, and is sufficiently shorter than the pupil position moving time due to eye movement. Therefore, by using the ellipse formula obtained from the previous image, the edge of the pupil can be detected with high accuracy in a small search range.

  In addition, since processing such as labeling in the first stage is unnecessary, higher speed processing is possible. Furthermore, since the processing is performed asymptotically, it is possible to cope with a case where the shadow of the pupil increases little by little like a blink.

(Delivery of pupil edge noise)
In the first-stage pupil part extraction described above, ellipse approximation is performed using the least square method (see, for example, Non-Patent Document 1). The ellipse approximation method will be described in detail with reference to FIG. Further, as shown in FIG. 9A, when a background image is reflected in the pupil part, the pupil part is detected as shown in FIG. 9B. At this time, if an ellipse equation is obtained using all the points of the edge of the pupil, an ellipse thinner than the actual one is detected as shown by the white line in FIG. Therefore, using the geometrical properties of the ellipse, points that do not exist on the circumference of the ellipse (points other than the edge of the pupil) are removed by the following method.

  1) Consider three parallel straight lines l, m, n that intersect an ellipse. Let l and n be equidistant from m. The intersections of the ellipse and the straight line l are a and b, and the intersections of the straight line n are c and d. Let o be the midpoint of the intersection of the ellipse and the straight line m. If the middle point of the line connecting the middle points of a and b and the middle points of c and d is defined as o ', o' has a property that it overlaps o (FIG. 8A).

  2) A point corresponding to o′i (i = 1−N) of a set of N parallel lines at equal distances is obtained from a straight line m drawn substantially at the center of the pupil of the original image. The obtained points are distributed on the straight line m. When the number of erroneously detected edges is sufficiently small, the position where the most points o 'are collected corresponds to the position o. Since the points away from the position include points that are not on the locus of the ellipse, they are excluded (FIG. 8B).

  However, since the set of N parallel lines is a set of four points, only one point has noise and the other three points cannot be used. If the circumference of the quarter ellipse is hidden in the shape as shown in FIG. 8C, it cannot be detected as an ellipse.

With reference to FIG. 10, a method for removing points other than the edge of the pupil in order to perform an accurate elliptical approximation will be described.
1) A plurality of line segments that are parallel to the x axis and intersect with the ellipse are drawn with respect to the edge of the pupil on the xy coordinates (Si to Si + 3). The left intersection on the edge of the pupil and the i-th line segment Si is xLi, the right intersection is xRi, the y coordinate is yi, and the middle point between xLi and xRi is xi.
2) The number of line segments Si is n (2n intersections). If it is a true ellipse, the midpoint (xi, yi) of the n line segments Si has a property of being on one straight line (indicated by a one-dot chain line in FIG. 10).
3) An arbitrary two points (xi, yi), (xj, yj) are taken out of n middle points (x, y), and an angle θ of a line segment connecting the two points is yi> yj.
θ (degree) = arctan ((yi-yj) / (xi-xj)) * 180 / π (Formula 1)
Is required.
4) n-1 inclination angles are obtained for yi, and n * (n-1) / 2 angles are obtained as a whole. If it is a true ellipse, all angles θ are the same.
5) Since the pupil edge of the original image has some degree of dispersion, the overall mode value of n * (n-1) / 2 angles and the maximum of n-1 angles with respect to yi. Find the frequent value. When both are compared, if they are not within a certain range, it is determined that they contain noise. That is, the coordinates of the intersection with the edge of the pupil on the line segment Si including yi that is not within a certain range are excluded from the coordinates for approximating the ellipse.

  As shown in FIG. 10, it is determined that the line segment Si + 1 and the line segment Si + 2 are within a certain range with respect to the line segment Si, and the line segment Si + 3 is within a certain range with respect to the line segment Si. It is determined that it does not fall within, and is not used when performing ellipse approximation. According to this method, since two points are set as one set, one point is wasted when noise is included in one point, and therefore, the data for performing elliptic approximation is discarded. Data can be reduced.

  Therefore, even in the case shown in FIG. 8C where a part of the pupil is hidden, an improvement in ellipse detection accuracy is expected. In addition, by comparing the size of the ellipse detected by ellipse approximation with the edge of the ellipse measured by extracting the edge of the pupil, it is possible to determine how much wrinkle is on the pupil it can.

  According to this embodiment, since a high-speed digital camera is used, it is possible to measure eye movement. In conventional NTSC video cameras, both blinking and fast eye movements cannot be discriminated because the image is blurred, but both can be discriminated by using a high-speed digital camera. In addition, since a digital image signal can be obtained with a size that depends on the number of pixels of a CCD / CMOS used in a high-speed digital camera, it can be easily obtained from a high-definition image to an image of any size. be able to. Furthermore, the interval at which images are acquired can be varied within the range allowed by the transfer speed of the CCD / CMOS performance and the transmission path (for example, IEEE1394b / USB3.0).

  In this embodiment, the case where the movement of one eyeball is measured by one camera has been described, but the movements of both eyeballs can also be measured by two cameras.

DESCRIPTION OF SYMBOLS 1,11 Subject 2,12 Chair 3,13 Display 4 CCD camera 5 Video input device 6,16 Computer 7,17 Infrared illumination device 14 High-speed digital camera

Claims (6)

In an eye movement measurement system, in a method of detecting an ellipse that approximates a pupil part from an image of an eyeball imaged by a camera by an image processing device,
A first step of inputting a digital image signal of an eyeball image captured by the camera and storing it as a luminance signal;
A second step of creating a histogram of the number of pixels for each luminance value from all the pixels of the image to be processed from the luminance signal;
A step of setting a threshold value from the luminance value of the histogram, wherein the pupil detected from the image one frame before the image to be processed is approximated by an ellipse, and is an area of the ellipse calculated from the equation of the ellipse A third step of calculating a reference pixel number, accumulating the number of pixels from the luminance value 0 with respect to the histogram of the image to be processed, and setting the luminance value when the reference pixel number is exceeded as a threshold value; ,
And a fourth step of extracting a pupil part from a pixel having a luminance value equal to or lower than the threshold and approximating the extracted pupil part with an ellipse.   In the third step, when the reference pixel number cannot be calculated from an image one frame before the image to be processed, the minimum luminance after exceeding the first peak from the lower luminance value of the histogram The method according to claim 1, wherein the threshold value is any one of a value, a preset fixed value, or a threshold value set in a most recent frame before the image to be processed. In an eye movement measurement system, in a method of detecting an ellipse that approximates a pupil part from an image of an eyeball imaged by a camera by an image processing device,
A first step of inputting a digital image signal of an eyeball image captured by the camera, creating a reduced image from the original image, and converting the reduced image into a binarized reduced image;
A second step of extracting a pupil part from the binarized reduced image and approximating the extracted pupil part with an ellipse;
The ellipse approximated in the second step is overlaid on the original image, the vicinity of the edge of the approximated ellipse is explored, and the point where the brightness change of the pixel is the maximum near the predetermined threshold is the edge of the pupil A third step of recording coordinates as
A fourth step of calculating an elliptical equation using the coordinates of the edge of the pupil of the original image,
The third step searches the vicinity of the ellipse using the ellipse equation of the image one frame before the image to be processed, and the first step only when the pupil portion cannot be extracted from the image one frame before. And using an elliptic equation approximated in the second step. In an eye movement measurement system, in a method of detecting an ellipse that approximates a pupil part from an image of an eyeball imaged by a camera by an image processing device,
A first step of inputting a digital image signal of an image of an eyeball imaged by the camera, binarizing and extracting a pupil part, and approximating the extracted pupil part with an ellipse;
N line segments parallel to an arbitrary line segment intersecting with the ellipse are set, and arbitrary two midpoints are set with respect to a midpoint of coordinates of two intersection points with the extracted pupil edge on each line segment. Calculate the angle of connecting line segments and compare the overall mode value of n * (n-1) / 2 angles with the mode value of n-1 angles for a specific midpoint And, if not within a certain range, a second step of excluding the coordinates of the edge of the pupil on the line segment including the specific midpoint;
And a third step of performing an ellipse approximation using a plurality of pupil edge coordinates left by the second step.   5. A computer program comprising computer-executable instructions for causing the image processing apparatus to execute the method according to claim 1. A camera that captures the subject's eyeballs;
An illumination device for irradiating the eyeball of the subject with infrared rays;
An image processing apparatus that detects an ellipse that approximates a pupil part from an image of an eyeball that is connected to the camera and captured by the camera, and that performs the method according to any one of claims 1 to 4 An eye movement measurement system comprising: a device.