JP2024090620A - Subject identification device, gaze detection device, subject identification method, and subject identification program - Google Patents

Abstract

To specify a subject with high accuracy.SOLUTION: A subject specification device includes a plurality of light sources for emitting detection light to irradiate at least one eyeball of a subject with the light, an imaging part for imaging an image of the eyeball to which the detection light is emitted, a cornea curvature calculation part for detecting the position of a corneal reflection center showing the center of corneal reflection in the eyeball with the detection light emitted from the picked-up image, and calculating a cornea curvature of the subject at the detected position, a table generation part for generating a cornea curvature table showing a relation between a position on the eyeball and the cornea curvature on the basis of cornea curvatures calculated at a plurality of different positions in the eyeball of the subject, and a collation part for collating a generation table being a newly generated cornea curvature table with a storage table being a cornea curvature table stored in a storage part in association with the subject specification information for specifying the subject, and specifying the subject on the basis of a collation result.SELECTED DRAWING: Figure 2

Description

The present invention relates to a subject identification device, a gaze detection device, a subject identification method, and a subject identification program.

A gaze detection device is known that emits detection light from a light source and irradiates the subject's eyeball, acquires an image of the eyeball irradiated with the detection light, calculates the pupil center and corneal curvature center based on the image of the pupil in the acquired image and the reflected image of the detection light, and detects the vector from the corneal curvature center to the pupil center as the gaze direction of the subject (see, for example, Patent Document 1).

Patent No. 4649319

In the above-mentioned gaze detection device, when the same subject performs detection multiple times, the data from the first detection may be used in the second and subsequent detections. For example, if the subject and the first data of the subject are associated with each other and the subject can be automatically identified, the first data can be automatically extracted. In this case, it is required to identify the subject with high accuracy.

The present invention has been made in consideration of the above, and aims to provide a subject identification device, gaze detection device, subject identification method, and subject identification program that are capable of identifying subjects with high accuracy.

The subject identification device according to the present invention includes a plurality of light sources that emit detection light and irradiate it on at least one eyeball of the subject; an imaging unit that captures an image of the eyeball irradiated with the detection light; a corneal curvature calculation unit that detects the position of the corneal reflection center indicating the center of the corneal reflection in the eyeball irradiated with the detection light from the captured image and calculates the corneal curvature of the subject at the detected position; a table generation unit that generates a corneal curvature table indicating the relationship between the position on the eyeball and the corneal curvature based on the corneal curvature calculated at a plurality of different positions on the subject's eyeball; and a matching unit that matches the generated table, which is the newly generated corneal curvature table, with a stored table, which is a corneal curvature table stored in a storage unit in association with subject identification information for identifying the subject, and identifies the subject based on the matching result.

The gaze detection device according to the present invention includes a plurality of light sources that emit detection light and irradiate it on at least one eyeball of a subject; an imaging unit that captures an image of the eyeball irradiated with the detection light; a corneal curvature calculation unit that detects the position of the corneal reflection center indicating the center of the corneal reflection in the eyeball irradiated with the detection light from the captured image and calculates the corneal curvature of the subject based on the detected position; a table generation unit that generates a corneal curvature table indicating the relationship between the position on the eyeball and the corneal curvature based on the corneal curvature calculated at different positions on the subject's eyeball; a storage unit that stores subject information, which is information on the subject, in association with the corneal curvature table; a comparison unit that determines whether the newly generated corneal curvature table corresponds to the memory table, which is the corneal curvature table stored in the memory unit; and a gaze processing unit that calculates the gaze of the subject. When it is determined that the generated table corresponds to the memory table, the gaze processing unit calculates the gaze of the subject using the subject information associated with the memory table.

The subject identification method according to the present invention includes emitting detection light from a light source and irradiating it onto at least one eyeball of a subject, capturing an image of the eyeball irradiated with the detection light, detecting from the captured image the position of a corneal reflection center indicating the center of the corneal reflection in the eyeball irradiated with the detection light, and calculating the corneal curvature of the subject at the detected position, generating a corneal curvature table indicating the relationship between the position on the eyeball and the corneal curvature based on the corneal curvatures calculated at multiple different positions on the subject's eyeball, comparing the newly generated corneal curvature table with a corneal curvature table stored in a storage unit in association with subject identification information for identifying the subject, and identifying the subject based on the comparison result.

The subject identification program of the present invention causes a computer to execute the following processes: emitting detection light from a light source and irradiating it on at least one eyeball of a subject; capturing an image of the eyeball irradiated with the detection light; detecting, from the captured image, the position of the corneal reflection center indicating the center of the corneal reflection in the eyeball irradiated with the detection light, and calculating the corneal curvature of the subject at the detected position; generating a corneal curvature table indicating the relationship between the position on the eyeball and the corneal curvature based on the corneal curvatures calculated at multiple different positions on the subject's eyeball; and comparing the newly generated corneal curvature table with a corneal curvature table stored in a storage unit in association with subject identification information for identifying the subject, and identifying the subject based on the comparison result.

The present invention makes it possible to identify subjects with high accuracy.

FIG. 1 is a diagram illustrating an example of a gaze detection device according to the present embodiment. FIG. 2 is a functional block diagram illustrating an example of a gaze detection device. FIG. 3 is a diagram showing a schematic diagram of the principle of detecting the gaze of a subject in the gaze detection device of this embodiment. FIG. 4 is a diagram showing a schematic diagram of a state in which detection light from a light source unit is irradiated onto the eyeball of a subject. FIG. 5 is a diagram showing a schematic diagram of a state in which detection light from a light source unit is irradiated onto the eyeball of a subject. FIG. 6 is a diagram showing an example of the positional relationship between the light source unit, the image capturing unit, and the eyeball of the subject. FIG. 7 is a diagram showing an example of image data of an eyeball captured by the imaging unit. FIG. 8 is a schematic diagram showing an example of a corneal curvature table used in this embodiment. FIG. 9 is a diagram illustrating an example of a process for generating a corneal curvature table. FIG. 10 is a diagram showing an example of an eyeball viewed from the front. FIG. 11 is a diagram illustrating an example of the interpolation process performed by the table generating unit. FIG. 12 is a diagram illustrating an example of gamma angle information. FIG. 13 is a diagram showing a schematic diagram of the principle of measuring the gamma angle by calibration. FIG. 14 is a diagram illustrating an example of a process for checking a generated table against a stored table. FIG. 15 is a flowchart showing an example of a subject identification method and a gaze detection method according to this embodiment. FIG. 16 is a flowchart showing an example of the gaze detection process in the gaze detection method according to the present embodiment.

Below, embodiments of a subject identification device, gaze detection device, subject identification method, and subject identification program according to the present invention will be described with reference to the drawings. Note that the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are replaceable and easy for a person skilled in the art, or those that are substantially the same.

In the following explanation, a three-dimensional global coordinate system is set up to explain the positional relationships of each part. The direction parallel to the first axis of the specified plane is the X-axis direction, the direction parallel to the second axis of the specified plane that is perpendicular to the first axis is the Y-axis direction, and the direction parallel to the third axis that is perpendicular to both the first and second axes is the Z-axis direction. The specified plane includes the XY plane.

[Subject Identification Device, Gaze Detection Device]
1 is a diagram showing an example of a gaze detection device 100 according to the present embodiment. The gaze detection device 100 according to the present embodiment detects the gaze of a subject and outputs the detection result. The gaze detection device 100 detects the gaze based on, for example, the position of the subject's pupil and the position of the corneal reflection.

As shown in FIG. 1, the gaze detection device 100 includes a display device 10, an image acquisition device 20, a computer system 30, an output device 40, an input device 50, and an input/output interface device 60. The display device 10, the image acquisition device 20, the computer system 30, the output device 40, and the input device 50 communicate data via the input/output interface device 60. The display device 10 and the image acquisition device 20 each have a drive circuit (not shown).

The display device 10 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). In this embodiment, the display device 10 has a display unit 11. The display unit 11 displays information such as images. The display unit 11 is substantially parallel to the XY plane. The X-axis direction is the left-right direction of the display unit 11, the Y-axis direction is the up-down direction of the display unit 11, and the Z-axis direction is the depth direction perpendicular to the display unit 11. The display device 10 may be a head-mounted display device. In the case of a head-mounted display, a configuration such as the image acquisition device 20 is arranged in a head-mounted module.

The image acquisition device 20 acquires image data of the subject's left and right eyes EB and transmits the acquired image data to the computer system 30. The image acquisition device 20 has a light source unit (light source) 21 and an imaging unit 22.

The light source unit 21 emits detection light to the subject's eye EB. The light source unit 21 has a first light source 21A and a second light source 21B arranged at different positions. The first light source 21A and the second light source 21B include an LED (light emitting diode) light source and can emit near-infrared light with a wavelength of, for example, 850 nm. The light source unit 21 emits detection light in synchronization with a frame synchronization signal of the imaging unit 22. The light source unit 21 may be composed of a single light source.

The imaging unit 22 generates image data by capturing images of the subject's left and right eyeballs EB. The imaging unit 22 has various cameras according to the method of detecting the subject's gaze. In the case of a method of detecting the gaze based on the position of the subject's pupil and the position of the corneal reflection, as in this embodiment, the imaging unit 22 has an infrared camera, and has an optical system that can transmit near-infrared light with a wavelength of 850 [nm], for example, and an imaging element that can receive the near-infrared light. The imaging unit 22 outputs a frame synchronization signal. The period of the frame synchronization signal can be, for example, 20 [msec], but is not limited to this. The imaging unit 22 is configured as a stereo camera having a first camera 22A and a second camera 22B. The first camera 22A and the second camera 22B are arranged at different imaging positions. The image data acquired by the first camera 22A and the second camera 22B is configured in such a way that pixels having a luminance set by, for example, an 8-bit value (0 to 255) are arranged two-dimensionally. The smaller the luminance value, the darker the image, and the larger the luminance value, the brighter the image. A pixel with a brightness value of 0 is displayed as black in the image data. A pixel with a brightness value of 255 is displayed as white in the image data. The first camera 22A and the second camera 22B have, for example, the same resolution. In this case, the image data acquired by the first camera 22A and the second camera 22B have the same number of pixels in both the vertical and horizontal directions. The imaging unit 22 may be configured with a single camera.

The computer system 30 controls the overall operation of the gaze detection device 100. The computer system 30 includes an arithmetic processing unit 30A and a storage device 30B. The arithmetic processing unit 30A includes a microprocessor such as a CPU (central processing unit). The storage device 30B includes a memory or storage such as a ROM (read only memory) and a RAM (random access memory). The arithmetic processing unit 30A performs arithmetic processing according to a computer program 30C stored in the storage device 30B.

The output device 40 includes a display device such as a flat panel display. The output device 40 may also include a printing device. The input device 50 generates input data when operated. The input device 50 includes a keyboard or a mouse for a computer system. The input device 50 may also include a touch sensor provided on the display unit of the output device 40, which is a display device.

In the gaze detection device 100 according to this embodiment, the display device 10 and the computer system 30 are separate devices. The display device 10 and the computer system 30 may be integrated. For example, the gaze detection device 100 may include a tablet-type personal computer. In this case, the tablet-type personal computer may be equipped with a display device, an image acquisition device, a computer system, an input device, an output device, etc.

2 is a functional block diagram showing an example of the gaze detection device 100. As shown in FIG. 2, the computer system 30 has an image capture control unit 31, a position detection unit 32, a corneal curvature calculation unit 33, a table generation unit 34, a matching unit 35, a gaze processing unit 36, and a memory unit 37. The image capture control unit 31, the position detection unit 32, the corneal curvature calculation unit 33, the table generation unit 34, and the matching unit 35 of the computer system 30, and the image acquisition device 20 described above constitute the subject identification device 70 according to this embodiment. The functions of the computer system 30 are performed by the calculation processing device 30A and the storage device 30B (see FIG. 1). Some functions of the computer system 30 may be provided outside the gaze detection device 100.

The imaging control unit 31 controls the light source unit 21 and the imaging unit 22. The imaging control unit 31 controls the timing and emission time of the detection light for each of the first light source 21A and the second light source 21B of the light source unit 21. The imaging control unit 31 controls the imaging timing for the first camera 22A and the second camera 22B of the imaging unit 22. The imaging control unit 31 causes the first light source 21A and the second light source 21B to emit the detection light in synchronization with the frame synchronization signal of the imaging unit 22. In this embodiment, the imaging control unit 31 causes the first light source 21A and the second light source 21B to alternately emit the detection light, for example, for each period of the frame synchronization signal. The imaging control unit 31 acquires image data acquired by the image acquisition device 20. The imaging control unit 31 stores the acquired image data in the storage unit 37.

The position detection unit 32 detects the position data of the pupil center indicating the center of the pupil of the subject's eyeball irradiated with the detection light based on the image data of the subject's eyeball imaged by the imaging unit 22. The position detection unit 32 also detects the position data of the corneal reflection center indicating the center of the corneal reflection of the subject's eyeball irradiated with the detection light based on the image data of the subject's eyeball imaged by the imaging unit 22. The position detection unit 32 detects the position data of the pupil center and the position data of the corneal reflection center for each of the subject's left and right eyes. The position detection unit 32 calculates the position of the corneal reflection center by the detection light from the light source unit 21 based on the image data of the subject's eyeball imaged by the imaging unit 22. The corneal reflection center is the center of the corneal reflection image by each detection light. The position detection unit 32 also detects the distance between the subject and the imaging unit 22 (hereinafter referred to as the camera-subject distance) by triangulation based on the image data captured by the two cameras, the first camera 22A and the second camera 22B of the imaging unit 22. The position detection unit 32 stores the detected pupil center, corneal reflection center positions, and camera-subject distance in the memory unit 37.

The corneal curvature calculation unit 33 detects the position of the corneal reflection center, which indicates the center of the corneal reflection in the eyeball irradiated with the detection light, from the captured image data, and calculates the corneal curvature of the subject based on the detected position. In this embodiment, the corneal curvature calculation unit 33 can calculate the corneal curvature radius of the subject as the corneal curvature. The corneal curvature calculation unit 33 may also calculate the curvature of the subject's cornea as the corneal curvature.

The table generating unit 34 generates a corneal curvature table indicating the relationship between the position on the eyeball and the corneal curvature based on the corneal curvature radius calculated at multiple different positions on the subject's eyeball. In this embodiment, the table generating unit 34 generates a corneal curvature table indicating the relationship between the position on the subject's eyeball and the corneal curvature radius. Details of the corneal curvature table will be described later.

The table generating unit 34 stores the generated corneal curvature table T in the storage unit 37 in association with subject information indicating the subject. The subject information includes subject identification information for identifying the subject and gamma angle information of the subject. Examples of subject identification information include a number for identifying the subject and the subject's name. The gamma angle information is information on the gamma angle, which is the deviation between the line of sight of the subject and a straight line connecting the center of the corneal curvature of the subject's eyeball and the center of the pupil. Details of the gamma angle will be described later.

The collation unit 35 compares the corneal curvature table newly generated by the table generation unit 34 (hereinafter referred to as the "generated table") with the corneal curvature table stored in the memory unit 37 (hereinafter referred to as the "memorized table"), and identifies the subject based on the collation result. The collation unit 35 determines whether the generated table and the memorized table correspond to each other. If the collation unit 35 determines that the generated table and the memorized table correspond to each other, it refers to the subject identification information stored in association with the memorized table, and identifies the subject based on the subject identification information.

The gaze processing unit 36 calculates the gaze of the subject. The gaze processing unit 36 detects the position of the subject's pupil center based on image data captured by the imaging unit 22. The pupil center is the center of the pupil. The gaze processing unit 36 detects the gaze vector of the subject's eyeball EB as the gaze based on the calculated position of the pupil center, the position of the light source unit 21, the position of the corneal reflection center calculated by the position detection unit 32, the value of the corneal curvature radius calculated or set in advance, and the value of the gamma angle.

The storage unit 37 stores various data and programs for performing processing in each part of the computer system 30. The storage unit 37 stores, for example, data on an image to be displayed on the display unit 11. The storage unit 37 stores various information such as image data acquired by the imaging control unit 31, the position of the pupil center and the position of the corneal reflection center calculated by the position detection unit 32, the corneal curvature table calculated by the table generation unit 34, and subject information (subject identification information, gamma angle information) associated with the corneal curvature table.

The memory unit 37 also stores a subject identification program that causes the computer to execute the following processes: emitting detection light from the light source unit 21 and irradiating it on at least one eyeball EB of the subject; capturing an image of the eyeball EB irradiated with the detection light; detecting from the captured image the position of the corneal reflection center indicating the center of the corneal reflection in the eyeball EB irradiated with the detection light and calculating the corneal curvature of the subject based on the detected position; generating a corneal curvature table indicating the relationship between the position on the eyeball EB and the corneal curvature based on the corneal curvature calculated at multiple different positions on the subject's eyeball EB; and comparing the newly generated corneal curvature table with a corneal curvature table stored in the memory unit in association with subject identification information for identifying the subject, and identifying the subject based on the comparison result.

The storage unit 37 also causes the computer to execute the following processes: emitting detection light from the light source unit 21 and irradiating it on at least one eyeball EB of the subject; capturing an image of the eyeball EB irradiated with the detection light; detecting the position of the corneal reflection center indicating the center of the corneal reflection in the eyeball EB irradiated with the detection light from the captured image and calculating the corneal curvature of the subject based on the detected position; generating a corneal curvature table indicating the relationship between the position on the eyeball EB and the corneal curvature based on the corneal curvature calculated at multiple different positions on the subject's eyeball EB; determining whether the newly generated corneal curvature table corresponds to the corneal curvature table stored in the storage unit in association with the subject's identification, which is information about the subject; and calculating the subject's gaze. In the process of calculating the subject's gaze, the storage unit stores a gaze detection program that calculates the subject's gaze using the subject information associated with the corneal curvature table stored in the storage unit when it is determined that the newly generated corneal curvature table corresponds to the corneal curvature table stored in the storage unit.

Figure 3 is a diagram showing a schematic diagram of the principle of detecting the gaze of a subject in the gaze detection device 100 of this embodiment. The following describes the subject's right eye EB (right eye ER), but a similar explanation can be applied to the left eye EB (left eye EL). In this embodiment, a straight line 130 connecting the pupil center 112 and the corneal curvature center 110 of the subject's eye EB is obtained, and the three-dimensional vector of this straight line 130 is detected as the gaze.

As shown in FIG. 3, detection light is irradiated onto the subject's eye EB from one of the first light source 21A and the second light source 21B (for example, the first light source 21A). A corneal reflection image 120 is formed on the cornea of the subject's eye EB by the detection light. By capturing an image of the subject's eye EB with the imaging unit 22, image data of the eye EB on which the corneal reflection image 120 is formed can be obtained. The image data also includes an image of the subject's pupil. Therefore, based on the image data, the position of the pupil center 112 and the position of the corneal reflection center 125 of the subject's eye EB can be calculated. The pupil center 112 indicates the center of the subject's pupil. The corneal reflection center 125 indicates the center of the corneal reflection image 120. At this time, by using two cameras, the first camera 22A and the second camera 22B, the positions of the subject's pupil center 112 and the corneal reflection center 125 can be calculated by triangulation.

In the subject's eye EB, the corneal curvature center 110 is a position on a straight line 123 that connects the first light source 21A, which is the light source that formed the corneal reflection image 120, and the corneal reflection center 125 of the corneal reflection image 120. Specifically, the position of the corneal curvature center 110 is a position on the straight line 123 that is a distance away from the corneal reflection center 125 in the direction opposite to the first light source 21A, which corresponds to the corneal curvature radius r. Therefore, when the position of the corneal reflection center 125 is calculated, the position of the corneal curvature center 110 can be calculated based on the value of the corneal curvature radius r. The value of the corneal curvature radius r may be, for example, a preset value, a value calculated by measurements or the like described below, or a value based on a corneal curvature table described below.

By determining the positions of the pupil center 112 and the corneal curvature center 110, it is possible to calculate a straight line 130 passing through the pupil center 112 and the corneal curvature center 110. Then, the three-dimensional vector of this straight line 130 can be calculated as the subject's line of sight. After detecting the line of sight vector, it is possible to detect the position of the gaze point indicating the intersection of the line of sight vector 130 (line of sight vector) and the display unit 11.

Here, an example of the principle of calculating the corneal curvature radius by the corneal curvature calculation unit 33 will be described. Figures 4 and 5 are schematic diagrams showing the state in which detection light from the light source unit 21 is irradiated onto the subject's eyeball EB. The subject's right eyeball EB (right eye ER) will be described below, but a similar description can be made for the left eyeball EB (left eye EL). Also, in Figures 4 and 5, only the first camera 22A is shown as the imaging unit 22, and the second camera 22B is omitted, but in reality both the first camera 22A and the second camera 22B are arranged.

As shown in Figures 4 and 5, when the eye EB is irradiated with detection light from the first light source 21A and the second light source 21B, a corneal reflection image is formed on the cornea of the eye EB. Here, the center of the corneal reflection image 120A of the first light source 21A is the first corneal reflection center 121. Also, the center of the corneal reflection image 120B of the second light source 21B is the second corneal reflection center 122.

As shown in FIG. 5, the center-to-center distance d between the first corneal reflection center 121 and the second corneal reflection center 122 changes as the distance between the imaging unit 22 and the subject changes. For example, d1>d2 holds between the center-to-center distance d1 when the camera-to-subject distance, which is the distance between the imaging unit 22 and the subject's eyeball EB, is x1 (position P1) and the center-to-center distance d2 when the camera-to-subject distance is x2 (position P2), which is larger than x1.

6 is a diagram showing an example of the positional relationship between the light source unit, the imaging unit, and the subject's eyeball. As shown in FIG. 6, the distance between the first light source 21A and the imaging unit 22 is a, the distance between the second light source 21B and the imaging unit 22 is b, the camera-subject distance between the imaging unit 22 and the subject is x, the distance between the first corneal reflection center 121 and the second corneal reflection center 122 detected by the eyeball EB is d, and the corneal curvature radius is r. In this example, the camera-subject distance x is the distance between the imaging unit 22 and the corneal curvature center 110 of the subject's eyeball EB. Note that the camera-subject distance x can be, for example, a value detected by triangulation based on image data captured by two cameras, the first camera 22A and the second camera 22B. Here, since r<<x, it can be approximated that the angle formed by the straight line L1 passing through the corneal curvature center 110 and the first light source 21A and the straight line L2 passing through the corneal curvature center 110 and the first corneal reflection center 121 is equivalent to the angle formed by the straight line L3 passing through the corneal curvature center 110 and the imaging unit 22 and the straight line L2. Hereinafter, this angle will be referred to as θ1. Similarly, it can be approximated that the angle formed by the straight line L4 passing through the corneal curvature center 110 and the second light source 21B and the straight line L5 passing through the corneal curvature center 110 and the first corneal reflection center 121 is equivalent to the angle formed by the straight line L3 passing through the corneal curvature center 110 and the imaging unit 22 and the straight line L5. Hereinafter, this angle will be referred to as θ2.

In this case, the following formulas 1 to 3 hold true.
θ1=arctan(a/x)/2 (Equation 1)
θ2=arctan(b/x)/2 (Equation 2)
d=r·sinθ1+r·sinθ2 (Equation 3)

Figure 7 is a diagram showing an example of image data of the eyeball EB captured by the imaging unit 22. As shown in Figure 7, in this embodiment, image data IM1 captured when the eyeball EB is irradiated with detection light from the first light source 21A, and image data IM2 captured when the eyeball EB is irradiated with detection light from the second light source 21B are acquired as separate image data. The image data IM1 shows a corneal reflection image (hereinafter referred to as the first corneal reflection image 120A) due to the detection light from the first light source 21A. The image data IM2 shows a corneal reflection image (hereinafter referred to as the second corneal reflection image 120B) due to the detection light from the second light source 21B. In this embodiment, the first light source 21A and the second light source 21B are turned on at different times. Therefore, the first corneal reflection image 120A and the second corneal reflection image 120B are reflected in one image data IM1 only the first corneal reflection image 120A, and in the other image data IM2 only the second corneal reflection image 120B. The position detection unit 32 calculates the positions of the first corneal reflection center 121 and the second corneal reflection center 122 based on the acquired first image data and second image data.

The corneal curvature calculation unit 33 calculates the actual distance d (mm) between the first corneal reflection center 121 and the second corneal reflection center 122 from the distance (number of pixels of the image sensor) d' (pixels) between the first corneal reflection center 121 included in the image data IM1 and the second corneal reflection center 122 included in the image data IM2. Note that in FIG. 7, the first corneal reflection center 121 and the second corneal reflection center 122 are shown as black dots to make the positions easier to distinguish, but in reality there are no black dots.

Here, if the focal length of the lens that constitutes the imaging unit 22 is f and the distance (pitch) between pixels of the imaging element that constitutes the imaging unit 22 is p, the following equation 4 holds.

d' = (f d/x)/p ... (Equation 4)

Solving the above equations 1 to 4 for d' gives
d'=r·(sin(arctan(a/x)/2)
+ sin(arctan(b/x)/2))·f/(p·x) ... (Equation 5)
It becomes.

In Equation 5, for example, if a, b = 100 (mm), f = 12 (mm), x = 600 (mm), and p = 0.0048 (mm/pixel), the relationship between x and d' can be obtained as shown in Equation 6 below.

d' = r * (2 * sin(arctan(100/600)/2)) * 12
/(0.0048.600)
= r α (where α is a constant) ... (Equation 6)

The corneal curvature calculation unit 33 can calculate the corneal curvature radius r based on the center-to-center distance d' by using the above formula 6.

Figure 8 is a schematic diagram showing an example of a corneal curvature table used in this embodiment. As shown in Figure 8, in the corneal curvature table T, a grid G is set with its center at the position of the pupil center 112 when the subject's eyeball is viewed from the front. The grid G is formed in a lattice shape by virtual straight lines. For example, the horizontal direction of the grid G can be the X direction and the vertical direction can be the Y direction. In the corneal curvature table T, positions on the cornea are defined by areas Ga partitioned in a lattice shape by the virtual straight lines that make up the grid G. One position on the cornea is defined for each area Ga. Each area Ga can be defined by, for example, the X coordinate and the Y coordinate. The corneal curvature table T is generated for each subject.

A portion of Figure 8 shows an enlarged view of grid G. Corneal curvature table T is data in which a calculated corneal radius of curvature is associated with each area Ga defined by grid G. In grid G in Figure 4, the same corneal radius of curvature is shown in the same color. In the example shown in the enlarged view of grid G, the corneal radius of curvature associated with area Ga of grid G increases stepwise as it spreads out in concentric circles around the subject's pupil center 112.

Figure 9 is a diagram showing a schematic example of a process for generating a corneal curvature table T. As shown in Figure 9, when the line of sight (eyeball EB) moves while a corneal reflex is formed on the subject's eyeball EB, the positions of the first corneal reflection center 121 and the second corneal reflection center 122 on the cornea move relatively. In Figure 9, an example is given in which the position of the second corneal reflection center 122 moves across the portion Ga of the grid G. A similar explanation can be given for the case in which the position of the first corneal reflection center 121 moves. When the position of the second corneal reflection center 122 on the cornea moves across the portion Ga of the grid G on the locus 122R shown in Figure 9, the corneal curvature calculation unit 33 calculates the value of the corneal curvature radius for each portion Ga. In addition, the table generation unit 34 stores the calculated value in association with the portion Ga. In addition, if the corneal curvature calculation unit 33 calculates the value of the corneal curvature radius multiple times at the same area Ga, the table generation unit 34 may associate the average value of the multiple calculation results as the value of the corneal curvature radius. In this way, the table generation unit 34 can update the generated corneal curvature table T. The table generation unit 34 can generate the corneal curvature table T shown in FIG. 8 by performing the above process for a predetermined period of time, for example.

In the corneal curvature table T shown in FIG. 8, the corneal curvature radius can be calculated for some areas Ga without following the above procedure. FIG. 10 is a schematic diagram showing an example of an eyeball EB viewed from the front. FIG. 10 shows approximate circles C1 to C3 indicating the corneal curvature radius of the pupil 114, iris 116 and their vicinity. As shown in FIG. 10, the corneal curvature radius of a human eyeball EB generally changes gently in a concentric manner around the pupil center 112 when viewed from the front. In other words, the diameter of the approximate circle gradually increases as it moves away from the pupil center 112. Furthermore, the corneal curvature radius of the portion along the approximate circle is approximately equal around one circumference of the approximate circle. Therefore, for the regions Ga of the grid G of the corneal curvature table T for which the value of the corneal radius of curvature has not been calculated, the table generating unit 34 performs an interpolation process so that the corneal radius of curvature increases concentrically around the pupil center 112 and the corneal radius of curvature of the regions Ga along the concentric circles is the same, thereby making it possible to calculate the corneal radius of curvature for all regions Ga.

FIG. 11 is a diagram showing an example of the interpolation process by the table generating unit 34. As shown in FIG. 11, the table generating unit 34 can set a plurality of concentric circles with different diameters centered on the pupil center 112 in the grid G as the interpolation process, and can interpolate the corneal curvature table T so that the corneal curvature radius of the sites Ga existing in the same region for a plurality of regions partitioned by the plurality of concentric circles is the same value. In addition, the table generating unit 34 can interpolate the corneal curvature table T so that the value of the corneal curvature radius increases as the diameter of the concentric circle increases. FIG. 11 shows an example in which the table generating unit 34 sets three virtual concentric circles. Hereinafter, the virtual circle with the smallest diameter is referred to as the first virtual circle, the virtual circle with the second smallest diameter is referred to as the second virtual circle, and the virtual circle with the largest diameter is referred to as the third virtual circle. The first to third virtual circles correspond to, for example, the above-mentioned approximate circles C1 to C3. As shown in FIG. 11, the table generating unit 34 can interpolate the corneal curvature radius so that the sites Ga located in the area inside the first virtual circle have the same value, the sites Ga located in the area between the first and second virtual circles have the same value, the sites Ga located in the area between the second and third virtual circles have the same value, and the sites Ga located in the area outside the third virtual circle have the same value. Unlike so-called conventional calibration operations, the generation of the corneal curvature table T by the table generating unit 34 does not require the subject to gaze at a specific position specified in advance, and can be executed as long as the subject is within a range that can be imaged by the imaging unit 22.

Figure 12 is a diagram showing an example of gamma angle information. As shown in the upper part of Figure 12, in the case where the physical center 118 of the retina 117 and the retinal visual field center 119 are the same in the eyeball EB, the pupil center 112, the corneal curvature center 110, and the retinal visual field center 119 are on the same line. In this case, the line 130 connecting the corneal curvature center 110 and the pupil center 112 is equal to the line 129 connecting the corneal curvature center 110 and the retinal visual field center 119. The line 130 indicates the direction of the subject's pupil. The line 130 is a line corresponding to the gaze vector detected by the gaze detection device 100. The line 129 indicates the direction of the image reflected at the retinal visual field center 119 of the subject's retina 117, that is, the subject's actual gaze.

In contrast, as shown in the lower part of Figure 12, when the physical center 118 of the retina 117 and the retinal visual field center 119 are in offset positions, the pupil center 112, the corneal curvature center 110, and the retinal visual field center 119 are not on the same line. In this case, the line 130 connecting the corneal curvature center 110 and the pupil center 112 is inclined with respect to the line 129 connecting the corneal curvature center 110 and the retinal visual field center 119. The angle of inclination between the lines 129 and 130, that is, the angle between the line 130 corresponding to the gaze vector detected by the gaze detection device 100 and the line 129 corresponding to the subject's actual gaze, is the gamma angle γ. The gamma angle γ is a value specific to each individual.

Figure 13 is a diagram showing the principle of measuring the gamma angle by calibration. As shown in Figure 13, the subject is instructed to gaze at the target 12 on the display unit 11 at the start of calibration. While the subject gazes at the target 12, detection light is emitted from, for example, the first light source 21A of the light source unit 21. The corneal curvature center 110 is obtained based on a straight line connecting the first light source 21A and the corneal reflection center 121 formed on the subject's eyeball EB and the value of the corneal curvature radius. The straight line 130 connecting the corneal curvature center 110 and the pupil center 112 is the direction of the subject's pupil. In this calibration, the straight line 129 connecting the corneal curvature center 110 and the target 12 is the subject's line of sight. In this case, the angle γ formed by the straight lines 130 and 129 can be calculated as the gamma angle.

By correcting the gamma angle γ with respect to the direction of the subject's pupil (straight line 130), the gaze vector can be calculated with high accuracy. In this way, the gamma angle γ can be measured by the subject's conscious action, such as by the subject gazing at the target 12. Therefore, in consideration of the burden on the subject and the measurer, if a correct gamma angle γ value is obtained by calibration, it is desirable to reuse the gamma angle γ value obtained above when detecting the subject's gaze from the next time onwards. In the above explanation, the angle between the straight line 130 corresponding to the gaze vector detected by the gaze detection device 100 and the straight line 129 corresponding to the subject's actual gaze is described as a planar angle when the subject is viewed from above, but in reality, the gamma angle γ is calculated as a three-dimensional angle including the angle when the subject is viewed from the side.

Figure 14 is a diagram showing a schematic example of a process for matching a generated table with a stored table. The example shown in Figure 14 shows a case where a generated table T1 and stored tables T2 and T3 are used. In order to simplify the explanation, the generated table T1 and stored tables T2 and T3 shown in Figure 14 are explained using an example where there are five grids G in each of the X direction and Y direction, that is, a 5 x 5 grid G. The numerical values on the X axis and Y axis respectively indicate the X coordinate and the Y coordinate when the part Ga corresponding to the pupil center is set as the origin. The number of grids G is not limited to 5 x 5. In addition, in Figure 14, a value based on the inverse of the corneal curvature radius (curvature value) is used as the value associated with the part Ga of the grid G. In addition, the same explanation can be made even if the corneal curvature radius is used as the value associated with the part Ga of the grid G in Figure 14.

As shown in FIG. 14, the collation unit 35 acquires a newly generated generated table T1. The collation unit 35 determines whether the acquired generated table T1 corresponds to the stored tables T2 and T3. In this case, the collation unit 35 compares the generated table T1 with the stored tables T2 and T3, respectively.

The collation unit 35 can calculate the differences between all values in the two-dimensional array for example between the generated table T1 and the storage tables T2 and T3, and determine the correspondence based on the sum β of the differences. Here, the curvature value of each part Ga in the generated table T1 is A(x, y), and the curvature value of each part Ga in the storage tables T2 and T3 is B(x, y), where x indicates the X coordinate and y indicates the Y coordinate. In this case, the sum β of the differences can be calculated using the following formula 7.

β = |A(-2,-2)-B(-2,-2)|
+ | A (-2,-2) - B (-2,-1) | +
+……
+ |A(2,2)-B(2,2)| (Equation 7)

In actual curvature measurements, there are almost no cases where the measurements match perfectly, and a certain amount of error occurs. For this reason, if the sum of the differences β is equal to or less than a threshold value, it can be assumed that the generated table T1 corresponds to the stored tables T2 and T3.

In FIG. 14, the sum β of the differences between generated table T1 and stored table T2 is 6.58. Also, the sum β of the differences between generated table T1 and stored table T3 is 2.34.

For example, if the threshold value is set to 3.0, the collation unit 35 can determine that a memory table T2 in which the sum of differences β is greater than the threshold value 3.0 does not correspond to the generated table T1. The collation unit 35 can also determine that a memory table T3 in which the sum of differences β is equal to or less than the threshold value 3.0 corresponds to the generated table T1.

In this case, for the memory table T3 that is determined by the comparison unit 35 to correspond to the generated table T1, the gaze processing unit 36 refers to the subject identification information among the subject information associated with the memory table T3, and identifies the subject based on the subject identification information.

When calculating the gaze of the subject related to the generated table T1, the gaze processing unit 36 refers to the gamma angle information in the subject information associated with the memory table T3 that is determined by the comparison unit 35 to correspond to the generated table T1, and calculates the gaze of the subject using the gamma angle γ based on the gamma angle information.

Furthermore, for example, if the threshold value is set to 1.0, the collation unit 35 can determine that both storage tables T2 and T3, for which the sum of differences β is greater than the threshold value 1.0, do not correspond to the generated table T1.

In this case, there is no memory table corresponding to the generated table T1. Therefore, the collation unit 35 stores the generated table T1 in the memory unit 37. As a result, the generated table T is stored in the memory unit 37 as a new memory table. In addition, the line of sight processing unit 36 calculates the gamma angle γ of the subject by calibration, and stores the gamma angle information including the calculated gamma angle γ in the memory unit 37 in association with the new memory table (T1).

[Subject Identification Method, Gaze Detection Method]
Next, an example of a subject identification method and a gaze detection method using the gaze detection device 100 according to this embodiment will be described. Fig. 15 is a flowchart showing an example of the subject identification method and the gaze detection method according to this embodiment. As shown in Fig. 15, first, the table generation unit 34 generates a corneal curvature table (generation table) indicating the relationship between the position on the eyeball and the corneal curvature based on the corneal curvature radii calculated at a plurality of different positions on the eyeball of the subject (step S101).

Next, the comparison unit 35 compares the generated table with the corneal curvature table (storage table) stored in the storage unit 37 (step S102). Based on the comparison result, the comparison unit 35 determines whether or not there is a storage table that corresponds to the generated table (step S103).

If it is determined in step S103 that a storage table corresponding to the generated table exists (Yes in step S103), the collation unit 35 identifies the subject based on the subject identification information among the subject information associated with the determined storage table (step S104).

On the other hand, if it is determined in step S103 that there is no storage table corresponding to the generated table (No in step S103), the collation unit 35 stores the generated table in the storage unit 37 as a new storage table (step S105).

The line-of-sight processing unit 36 measures the subject's gamma angle γ by calibration (step S106). The line-of-sight processing unit 36 stores the measured gamma angle γ in a new storage table in association with the new gamma angle γ (step S107).

When the processing of step S104 is performed, the line-of-sight processing unit 36 acquires the gamma angle γ included in the gamma angle information of the subject information associated with the memory table of the identified subject (step S105) and calculates the line of sight (step S110). On the other hand, when the processing of step S107 is performed, the line-of-sight processing unit 36 acquires the measured gamma angle γ (step S109) and calculates the line of sight (step S110).

Figure 16 is a flowchart showing an example of the gaze detection process in the gaze detection method according to this embodiment. As shown in Figure 16, in the gaze detection process, detection light is emitted from either the first light source 21A or the second light source 21B of the light source unit 21 to illuminate the eyeball EB (step S201), and the eyeball EB is imaged by both the first camera 22A and the second camera 22B of the imaging unit 22 (step S202). The position detection unit 32 detects the positions of the pupil center 112 and the corneal reflection center 125 based on the acquired image data of the eyeball EB (step S203). The position detection unit 32 converts the positions of the pupil center 112 and the corneal reflection center 125 into world coordinates (global coordinate system) by triangulation or the like (step S204).

The line-of-sight processing unit 36 calculates the position of the corneal curvature center 110 based on the position of the light source that emitted the detection light, either the first light source 21A or the second light source 21B, the position of the pupil center 112, the position of the corneal reflection center 125, and the corneal curvature radius r (step S205). In this case, the line-of-sight processing unit 36 obtains a straight line 123 that connects the position of the light source that emitted the detection light and the corneal reflection center 125. The line-of-sight processing unit 36 obtains, on the obtained straight line 123, a position that is a distance equivalent to the corneal curvature radius r away from the corneal reflection center 125 toward the inside of the eyeball EB as the position of the corneal curvature center 110.

The line of sight processing unit 36 obtains a straight line 130 connecting the pupil center 112 and the corneal curvature center 110, and calculates the three-dimensional vector of the straight line 130 as the line of sight vector by correcting the inclination of the straight line 130 using the gamma angle γ obtained in step S105 or step S109 (step S206).

As described above, the subject identification device 70 according to this embodiment includes a plurality of first light sources 21A and second light sources 21B that emit detection light and irradiate it on at least one eyeball of the subject, an imaging unit 22 that captures an image of the eyeball irradiated with the detection light, a corneal curvature calculation unit 33 that detects the position of the corneal reflection center indicating the center of the corneal reflection in the eyeball irradiated with the detection light from the captured image and calculates the subject's corneal curvature at the detected position, a table generation unit 34 that generates a corneal curvature table indicating the relationship between the position on the eyeball and the corneal curvature based on the corneal curvature calculated at multiple different positions on the subject's eyeball, and a comparison unit 35 that compares the generated table, which is the newly generated corneal curvature table, with the storage table, which is the corneal curvature table stored in the storage unit 37 in association with subject identification information for identifying the subject, and identifies the subject based on the comparison result.

The subject identification method according to this embodiment includes emitting detection light from the first light source 21A and the second light source 21B and irradiating it onto at least one eyeball of the subject, capturing an image of the eyeball irradiated with the detection light, detecting from the captured image the position of the corneal reflection center indicating the center of the corneal reflection in the eyeball irradiated with the detection light, and calculating the corneal curvature of the subject at the detected position, generating a corneal curvature table indicating the relationship between the position on the eyeball and the corneal curvature based on the corneal curvature calculated at multiple different positions on the subject's eyeball, comparing the newly generated corneal curvature table with the corneal curvature table stored in the memory unit 37 in association with subject identification information for identifying the subject, and identifying the subject based on the comparison result.

The subject identification program according to this embodiment causes a computer to execute the following processes: emitting detection light from the first light source 21A and the second light source 21B and irradiating it on at least one eyeball of the subject; capturing an image of the eyeball irradiated with the detection light; detecting from the captured image the position of the corneal reflection center indicating the center of the corneal reflection in the eyeball irradiated with the detection light and calculating the subject's corneal curvature at the detected position; generating a corneal curvature table indicating the relationship between the position on the eyeball and the corneal curvature based on the corneal curvature calculated at multiple different positions on the subject's eyeball; and comparing the newly generated corneal curvature table with a corneal curvature table stored in the memory unit 37 in association with subject identification information for identifying the subject, and identifying the subject based on the comparison result.

According to this configuration, a corneal curvature table is generated, the newly generated table is compared with a memory table stored in the memory unit 37, and the subject can be identified based on the comparison result using the subject identification information associated with the memory table. The corneal curvature table is information that differs depending on the subject, that is, information specific to the subject. Therefore, by using the subject-specific information, the subject can be identified with high accuracy.

The gaze detection device 100 according to this embodiment includes a plurality of first light sources 21A and second light sources 21B that emit detection light and irradiate it on at least one eyeball of the subject, an imaging unit 22 that captures an image of the eyeball irradiated with the detection light, a corneal curvature calculation unit 33 that detects the position of the corneal reflection center that indicates the center of the corneal reflection in the eyeball irradiated with the detection light from the captured image and calculates the corneal curvature of the subject based on the detected position, and a corneal curvature table that indicates the relationship between the position on the eyeball and the corneal curvature based on the corneal curvature calculated at multiple different positions on the subject's eyeball. The system is equipped with a table generation unit 34 that generates a table, a storage unit 37 that stores subject information, which is information about the subject, in association with the corneal curvature table, a comparison unit 35 that determines whether the generated table, which is a newly generated corneal curvature table, corresponds to the stored table, which is a corneal curvature table stored in the storage unit 37, and a gaze processing unit 36 that calculates the subject's gaze, and when it is determined that the generated table corresponds to the stored table, the gaze processing unit 36 calculates the subject's gaze using the subject information associated with the stored table.

According to this configuration, if the collation unit 35 determines that the generated table and the stored table correspond to each other, the subject information associated with the stored table is used in calculating the subject's gaze. This allows the subject information to be acquired appropriately, reducing the burden on the subject and the operator and improving the performance of the gaze detection device 100.

In the gaze detection device 100 according to this embodiment, the subject information includes information on the gamma angle, which is the deviation between the line connecting the center of the corneal curvature of the subject's eyeball and the center of the pupil and the subject's gaze.

With this configuration, gamma angle information that can be obtained through measurements that require the subject's conscious actions can be used as subject information, making it possible to eliminate the need to remeasure the gamma angle. This reduces the burden on the subject and the operator and improves the performance of the gaze detection device 100.

The technical scope of the present invention is not limited to the above-described embodiment, and appropriate modifications can be made without departing from the spirit of the present invention. For example, the above-described embodiment has been described with reference to an example in which gamma angle γ is used as subject information, but the present invention is not limited to this configuration. For example, correction gaze points are displayed in sequence at different locations on the display unit 11, the subject is asked to gaze at the correction gaze points to measure the position of the subject's gaze point, and subject-specific correction data for correcting the subject's line of sight based on the measurement results is generated, and the correction data is used as subject information. In this case, the generation of the correction data can be performed, for example, as calibration.

EB...eyeball, G...grid, Ga...area, IM1, IM2...image data, r...corneal radius of curvature, x...camera-subject distance, T...corneal curvature table, T1...generated table, T2, T3...storage table, 10...display device, 11...display unit, 12...target, 20...image acquisition device, 21...light source unit, 21A...first light source, 21B...second light source, 22...imaging unit, 22A...first camera, 22B...second camera, 30...computer system, 30A...arithmetic processing unit, 30B...storage unit, 30C...computer program, 31...imaging control unit, 32...position detection unit, 33... Corneal curvature calculation unit, 34...table generation unit, 35...matching unit, 36...gaze processing unit, 37...storage unit, 40...output device, 50...input device, 60...input/output interface device, 70...subject identification device, 100...gaze detection device, 110...corneal curvature center, 112...pupil center, 114...pupil, 116...iris, 117...retina, 118...physical center of retina, 119...retinal visual field center, 120...corneal reflection image, 120A...first corneal reflection image, 120B...second corneal reflection image, 121...first corneal reflection center, 122...second corneal reflection center, 122R...trajectory, 125...corneal reflection center

Claims (5)

A plurality of light sources that emit detection light and irradiate at least one eye of the subject;
an imaging unit that captures an image of the eyeball irradiated with the detection light;
a corneal curvature calculation unit that detects a position of a corneal reflection center indicating a center of a corneal reflection in the eyeball irradiated with the detection light from the captured image, and calculates a corneal curvature of the subject at the detected position;
a table generating unit configured to generate a corneal curvature table indicating a relationship between a position on the eyeball and the corneal curvature based on the corneal curvatures calculated at a plurality of different positions on the eyeball of the subject;
a comparison unit that compares a generated table, which is a newly generated corneal curvature table, with a stored table, which is a corneal curvature table stored in a storage unit in correspondence with subject identification information for identifying the subject, and identifies the subject based on a comparison result. A plurality of light sources that emit detection light and irradiate at least one eye of the subject;
an imaging unit that captures an image of the eyeball irradiated with the detection light;
a corneal curvature calculation unit that detects a position of a corneal reflection center indicating a center of a corneal reflection in the eyeball irradiated with the detection light from the captured image, and calculates a corneal curvature of the subject based on the detected position;
a table generating unit configured to generate a corneal curvature table indicating a relationship between a position on the eyeball and the corneal curvature based on the corneal curvatures calculated at a plurality of different positions on the eyeball of the subject;
a storage unit that stores subject information, which is information about the subject, in association with a corneal curvature table;
a comparison unit that determines whether or not a generated table, which is a newly generated corneal curvature table, corresponds to a stored table, which is a corneal curvature table stored in the storage unit;
a gaze processing unit that calculates the gaze of the subject,
When it is determined that the generated table and the stored table correspond to each other, the gaze processing unit calculates the gaze of the subject using the subject information associated with the stored table. The gaze detection device according to claim 2 , wherein the subject information includes information on a gamma angle that is a deviation between a line connecting a center of corneal curvature and a center of a pupil of the subject's eyeball and the gaze of the subject. Emitting detection light from a light source and irradiating at least one eyeball of the subject;
capturing an image of the eyeball irradiated with the detection light;
detecting a position of a corneal reflection center indicating a center of a corneal reflection in the eyeball irradiated with the detection light from the captured image, and calculating a corneal curvature of the subject at the detected position;
generating a corneal curvature table indicating a relationship between a position on the eyeball and the corneal curvature based on the corneal curvatures calculated at a plurality of different positions on the eyeball of the subject;
comparing the newly generated corneal curvature table with a corneal curvature table stored in a storage unit in association with subject identification information for identifying the subject, and identifying the subject based on a comparison result. A process of emitting detection light from a light source and irradiating at least one eyeball of the subject;
A process of capturing an image of the eyeball irradiated with the detection light;
detecting a position of a corneal reflection center indicating a center of a corneal reflection in the eyeball irradiated with the detection light from the captured image, and calculating a corneal curvature of the subject at the detected position;
A process of generating a corneal curvature table indicating a relationship between a position on the eyeball and the corneal curvature based on the corneal curvature calculated at a plurality of different positions on the eyeball of the subject;
A subject identification program that causes a computer to execute a process of comparing the newly generated corneal curvature table with a corneal curvature table stored in a storage unit in association with subject identification information for identifying the subject, and identifying the subject based on the comparison result.

Images