JP7540711B2 - Automatic eye movement recording system, calculation device, calculation method and program - Google Patents

Description

Article 30, Paragraph 2 of the Patent Act applies Date of meeting (publication date): November 5, 2020 Name of meeting, location of meeting: 74th Japanese Society of Clinical Ophthalmology (Dates: November 5 - December 6, 2020) Held online using the online viewing system "MICEnavi" Organizing institution: Department of Medical Ophthalmology, School of Medical, Pharmaceutical and Health Sciences, Kanazawa University Research presentation abstracts and research presentation data public URL: <https://www.micenavi.jp/74ringan/refresh> <https://www.micenavi.jp/74ringan/search/detail_branch/id:7> <https://www.micenavi.jp/74ringan/search/detail_branch/id:7> <https://www.micenavi.jp/74ringan/search/detail_branch/id:7> jp/74ringan/search/detail_session/id:10135 > < https://www. micenavi. jp/74ringan/search/detail_program/id:425 >

Article 30, paragraph 2 of the Patent Act applies. Preprint server publication date: November 12, 2020/November 16, 2020 Publication: Research paper "Automatic Measurements of Smooth Pursuit Eye Movements by Video-Oculography and Deep Learning-Based Object Detection" [Preprint server MedRxiv submission public URL] (jointly operated by Yale University, CSHL, and BMJ) [First draft submission] <https://www.medrxiv. org/content/10.1101/2020.11.09.20227736v1> <https://www.medrxiv.org/content/10.1101/2020.11.09.20227736v1.full.pdf> [Second Edition Posted] <https://www.medrxiv.org/content/10.1101/2020.11.09.20227736v2> <https://www.medrxiv. org/content/10.1101/2020.11.09.20227736v2. full. pdf>

The present invention relates to an automatic eye movement recording system, a calculation device, a calculation method, and a program.

Conventionally, a nystagmus analysis system that is composed of VOG (Video-oculography) goggles, an analysis unit, an input unit, an output unit, and a database is known (see Patent Document 1). In the technology described in Patent Document 1, the VOG goggles include a camera, a microphone, and a head position sensor. The VOG goggles include a hot mirror, a camera lens, and a camera body as the camera. Furthermore, the VOG goggles include a CMOS (complementary metal-oxide semiconductor) substrate as an imaging IC (integrated circuit), and an infrared LED (Light Emitting Diode) as a light source.
Incidentally, the technique described in Patent Document 1 can analyze the presence or absence of nystagmus in a subject, but cannot perform an eye-tracking test on the subject.

Further, a video nystagmus measuring device capable of performing an eye-tracking test on a subject has been conventionally known (see, for example, the following URL).
http://0c7.co.jp/products/interacoustics/products/vog/

However, in the above-mentioned video nystagmus measuring device, a visual target is displayed on a display screen. Therefore, when the video nystagmus measuring device is installed in an ophthalmological examination room, there is a risk that it will take up space in the ophthalmological examination room.
In addition, since many of the subjects who require eye-tracking tests are children with eye movement disorders, if an eye target displayed on a display is used as the eye target, there is a risk that the subject (child) will not be able to track the eye target.
Furthermore, when a visual target displayed on a display is used as the visual target, the movement of the visual target, etc. is preset, so it is not possible to respond flexibly, for example, by presenting the visual target in accordance with the subject's situation.
On the other hand, in the conventional technology in which a real target is used instead of a target displayed on a display, a qualitative eye-tracking test can be performed, but a quantitative eye-tracking test cannot be performed.
Furthermore, if the process of identifying the position of the optotype contained in each of the numerous still images that make up the video of the optotype captured during an eye tracking test is performed manually, for example by an examination technician or a doctor, it will take an enormous amount of time to identify the position of the optotype.

JP 2020-018704 A

In view of the above-mentioned problems, the present invention aims to provide an automatic eye movement recording system, a calculation device, a calculation method, and a program that can realize quantitative eye tracking testing and rapid identification of the position of the eye target.

One aspect of the present invention is an automatic eye movement recording system including a first image capturing unit that captures a first image that is an image of the subject's eye, a second image capturing unit that captures a second image that is an image including an optotype used in an optotype tracking test of the subject, and a calculation device, the calculation device including a first image capture unit that captures the first image captured by the first image capturing unit, a gaze position measurement unit that measures the gaze position of the subject based on the first image captured by the first image capture unit by using VOG (Video-oculography), and a second image capture unit that captures the second image captured by the second image capturing unit. an acquisition unit; a two-dimensional area identification unit that uses an object recognition algorithm to identify a two-dimensional area corresponding to the visual target included in the second image acquired by the second image acquisition unit; an eye target position calculation unit that calculates the position of the visual target based on the two-dimensional area identified by the two-dimensional area identification unit; and a merging processing unit that merges the gaze position of the subject measured by the gaze position measurement unit, which is the gaze position of the subject at the time the second image acquired by the second image acquisition unit was photographed by the second photographing unit, into the second image.

One aspect of the present invention is a computing device including: a first image acquisition unit that acquires a first image that is an image of the subject's eye captured by a first image capture unit; a gaze position measurement unit that uses VOG to measure the gaze position of the subject based on the first image captured by the first image capture unit; a second image acquisition unit that acquires a second image that is an image including a gaze target used in an eye tracking test of the subject captured by a second image capture unit; a two-dimensional area identification unit that uses an object recognition algorithm to identify a two-dimensional area corresponding to the gaze target included in the second image captured by the second image capture unit; an eye target position calculation unit that calculates the position of the gaze target based on the two-dimensional area identified by the two-dimensional area identification unit; and a merging processing unit that merges the gaze position of the subject measured by the gaze position measurement unit, which is the gaze position of the subject at the time the second image captured by the second image capture unit was captured by the second image capture unit, into the second image.

One aspect of the present invention is a calculation method including a first image acquisition step of acquiring a first image, which is an image of the subject's eye captured by a first image capture unit; a gaze position measurement step of measuring the gaze position of the subject based on the first image acquired in the first image acquisition step by using VOG; a second image acquisition step of acquiring a second image, which is an image including a gaze target used in an eye tracking test of the subject captured by a second image capture unit; a two-dimensional area identification step of identifying a two-dimensional area corresponding to the gaze target included in the second image acquired in the second image acquisition step by using an object recognition algorithm; an eye target position calculation step of calculating the position of the gaze target based on the two-dimensional area identified in the two-dimensional area identification step; and a merging process step of merging the gaze position of the subject measured in the gaze position measurement step, which is the gaze position of the subject at the time when the second image acquired in the second image acquisition step was captured by the second image capture unit, into the second image.

One aspect of the present invention is a program for causing a computer to execute a first image acquisition step of acquiring a first image, which is an image of the subject's eye captured by a first image capture unit; a gaze position measurement step of measuring the gaze position of the subject based on the first image captured in the first image acquisition step by using VOG; a second image acquisition step of acquiring a second image, which is an image including a gaze target used in an eye tracking test of the subject captured by a second image capture unit; a two-dimensional area identification step of identifying a two-dimensional area corresponding to the gaze target included in the second image captured in the second image acquisition step by using an object recognition algorithm; an eye target position calculation step of calculating the position of the gaze target based on the two-dimensional area identified in the two-dimensional area identification step; and a merging process step of merging the gaze position of the subject measured in the gaze position measurement step, which is the gaze position of the subject at the time when the second image captured in the second image acquisition step was captured by the second image capture unit, into the second image.

The present invention provides an automatic eye movement recording system, a calculation device, a calculation method, and a program that can realize quantitative eye tracking testing and rapid identification of the position of the eye target.

1 is a diagram illustrating an example of the configuration of an automatic eye movement recording system according to a first embodiment. 4 is a flowchart showing an example of processing executed in the automatic eye movement recording system of the first embodiment. FIG. 2 is a diagram showing equipment used as part of the automatic eye movement recording system 1 of the first embodiment in the examples. FIG. 4 is an enlarged view of the camera module shown in FIG. FIG. 1 is a diagram showing an optotype used in an example. FIG. 2 is a diagram for explaining an SSD used in the examples. 7 is a diagram for explaining source 1 (38×38), source 5 (3×3), and source 6 (1×1) shown in FIG. 6. FIG. 13 is a diagram for explaining SSD training. FIG. 13 is a diagram for explaining data analysis. FIG. 1 shows representative examples of results of examples. FIG. 11 is a diagram visualizing the relationship between the horizontal and vertical eye positions and the target position in the representative example shown in FIG. 10 . FIG. 13 is a diagram showing the relationship between the eye position (the gaze position of the subject) and the position of the visual target obtained in the examples. FIG. 13 is a diagram showing the relationship between SPEM velocity and target velocity obtained in the examples.

Below, we will explain the embodiments of the automatic eye movement recording system, calculation device, calculation method, and program of the present invention with reference to the drawings.

[First embodiment]
FIG. 1 is a diagram showing an example of the configuration of an automatic eye movement recording system 1 according to the first embodiment.
In the example shown in Fig. 1, the automatic eye movement recording system 1 of the first embodiment includes a first image capturing unit 11, a second image capturing unit 12, and a calculation device 13. The first image capturing unit 11 captures a first image which is an image of the subject's eye. The second image capturing unit 12 captures a second image which is an image including an eye target 10 (see Fig. 5) used in an eye target tracking test of the subject.
The calculation device 13 includes a first image acquisition unit 13A, a gaze position measurement unit 13B, a second image acquisition unit 13C, a two-dimensional area identification unit 13D, a visual target position calculation unit 13E, a merge processing unit 13F, a position deviation amount calculation unit 13G, and a movement speed difference calculation unit 13H.

The first image acquisition unit 13A acquires the first image (image of the subject's eye) captured by the first imaging unit 11 .
The gaze position measuring unit 13B uses VOG (Video-oculography) to measure the gaze position of the subject based on the first image (image of the subject's eyes) acquired by the first image acquiring unit 13A. VOG is a technology that irradiates both eyes of the subject with near-infrared light and acquires gaze information of the subject from the relative positional relationship between the corneal reflex and the pupil. VOG can quantify and visualize the eye movement of the subject.

The second image acquisition unit 13C acquires a second image (an image including the visual target 10) captured by the second photographing unit 12.
The two-dimensional region specifying unit 13D uses an object recognition algorithm to specify a two-dimensional region 10A (see FIG. 10 ) corresponding to the visual target 10 included in the second image acquired by the second image acquiring unit 13C. In other words, the two-dimensional region specifying unit 13D specifies which region on the second image is the two-dimensional region 10A corresponding to the visual target 10.
The two-dimensional region specifying unit 13D uses SSD (Single Shot MultiBox Detector) as an object recognition algorithm. SSD is an object recognition deep learning algorithm that predicts what an object contained in an image is and outputs the area of the object and a predicted value. In addition, the two-dimensional region 10A specified by the two-dimensional region specifying unit 13D is a rectangular region called a "bounding box."
In the example shown in Figure 1, the two-dimensional region 10A (bounding box) is a rectangular region surrounding the visual target 10 on the second image (a rectangular region circumscribing the visual target 10 on the second image), but in other examples, the visual target 10 on the second image may extend beyond the two-dimensional region 10A (bounding box).

1, the target position calculation unit 13E calculates the position of the target 10 (the position of the target 10 on the second image) based on the two-dimensional area 10A specified by the two-dimensional area specification unit 13D. In detail, the target position calculation unit 13E calculates the center of gravity position of the bounding box as the position of the target 10.
In another example, the target position calculation unit 13E may calculate the position of the target 10 on the second image by using a moving average, a moving median, or the like.

1, the merging processing unit 13F merges the gaze position of the subject measured by the gaze position measuring unit 13B and the gaze position of the subject at the time when the second image acquired by the second image acquiring unit 13C was photographed by the second photographing unit 12 into the second image. As a result, it becomes possible to compare the position of the visual target 10 on the second image with the gaze position of the subject at the time when the second image was photographed (the gaze position of the subject tracking the visual target 10) on the second image.
The positional deviation amount calculation unit 13G calculates the amount of deviation of the gaze position of the subject measured by the gaze position measurement unit 13B with respect to the position of the gaze target 10 calculated by the gaze target position calculation unit 13E.
In the example shown in FIG. 1, the calculation device 13 is equipped with a positional deviation amount calculation unit 13G, but in other examples, the calculation device 13 may not be equipped with the positional deviation amount calculation unit 13G (for example, the amount of deviation of the subject's gaze position relative to the position of the visual target 10 may be calculated outside the calculation device 13).

In the example shown in FIG. 1, the movement speed difference calculation unit 13H calculates the difference between the movement speed of the position of the visual target 10 calculated by the visual target position calculation unit 13E and the movement speed of the subject's gaze position measured by the gaze position measurement unit 13B.
In the example shown in FIG. 1, the calculation device 13 is equipped with a movement speed difference calculation unit 13H, but in other examples, the calculation device 13 may not be equipped with the movement speed difference calculation unit 13H (for example, the difference between the movement speed of the position of the visual target 10 and the movement speed of the subject's gaze position may be calculated outside the calculation device 13).

FIG. 2 is a flowchart showing an example of a process executed in the automatic eye movement recording system 1 of the first embodiment.
In the example shown in FIG. 2, in step S11, the first image capturing unit 11 captures a first image which is an image of the subject's eye.
Also, in step S12, the second image capturing unit 12 captures a second image that is an image including the optotype 10 used in the eye-tracking test of the subject.
Next, in step S13, the calculation device 13 executes processing such as calculation.

Specifically, in step S13A, the first image acquisition unit 13A acquires the first image (image of the subject's eye) captured in step S11.
Next, in step S13B, the gaze position measuring unit 13B uses VOG to measure the gaze position of the subject based on the first image (image of the subject's eye) acquired in step S13A.

Furthermore, in step S13C, the second image acquisition unit 13C acquires the second image (an image including the visual target 10) captured in step S12.
Next, in step S13D, the two-dimensional area specifying unit 13D uses an object recognition algorithm to specify the two-dimensional area 10A corresponding to the visual target 10 included in the second image acquired in step S13C.
Next, in step S13E, the target position calculation unit 13E calculates the position of the target 10 (the position of the target 10 on the second image) based on the two-dimensional area 10A specified in step S13D.

Next, in step S13F, the merging processing unit 13F merges the subject's gaze position measured in step S13B, which is the gaze position of the subject at the time the second image acquired in step S13C was photographed by the second photographing unit 12, into the second image.
Next, in step S13G, the positional deviation calculation unit 13G calculates the amount of deviation of the eye gaze position of the subject measured in step S13B from the position of the visual target 10 calculated in step S13E.
Also, in step S13H, the movement speed difference calculation unit 13H calculates the difference between the movement speed of the position of the visual target 10 calculated in step S13E and the movement speed of the eye gaze position of the subject measured in step S13B.

The present invention will be described in detail below with reference to examples, but the present invention is not limited to the following examples and can be modified as appropriate without departing from the spirit of the invention.

<Example>
The present inventors have evaluated smooth pursuit eye movement (SPEM) using the automatic eye movement recording system 1 of the first embodiment. Smooth pursuit eye movement is an eye movement for keeping a moving object in the vicinity of the fovea (Robinson DA, J Physiol, 1965).
In clinical ophthalmology, a nine-direction eye position test is used to screen for eye movement abnormalities using SPEM. However, because the examiner judges the results visually, there are problems with this, such as no records being kept during the test and the test results being qualitative.

Eye movement can be objectively and quantitatively evaluated by using VOG, which irradiates both eyes with near-infrared light and measures the eye position of both eyes non-invasively from the corneal reflex and the positional relationship of the pupil. Previous studies have reported that the SPEM speed of strabismus patients is significantly slower than that of healthy subjects (Lions C et al., Plos one, 2013).
VOG is often used in the laboratory as a device for objective quantitative evaluation of eye movement, but it has not yet become widespread in clinical ophthalmology. The reasons for this include the fact that the movement of the visual target is limited, i.e., because programmed movements are repeated, it is not possible to focus on testing areas suspected of abnormality, and because the testing method changes, it is difficult to compare results with previous tests.
Therefore, it is necessary to objectively quantitatively evaluate SPEM using the same method as the nine-direction eye position test performed in ophthalmology outpatient clinics.

Therefore, the inventors have focused on convolutional neural networks (CNN), an artificial intelligence technology. CNN extracts features by applying multiple filters to the brightness values of each pixel in an image.
Previous research has reported that SSD, an object detection algorithm that uses CNN, has a processing speed more than twice as fast as the conventional raster scan method for object detection (Liu W et al., arXiv, 2015), making it promising for application in real-time analysis.

The inventors developed a SPEM measurement system that combines VOG and SSD, and evaluated its measurement accuracy.

The subjects for the evaluation were 11 healthy subjects with no ophthalmic diseases other than refractive errors. The average age was 21.3 years, and the spherical equivalent (SE) was -2.95D for the right eye and -2.70D for the left eye. Stereopsis was 40 seconds in all subjects. The alternating prism occlusion test showed -3.1PD for near vision and -0.9PD for distance vision ("-" indicates exodeviation, "+" indicates esodeviation).

FIG. 3 is a diagram showing devices used as part of the automatic eye movement recording system 1 of the first embodiment in the examples.
In this example, we used an EMR-9 manufactured by NAC Image Technology Inc. as the VOG for measuring eye movements. This device combines the image of the outside world captured by the camera module with the gaze position calculated from the subject's eye position using a controller, and outputs it on the measurement screen.

FIG. 4 is an enlarged view of the camera module shown in FIG.
The entire camera module can be moved horizontally and vertically by 8 cm, and the scene camera attached in the center has a sampling rate of 29.97 Hz, a viewing angle of 62 degrees, and can be rotated 60 degrees in the pitch direction from the center position.
The eye cameras installed above the left and right eyes have a sampling rate of 240 Hz, a viewing angle of 43 degrees horizontally and 28.6 degrees vertically, and the center position can be adjusted horizontally by 1.3 cm for each eye. The near-infrared light emitted from the eye camera is reflected by a half mirror. The half mirror can also be adjusted, and can be rotated 2.5 cm vertically and 30 degrees in the pitch direction.

Images from the eye camera can be displayed in the lower left and lower right parts of the measurement screen, and a binary image of the single eye can be displayed in the upper left part of the measurement screen. The eye position is calculated from the relative positions of the corneal reflex and the pupil, and the subject's gaze position is measured. The measured gaze position of the subject can be displayed, included in the scene camera image in the upper right part of the measurement screen, with a delay time of 52 ms or less.

FIG. 5 is a diagram showing an optotype 10 used in the examples.
In this example, a 10 cm x 10 cm character fixation target was used as the visual target 10 for the eye movement test (eye tracking test), and a nine-direction eye position test was performed at a viewing distance of 1.0 m (the visual angle of the visual target 10 is 5.7°). During the test, each subject was instructed to keep their eyes fixed on the character's nose.

FIG. 6 is a diagram for explaining the SSD used in the examples.
As shown in Fig. 6, SSD is a model in which an extra module for object recognition is added to VGG16, a CNN model. In SSD, the feature of the third convolutional layer in the fourth block of the VGG network is set as source 1, and the feature of the final output layer is set as source 2. Convolution is performed on source 2 to obtain feature values of 10x10, 5x5, 3x3, and 1x1, and features from source 3 to source 6 are output.

FIG. 7 is a diagram for explaining source 1 (38×38), source 5 (3×3), and source 6 (1×1) shown in FIG.
Source 1, output from the third convolutional layer of the 4th block of the VGG network shown in Fig. 6, has a feature map of 38 x 38 as shown in Fig. 7. Source 5 has a feature map of 3 x 3, and source 6 has a feature map of 1 x 1. Each source has a different number of feature maps, and the finer the feature map, the higher the detection accuracy of small objects.

FIG. 8 is a diagram for explaining SSD training.
In SSD training, multiple bounding boxes are prepared for the correct position label of an object, called the ground truth, and a bounding box that is close to the correct box is learned as training is repeated.

In this embodiment, 500 images taken during a test of the subject's nine-direction eye position, measured with the scene camera of the EMR-9, were used as the data set.
All images of 640 × 480 pixels were resized to 300 × 300 pixels, and 300 images were randomly selected from the dataset for training. Training was performed for 100 epochs. The training images were randomly swapped to randomly change brightness, contrast, color, and color channels, and image enlargement and cropping were performed.
The verification data was 100 sheets, and the accuracy was evaluated using the verification data every 10 epochs. In addition, 100 sheets of test data were prepared.

FIG. 9 is a diagram for explaining data analysis.
In this example, the accuracy of SSD was evaluated at 75% average precision (AP75), which is the bounding box predicted by SSD for the ground-truth box in the test data.
If the SSD predicted bounding box overlaps with the correct box by 75% or more and the label of the object is also correct, it is called True positive (TP), and if the overlap is 75% or less, it is called False positive (FP). In addition, if the correct box exists in the image but the SSD does not detect the object, it is called False negative (FN), precision is the value obtained by dividing TP by the sum of TP and FP, recall is the value obtained by dividing TP by the sum of TP and FN, and the area of the graph with recall on the horizontal axis and precision on the vertical axis is AP75.

In this example, the statistical analysis was performed using simple linear regression analysis of the relationship between the eye position of both eyes (subject's gaze position) measured by VOG and the target position calculated from the center coordinates (center of gravity coordinates) of the bounding box predicted by SSD, and the relationship between SPEM speed and target speed.

Fig. 10 is a diagram showing a representative example of the results of the embodiment. In detail, Fig. 10 is a diagram showing a second image (an image including the visual target 10) captured by the second image capturing unit 12, merged with the subject's gaze position (the subject's gaze position measured by the gaze position measuring unit 13B) at the time the second image was captured.
In FIG. 10, "right eye" indicates the gaze position of the subject's right eye (the gaze position of the subject's right eye measured by the gaze position measuring unit 13B), and "left eye" indicates the gaze position of the subject's left eye (the gaze position of the subject's left eye measured by the gaze position measuring unit 13B).
In the example shown in FIG. 10, the SSD recognized the visual target 10 with high accuracy, and the AP75 in the test data was 97.7%.

Fig. 11 is a diagram visualizing the relationship between the horizontal and vertical eye positions and the target position in the typical example shown in Fig. 10. In detail, Fig. 11(A) shows the relationship between the horizontal eye position and the target position in the typical example, and Fig. 11(B) shows the relationship between the vertical eye position and the target position in the typical example.
11(A) and 11(B), “Dominant eye” indicates the gaze position of the dominant eye, “Nondominant eye” indicates the gaze position of the nondominant eye, and “Target” indicates the position of the visual target 10.

Fig. 12 is a diagram showing the relationship between the eye position (subject's gaze position) obtained in the example and the position of the visual target 10. In detail, Fig. 12(A) shows the relationship between the horizontal eye position (horizontal axis) of the dominant eye and the horizontal position (vertical axis) of the visual target 10, and Fig. 12(B) shows the relationship between the horizontal eye position (horizontal axis) of the non-dominant eye and the horizontal position (vertical axis) of the visual target 10.
As shown in Figures 12(A) and 12(B), it was confirmed that the horizontal eye position had a significant positive correlation with the horizontal target position for both eyes.

Fig. 13 is a diagram showing the relationship between SPEM velocity and target velocity obtained in the embodiment. In detail, Fig. 13(A) shows the relationship between the horizontal SPEM velocity (horizontal axis) of the dominant eye and the horizontal target velocity (vertical axis), and Fig. 13(B) shows the relationship between the horizontal SPEM velocity (horizontal axis) of the non-dominant eye and the horizontal target velocity (vertical axis).
As shown in Figures 13(A) and 13(B), it was confirmed that the horizontal SPEM velocity had a significant positive correlation with the horizontal target velocity for both eyes.

As described above, in this embodiment, it was confirmed that the SSD detects the visual target with high accuracy. It was confirmed that there is a significant positive correlation between the eye positions of both eyes detected by VOG and the visual target position calculated using the SSD.

By using the automatic eye movement recording system 1 of the first embodiment, the eye movement of a subject can be quantified and visualized without changing the examination method in ophthalmological clinical practice.
Furthermore, by using the automatic eye movement recording system 1 of the first embodiment, it is possible to compare the results with those of tests carried out in the past, and to clarify the criteria for diagnosing a disease.
Furthermore, by using an optotype 10 that is of interest to children as the optotype 10 whose position can be calculated with high accuracy using the SSD, it is possible to reduce the risk that the subject (child) will not track the optotype 10 during the eye tracking test, and as a result, the test time can be shortened.

[Second embodiment]
A second embodiment of the automatic eye movement recording system, the calculation device, the calculation method, and the program of the present invention will be described below.
Except for the points described below, the automatic eye movement recording system 1 of the second embodiment is configured similarly to the automatic eye movement recording system 1 of the above-mentioned first embodiment. Therefore, the automatic eye movement recording system 1 of the second embodiment can achieve the same effects as the automatic eye movement recording system 1 of the above-mentioned first embodiment, except for the points described below.

As described above, in the automatic eye movement recording system 1 of the first embodiment, the two-dimensional region specifying unit 13D uses SSD as an object recognition algorithm.
On the other hand, in the automatic eye movement recording system 1 of the second embodiment, the two-dimensional region identification unit 13D uses an object recognition algorithm other than SSD (e.g., R-CNN, YOLO, etc.), or uses the raster scan method as the object detection method.

Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and appropriate modifications can be made without departing from the spirit of the present invention. The configurations described in the above-mentioned embodiments and examples may be combined.

It should be noted that all or part of the automatic eye movement recording system 1 in the above embodiment may be realized by dedicated hardware, or may be realized by a memory and a microprocessor.
In addition, all or part of the automatic eye movement recording system 1 may be composed of a memory and a CPU (central processing unit), and the functions of each part of each system may be realized by loading into memory and executing the programs for realizing those functions.
A program for realizing all or part of the functions of the automatic eye movement recording system 1 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to process each part. Note that the term "computer system" here includes hardware such as the OS and peripheral devices. In addition, if a WWW system is used, the term "computer system" also includes the home page providing environment (or display environment).
In addition, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes devices that dynamically hold a program for a short period of time, such as communication lines when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

1...Automatic eye movement recording system, 10...Target, 10A...2-dimensional area, 11...First image capture unit, 12...Second image capture unit, 13...Calculation device, 13A...First image acquisition unit, 13B...Gaze position measurement unit, 13C...Second image acquisition unit, 13D...2-dimensional area identification unit, 13E...Target position calculation unit, 13F...Merge processing unit, 13G...Position deviation amount calculation unit, 13H...Movement speed difference calculation unit

Claims (8)

A first image capturing unit that captures a first image which is an image of the subject's eye;
A second image capturing unit captures a second image including an eye target used in an eye tracking test of the subject;
A system for automatically recording eye movement comprising:
The computing device includes:
a first image acquisition unit that acquires the first image captured by the first imaging unit;
a gaze position measuring unit that measures a gaze position of a subject based on the first image acquired by the first image acquiring unit by using VOG (Video-oculography);
A second image acquisition unit that acquires the second image captured by the second imaging unit;
a two-dimensional area specifying unit that specifies a two-dimensional area corresponding to the visual target included in the second image acquired by the second image acquisition unit by using an object recognition algorithm;
a target position calculation unit that calculates a position of the target based on the two-dimensional area specified by the two-dimensional area specification unit;
a merging processing unit that merges the gaze position of the subject measured by the gaze position measuring unit, the gaze position of the subject at the time when the second image acquired by the second image acquiring unit was photographed by the second photographing unit, into the second image,
Automatic eye movement recording system. The two-dimensional region specifying unit uses SSD (Single Shot MultiBox Detector) as the object recognition algorithm.
2. The automatic eye movement recording system according to claim 1. the two-dimensional region is a bounding box;
The target position calculation unit calculates a center of gravity position of the bounding box as the target position.
3. The automatic eye movement recording system according to claim 2. The computing device includes:
a position deviation amount calculation unit that calculates a deviation amount of the gaze position of the subject measured by the gaze position measurement unit with respect to the position of the gaze target calculated by the gaze target position calculation unit,
2. The automatic eye movement recording system according to claim 1. The computing device includes:
a movement speed difference calculation unit that calculates a difference between a movement speed of the position of the visual target calculated by the visual target position calculation unit and a movement speed of the gaze position of the subject measured by the gaze position measurement unit,
2. The automatic eye movement recording system according to claim 1. a first image acquisition unit that acquires a first image, which is an image of the subject's eye captured by the first imaging unit;
a gaze position measuring unit that measures a gaze position of a subject based on the first image acquired by the first image acquiring unit by using a VOG;
a second image acquisition unit that acquires a second image, which is an image including an optotype used in an eye-tracking test of the subject, captured by the second imaging unit;
a two-dimensional area specifying unit that specifies a two-dimensional area corresponding to the visual target included in the second image acquired by the second image acquisition unit by using an object recognition algorithm;
a target position calculation unit that calculates a position of the target based on the two-dimensional area specified by the two-dimensional area specification unit;
a merging processing unit that merges the gaze position of the subject measured by the gaze position measuring unit, the gaze position of the subject at the time when the second image acquired by the second image acquiring unit was photographed by the second photographing unit, into the second image,
Calculation device. a first image acquiring step of acquiring a first image which is an image of the eye of the subject captured by a first image capturing unit;
a gaze position measuring step of measuring a gaze position of a subject based on the first image acquired in the first image acquiring step by using a VOG;
A second image acquisition step of acquiring a second image including an optotype used in an eye-tracking test of the subject, the second image being captured by the second imaging unit;
a two-dimensional area specifying step of specifying a two-dimensional area corresponding to the visual target included in the second image acquired in the second image acquiring step by using an object recognition algorithm;
a target position calculation step of calculating a position of the target based on the two-dimensional area specified in the two-dimensional area specification step;
a merging process step of merging the gaze position of the subject measured in the gaze position measuring step, the gaze position of the subject at the time when the second image acquired in the second image acquiring step was photographed by the second photographing unit, into the second image.
Calculation method. On the computer,
a first image acquiring step of acquiring a first image which is an image of the eye of the subject captured by a first image capturing unit;
a gaze position measuring step of measuring a gaze position of a subject based on the first image acquired in the first image acquiring step by using a VOG;
A second image acquisition step of acquiring a second image including an optotype used in an eye-tracking test of the subject, the second image being captured by the second imaging unit;
a two-dimensional area specifying step of specifying a two-dimensional area corresponding to the visual target included in the second image acquired in the second image acquiring step by using an object recognition algorithm;
a target position calculation step of calculating a position of the target based on the two-dimensional area specified in the two-dimensional area specification step;
a merging processing step of merging the gaze position of the subject measured in the gaze position measurement step, which is the gaze position of the subject at the time the second image acquired in the second image acquisition step was photographed by the second photographing unit, into the second image.

Images