JPH0738848B2 - Eye tracer - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION [Industrial field of application] The present invention recognizes a pattern obtained by performing a fast Fourier transform of an ocular myoelectric signal detected from a skin surface electrode and performing frequency analysis by a neural network to detect a line of sight. The present invention relates to a line-of-sight tracer that detects the direction of the line of sight and outputs the line-of-sight direction.

[Conventional technology]

Conventionally, as a method for detecting the direction of the line of sight of a person, firstly,
The near infrared light emitted from the LED mounted close to the eyeball is applied to the eyeball, and the reflected light is received by the phototransistor, and the intensity of the reflected light varies depending on the location of the eyeball. There was a method to detect the line-of-sight direction from the two-dimensional intensity distribution, but it is not easy to calibrate the optical axes of the system components LED, eyeball, and phototransistor as a light-receiving element, and the line-of-sight direction depends on the calibration accuracy. There is a problem that the measurement accuracy of the line-of-sight angle during measurement is greatly affected.

Secondly, as a method for clinically examining the movement of the eyeball, there is a nystagmus that examines the movement of the eyeball by attaching skin surface electrodes to several places on the muscle close to the eyeball. The high-frequency component of the myoelectric signal obtained from the moving muscle is blocked, and the measurer can visually recognize the nystagmus recorded on the recording paper and detect the direction of movement of the eyeball.
It was necessary to attach the skin surface electrode directly above the muscle that is directly involved in eye movement. Further, the eye movement, that is, the angle of the line of sight cannot be detected.

The nystagmus is to detect the electromyogram obtained from the muscles for eye movements, and the relationship between the electromyography and the movements of the eyeball is to recognize the hand movement from the electromyography for the conventional EMG artificial hand. As an application of the automatic method, (1) In the case where the electrode is mounted directly on the muscle directly involved in the eye movement as described above, the obtained myoelectricity is smoothed and rectified, and if the set range value is exceeded, A method of detecting that the eyeball has moved, and (2) a method of detecting the eyeball movement direction by a linear discriminant function with respect to the obtained myoelectricity band spectrum filtered myoelectric frequency spectrum of the myoelectricity can be easily considered. However, if there is noise due to the nature of the linear discriminant function, or if several myoelectrics of eye muscle movements are mixed at the same time, learning of the discriminant function does not converge, or movements cannot be recognized. Angle detection Enough to has a drawback that can not be.

[Problems to be Solved by the Invention]

In the above-mentioned conventional technique, the method using the first near-infrared light has a problem that calibration in setting the optical axes of the eyeball and the sensor is not simple, and detects the myoelectricity of the eye muscle in the second eyeball movement. The method based on the nystagmus that is usually performed by the operator usually only visually detects the rough eye movement direction, and at the same time, there is a problem that the mounting position of the skin surface electrode is required to be strict. A method of applying the conventional myoelectric recognition technology found in the artificial hand can be easily considered, but the application of the conventional method has problems such as convergence of a discriminant function for recognition and poor recognition rate.

An object of the present invention is to provide a line-of-sight tracer that can accurately and easily measure the line of sight by recognizing and learning the ocular myoelectric pattern.

[Means for Solving the Problems]

The line-of-sight tracer according to the present invention includes an ophthalmo-myoelectric signal detection unit that detects an ocular myoelectric signal, a fast Fourier transform unit that performs a fast Fourier transform on the ocular myoelectric signal to perform a frequency analysis, and a frequency analysis is performed. Matrix pattern conversion unit that converts the signal component level for each bandwidth of the ocular myoelectric signal into a numerical matrix pattern, and a neural network having a learning function consisting of a plurality of units and weighted links connecting the units And a recognition unit that receives the numerical matrix pattern as an input and outputs the line-of-sight direction of the eyeball corresponding to the pattern, and a line-of-sight that outputs learning-use line-of-sight angle teaching data to the recognition unit that is configured by the neural network. A direction angle teacher data generator, and.

[Action]

According to the present invention, when using, the eye-gaze direction angle teacher data generation unit by chasing the target with the eyes presents the teacher data to the recognition neural network and learns the recognition unit to attach the skin surface electrode. Has a large degree of freedom, and after the learning of the recognition unit, the gaze direction and the gaze angle can be detected independently for the left and right eyeballs.

〔Example〕

FIG. 1 is a block diagram showing the overall construction of an embodiment of the present invention. In the following subscripts of signals, L is a left eyeball, R is a symbol related to the right eyeball, t is a teacher signal, y is a yaw angle, and p is a pitch angle. Then, when the distinction by the subscript is not necessary, the subscript is taken.

In FIG. 1, 1-L1,1-R1,1-L2,1-R2 are skin surface electrodes, 2 is a spectacle frame, and each skin surface electrode 1-L1,1-R1 is a spectacle frame 2 Also serves as a nose, and each skin surface electrode 1-L2,1-R2 is a glasses frame 2
Located on the ear ears of. 3-L, 3-R is an amplifier, 4-L, 4-
R is a low-cut filter, and 5-L and 5-R are high-cut filters, which constitute the ocular myoelectric signal detection unit. 6-L and 6-R are fast Fourier transform units, 7-L and 7-R are matrix pattern conversion units, 8-L and 8-R are recognition units, and 9
Is a gaze direction angle teacher data creation unit. Also, a-
L1, a-L2, a-R1, a-R2 are the skin surface electrodes 1-L1,1-L
Signals detected by 2,1-R1 and 1-R2, bL and bR are myoelectric signals, cL and cR are myoelectric signals with low frequency components blocked, and dL and dR are myoelectric signals cL and cR, respectively. On the other hand, high-frequency components are blocked, eL and eR are FFT-transformed myoelectric frequency-analyzed signals, and fL and fR are frequency-analyzed myoelectric signal frequency-analyzed signal component level numerical matrix patterns , ΘyL, θy
R is the left / right eye direction angle, the right eye's left / right eye direction angle, θpL, θpR is the left eye up / down eye direction angle, the right eye up / down eye direction angle, θtyL, θtyR are the left eye left / right eye direction angle teacher data, respectively. Right / left eye gaze angle teacher data, θtpL, θtpR are left eye up / down eye direction angle teacher data and right eye up / down eye direction angle teacher data, respectively.

FIG. 2 is a view direction angle teacher data creation unit 9 of FIG.
9A and 9B show the details of the example of the configuration, 9-1 is a display for displaying a line-of-sight target, 9-2 is a forehead pressing portion for fixing the head, and 9
-3 is a collar holding part for fixing the head, 9-4 is a scale for measuring the distance between the displays for one eye gaze target display, 9-5
Is a moving pointer for the measurement operator, 9-6 is a line-of-sight direction angle calculation processing section, and K is a gazing point.

3 (a) and 3 (b) are views for explaining the generation of the viewing direction angle teacher data, where K is the gazing point, E-L and ER are the left eyeballs,
Shows the right eye.

FIG. 4 shows the details of the spectacle frame 2. Reference numeral 2 is a spectacle frame, which is composed of a lens 2-1 and an ear wheel part 2-2, and skin surface electrodes 1-L1,1-R1,1-L2, 1-R2 is provided. 2-3 is a lead wire from the electrode.

FIG. 5 is a configuration example of the recognition unit 8 in FIG. 1, 8-1 is an input pattern for recognition, 8-2 is an output pattern, 8-3 is a teacher signal pattern for learning, and 8-4 is a neural network. Input layer, 8
-5 is an intermediate layer of the neural network, and 8-6 is an output layer of the neural network.

FIG. 6 is a block diagram showing the overall configuration in the learning mode,
Reference numeral 9-7 is a target control unit, and the others are the same as those shown in FIGS. 1 and 2.

7 and 8 are flow charts of the process of the present invention, and FIGS. 9 (a) to 9 (c) are diagrams showing the process of converting myoelectric signals.

The present invention is broadly classified into two modes, a learning mode (A) for a neural network and a recognition mode (B) for recognizing the line-of-sight direction angle from the detected myoelectricity near the eyeball, based on FIGS. 1 to 9. Will be explained.

First, the learning mode of the neural network of the recognition unit 8 will be described.

First, the outline will be described. The recognition unit 8 is composed of a neural network as an input of the myoelectric potential pattern from the spectacle frame 2 and an output of the line-of-sight angle calculated by the line-of-sight direction angle teacher generator 9. The recognition unit 8 stores the correspondence between these inputs and outputs. By repeating this, the line-of-sight direction angle corresponding to the input pattern is output in the recognition mode.

Now, the measurement operator or the examiner looks into the distance and measures the eyeball interval l ₀ (Fig. 3 (a)) of the subject.

Next, in FIG. 2, the subject wears the spectacles frame 2, puts the collar on the collar holding portion 9-3 of the line-of-sight direction angle teacher data generating portion 9 as it is, and gently presses the forehead against the forehead holding portion 9-2. Fix the head. The operator aligns the measuring operator's pointer 9-5 with the eyeball surface from the right 90 ° direction of the subject, and reads the scale 9-4 to measure the distance between the subject's eyeball and the eye-gaze target display 9-1. Next, the line of sight is adjusted to various target positions, and the neural network of the recognition unit 8 is made to learn the relationship between the simultaneously measured myoelectric pattern and the line-of-sight angle calculated from the target position.

Next, referring to FIGS. 3 (a) and 3 (b), the target displayed on the visual line target display 9-1 and
The relationship between the line-of-sight direction angles will be described.

3 (a), the distance of the eye displayed gaze target display for the display 9-1 in (b), the left and right eyeballs center spacing distance l ₀ are known measured by the above as an operator, eyeball radius l ₂ is Although there are individual differences, the average value of the normal human eye radius l ₂ is used as a constant. Now, the line-of-sight target on the line-of-sight target display 9-1 moves to the left from the front in the middle of the left and right eyeballs EL and ER,
With respect to the yaw direction, the front left side of the eyeball is a + angle, the right side is a − angle, and the middle of the left and right eyeballs is + with respect to the front left and the right side with respect to l ₃ , the line-of-sight angle θyL of the left eye is relative to the yaw direction. Is And the right eye gaze direction angle θyR is Is. Further, in FIG. 3 (b), the line-of-sight target moves upward on the display 9-1, and the upper side of the face in the pitch direction is a + angle, and the lower side is a − angle,
When the horizontal direction is + and the downward direction is − with respect to l ₄ , the angle θpL of the line of sight of the left eye is And the right eye gaze direction angle θpR is Is.

The line-of-sight direction angle obtained above is given to the output layer 8-6 of the neural network as normalization data, which will be described later, as teacher data for the neural network of the recognition unit 8.

Next, mainly based on FIGS. 5 and 9, the skin surface electrode 1
The conversion process of the myoelectric signal detected from -L1,1-R1,1-L2,1-R2 is shown. The line-of-sight direction angle teacher data generator 9
Causes the subject to display the line of sight 9-
The signal a detected by the skin surface electrode 1 in FIG.
The left and right eyes are separately amplified by the L and 3-R, and become the time c myoelectric signal b through the low cut filters 4-L and 4-R to become the signal c, and the high cut filter 5-L and
It becomes a signal d through 5-R. FIG. 9A shows an example in which the signal d is measured at a low cut frequency of 50 Hz and a high cut frequency of 15000 Hz. The time-series myoelectric signal d is taken into the fast Fourier transform units 6-L and 6-R for the lengths of the time windows during the movement of each eyeball, and one of the detected signal d is detected in the movement state of each eyeball. / 3 octave frequency Converted to pattern e analyzed. 30 as pattern e
An example of fast Fourier transform from 50 Hz to 500 Hz in band 1/3 octave is shown in FIG. 9 (b). The signal level of the pattern e is converted into a numerical matrix f (pattern f) for each bandwidth by the matrix conversion means. Hereinafter, learning of the neural network of the recognition unit 8 will be described.

In the example of FIG. 5, the neural network has an input layer 8-4 of 10 units, an intermediate layer 8-5 of 6 units, and an output layer 8-6 of 2 units. The 10 units of the input layer 8-4 correspond to the level values of the 10 bands of the pattern f subjected to the fast Fourier transform, and the 2 units of the output layer 8-6 include the two eyeballs and the viewing direction of the eyeball on one side. It corresponds to two yaw angles and pitch angles. The numerical matrix f of the fast Fourier transform is input to the input layer 8-4 of the recognition unit 8. At the same time, the output layer 8-6 of the recognition unit 8 is provided with the eye-gaze direction angle data θyL, θpL, θyR, θyR corresponding to the pattern f as teacher signal data θtyL, θtyL, θtyR, θtpR. here,
Due to the nature of the neural network of the recognition unit 8, the teacher data is As described above, it is given to the recognition unit 8 after being normalized. The neural network of the recognition unit 8 is learned by inputting the pattern f and the pattern θt a plurality of times with respect to eye movements in the same line-of-sight direction or different line-of-sight directions, for example, the back propagation method (reference DE Rumelhart et al., Parallel Distribute).
d Processing, MIT Press 1986). For example, when the pattern f is input to the neural network, the eye direction angles θyL, θpL, θyR, θpR of the neural network are compared with the teacher signal data θtyL, θtpL, θtyR, θtpR, and both values of each unit are both. The learning ends when the error of is within 1%. FIG. 7 summarizes the flow of the learning mode (A) described above.

Next, the recognition mode (B) of the neural network will be described. The recognition mode (B) is a myoelectric potential pattern detected by the skin surface electrode 1 attached to the spectacle frame 2 in the free state of the line of sight of the subject when the subject leaves the head from the line-of-sight angle teacher data generator 9. This is a mode in which the neural network of the recognition unit 8 recognizes and outputs the line-of-sight direction angle.
This will be described with reference to FIGS. 1 and 8. When a subject wearing the spectacle frame 2 moves the eyeball, the time-series myoelectric signals detected during eyeball movement are discretized and frequency-analyzed for each time window of the fast Fourier transform opened on the time axis, and matrix pattern conversion is performed. The pattern e is input to the unit 7. The pattern e is converted as the pattern f as in the learning mode (A). The pattern f outputs the line-of-sight angle θyL, θpL, θyR, θyR of the corresponding eyeball based on the pattern already learned in the learning mode (A). Also, for an unlearned pattern, the output value interpolated by the general property of the neural network is output. Here, the output line-of-sight direction angle is a value normalized between 0 and 1 due to the nature of the neural network, and is the inverse of the teaching data taught to the output layer 8-6 in the learning mode (A). It is converted and output as an angle value.

In the present embodiment, an example in which the skin surface electrode 1 is attached to the nose pad and the ear strap portion 2-2 of the spectacle frame 2 has been shown.
Affixing the skin surface electrode 1 separately without mounting it on the eyeglass frame 2 and setting the electrode position to another place near the eyeball,
The skin surface electrode 1 is built in the contact area between the ski goggles and the face, the number of pairs of the skin surface electrode 1 is increased, and the signal from each electrode is processed by the fast Fourier transform to be converted into a level numerical pattern for each band. Then 1 for each eyeball
Inputting patterns not only from pairs of electrodes but also from several pairs of electrodes, bandwidth of fast Fourier transform processing,
Changing the number of bands, using a neural network having the ability to separate linearly inseparable patterns other than the backpropagation method according to the attribute of the pattern in the neural network of the recognition unit 8, and viewing direction angle teaching data Instead of detecting the line-of-sight direction in the generation unit 9 by tracking the target, in an image in which the eyeball is captured by the camera from the front,
The method of extracting the image of the black-eye part of the eyeball, detecting the line-of-sight direction from the distortion of the true-eye circle of the eyeball, and generating the teacher data of the line-of-sight direction for the recognition unit 8 can be implemented independently.

〔The invention's effect〕

As described above, the present invention is a line-of-sight tracer that recognizes myoelectric signals from a skin surface electrode mounted near the eyeball and outputs the angle of the line-of-sight direction, the skin surface equipped at a site away from the eye muscle. The myoelectric signal from the electrode is a mixture of signals from many muscles, but it is separated into muscle activities of each muscle to some extent by frequency analysis by fast Fourier transform, and further to some extent after fast Fourier transform. It has the ability to discriminate linearly inseparable patterns according to pattern attributes from the relationship between the pattern in which activities are separated and the eye direction angle of the eyeball. Since the neural network recognizes the myoelectricity detected from the electrodes and detects the line-of-sight angle, the pattern features of the neural network are automated and similar patterns are detected. High identification of over emissions from the high resistance capability,
Even if the electrode is not located directly above the eye muscles in the learning mode, even if there are many learning patterns or if noise is mixed in the learning pattern, the learning is converged and the electrode position shift or the myograph is recognized in the recognition mode. Even if noise or artifacts are mixed in the signal, several types of muscle groups move in parallel Even if the electrodes are not located directly above the eye muscles during eye movement, the myoelectric potentials generated by several types of muscle groups are nonlinear at the same time. Since the eye-gaze direction angle, which is the movement of the corresponding eyeball, can be detected from the skin-surface electrode that interferes with and mixes with the skin-surface electrode, the position where the skin-surface electrode is attached is limited, as in the conventional measurement of nystagmus. In addition, since the degree of freedom regarding the electrode position is large, it is possible to incorporate the skin surface electrode at the contact position between the normal eyeglass frame and the face, and there is an advantage that the aesthetic appearance of the subject during measurement is unlikely to be impaired.
In addition, since the optical direction is not used in the detection of the line-of-sight direction when recognizing the line-of-sight direction, there is an advantage that calibration such as adjustment of the optical axis is not necessary. Further, there is an advantage that the viewing direction angles of the left and right eyes can be independently detected.

Application fields include use as a line-of-sight tracer as a tool for ergonomic research and clinical medicine research, and the robot operator's eye movement of the line-of-sight direction of the robot's visual device when operating a remote robot with a visual device. Use for remote control by aircraft, supplementing and tracking targets by visual field of aircraft pilot, pointing tool as a computer-assisted self-help tool for people with disabilities, flight simulator visual system, near the visual field of the pilot to reduce the computational load of CG It can be used as a line-of-sight tracer for high-definition images, as a cursor pointing tool for computer operators, and as an eye movement measurement device for REM sleep detection.

[Brief description of drawings]

FIG. 1 is an overall configuration diagram showing an embodiment of the present invention, and FIG.
FIG. 1 is a diagram showing a configuration example of a gaze direction angle teacher data generation unit in FIG. 1, FIGS. 3 (a) and 3 (b) are diagrams explaining gaze direction angle teacher data generation, and FIG. FIG. 5 is a perspective view showing details of the eyeglass frame in FIG.
FIG. 1 is a diagram showing an example of the configuration of a recognition unit by a neural network in FIG. 1, FIG. 6 is an explanatory diagram of a learning mode in the present invention, and FIGS. 9 (a) to 9 (c) are diagrams showing a process of converting myoelectric signals. In the figure, 1 is a skin surface electrode, 2 is a spectacle frame, 3 is an amplifier, 4 is a low cut filter, 5 is a high cut filter, 6 is a fast Fourier transform unit, 7 is a matrix pattern conversion unit, 8 is a recognition unit, and 9 is a line of sight. A direction angle teacher data generation unit, E is an eyeball.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI technical display location 8825-4C A61B 5/10 310 A

Claims (1)

[Claims] 1. An ophthalmo-myoelectric signal detecting section for detecting an ocular myoelectric signal, a fast Fourier transform section for performing a fast Fourier transform on the ocular myoelectric signal to perform a frequency analysis, and the frequency-analyzed ocular muscle A matrix pattern conversion unit for converting the signal component level for each bandwidth of the myoelectric signal into a numerical matrix pattern, and a neural network having a learning function consisting of a plurality of units and a weighted link connecting the units, A recognition unit that inputs a numerical matrix pattern and outputs the eye gaze direction corresponding to the pattern, and a gaze direction angle teacher that outputs learning gaze angle teacher data to the recognition unit configured by the neural network. A line-of-sight tracer, comprising: