JPS61243508A - communication device - Google Patents

Abstract

PURPOSE:To transmit what a patient intends to others by detecting it by an eyeball motion detector that his eyes are turned on a specific gaze point. CONSTITUTION:The patient wears an eyeball motion sensor 2, and the sensor output signal obtained by watching a presentation monitor 8 is introduced to an eyeball motion analyzer 4. The sensor 2 and the analyzer 4 constitute the eyeball motion detector. With respect to detection of eyeball motion, a threshold Vth of the eyeball motion speed is set to 5 deg./sec to separate a gaze component and a motion component from each other distinctly. When the next sample point is in a circle prescribed by VthX(sampling period), it is discriminated that he sees the gaze point in the center of the circle. When his eyes are turned upon a point on the outside of this circle within one sampling time, it is discriminated that he averts hie eyes from the gaze point. The output signal of the analyzer 4 is subjected to a prescribed processing by a processor 6, and the result is presented on the presentation monitor 8 again.

Description

[Detailed Description of the Invention] [Technical Field] The present invention relates to a communication device that uses words or letters,
Alternatively, instead of conveying one's intentions to others by pressing an electrical switch, one can convey one's intentions to others simply by gazing at a specific object.

[Prior Art] Research has been carried out in various fields such as medical technology to detect the movement of the line of sight and identify a specific point of gaze. For this reason, so-called eye cameras and the like have been developed, but it is difficult for these eye cameras to identify the object to be gazed at due to fixation micromovements caused by eyeball movements.

- To take another problem in the medical field as an example, patients with amyotrophic lateral sclerosis (ALS), who are unlucky enough to be unable to move anything other than their eyeballs, are unable to move their entire muscular system. Because of his paralyzed functions, he was unable to communicate verbally as usual, and since he was unable to move his hands, he was unable to write, nor was he able to press a switch to notify an emergency situation.

Therefore, as a conventional method of reading the patient's will, as shown in Fig. 13, a large acrylic board B with letters written on it is interposed between the patient P and the attendant N, and the board B is moved and the attendant By moving the position of N's eyes, the lines of sight of both patients were brought into alignment, and thereby the letters, etc. intended by patient P were discriminated. Therefore, the amount of information that the attendant N can use to make a distinction is very small, and there is the inconvenience of not being able to leave the patient P even for a moment.

[Purpose] In view of the above-mentioned points, an object of the present invention is to detect that the line of sight has come to a specific gaze point, for example, to read the intended words and display that fact, or to call a nurse. An object of the present invention is to provide an intention communication device that can perform on/off control of television/radio broadcast lighting equipment, etc.

[Structure of the Invention] In order to achieve the above-mentioned object, the intention communication device according to the present invention includes an information presentation means for presenting information to a person who intends to communicate the intention, and information presented on the information presentation means. To detect a gaze plane surrounded by a finite frame that is set up so that the transmitter looks at the information intended by the transmitter, and whether or not the transmitter is gazing at the gaze surface. an eye movement detector, wherein the eye movement detector is generated based on the communicator gazing at the gaze plane when the information presented on the information presentation means is information intended by the communicator. The output is converted into information that can be recognized by humans, or the electrical signal is extracted as information intended by the communicator, and the person to whom the intention is communicated or the controlled device is controlled based on the communicator's will or the will, respectively. The configuration is configured to transmit it as a control signal for performing the operation.

[Example] Hereinafter, the present invention will be described in detail based on Examples.

FIG. 1 is a diagram illustrating an example of how an intention communication device to which the present invention is applied is used. In this figure, a patient P who can only move his eyeballs is fitted with an eye movement sensor 2, and a sensor output signal obtained by viewing a presentation monitor 8 is introduced into an eye movement analyzer 4. Here, both the eye movement sensor 2 and the eye movement analyzer 4 constitute an eye movement detector. Then, the output signal of the eye movement analyzer 4 is subjected to predetermined processing by the processing device 6 according to the procedure described later, and the result is presented again on the presentation monitor 8.

FIG. 2 is a block diagram showing the overall configuration of the embodiment shown in FIG. 1. In this figure, the same components as those shown in FIG. 1 are designated by the same numbers.

According to this embodiment, the eye movement state of the patient P is detected and various actions such as: ■ Creating a text using a printer; ■ Making a nurse call; ■ Turning on/off lighting equipment, television, etc.; can be controlled.

Therefore, the process of detecting eyeball movements and creating sentences will be explained next.

FIGS. 3(A) and 3(B) show the screen initially presented on the presentation monitor 8. FIG. That is, on the first presentation screen, as shown in Figure 3 (A), katakana characters and other characters in the "A" line to "C" line are displayed, and on the next screen, the third As shown in Figure (B), katakana characters, numbers, etc. starting from the "ma" line are displayed. Further, the alphabets A to Z, etc. are displayed on the screen presented third (not shown).

Here, when the screen shown in FIG. 3(A) is displayed on the presentation monitor 8, a procedure for the patient P to select one character from the screen will be described.

When the screen shown in FIG. 3(A) is displayed on the presentation monitor 8, a cursor (a rectangular frame in this embodiment) as shown in the figure is displayed.
moves from the leftmost column to the rightmost column at intervals of several seconds. The movement speed of this frame is 5 to IO at the initial stage.
Move it every second, and as you get used to it, move it every 2 to 3 seconds (adjustment means (not shown) is provided).

Now, as shown in FIG. 3(A), when the cursor C arrives at the column indicating the "S" row, the patient P will be able to see the input mark 10 displayed at the right end of the screen (the shape of the eye is marked by a rectangular frame). (surrounding the area) for 200 milliseconds or more. Then, the eye movement sensor 2 and the eye movement analyzer 4 detect this fact, and the processing device 6 notifies the patient P that the character string in the "S" line has been selected.

Then, as shown in FIG.
Only the lines are displayed on the presentation monitor 8.

In this case as well, the cursor C displayed in a rectangular frame sequentially moves from the left end to the right end at intervals of several seconds. In this example shown in FIG. 4, it is assumed that when the cursor C points to the katakana character "so", the patient P gazes at the input mark IO for more than 200 milliseconds. Due to this,
The processing device 6 determines that the character "S" has been designated S,
This character "S" is displayed in a predetermined area (not shown) of the presentation monitor 8, and the screen is returned to the display screen shown in FIG. 3(A).

By repeating the above operations, the patient P can create a desired sentence, or make corrections, deletions, etc., and then print it out using a printer or the like.

In addition, "Correction" shown in each of the above screens is an instruction to correct a single character, "Erase" is an instruction to erase a character, "Call" is an instruction to make a nurse call, and "Blank" is an instruction to delete a character.
is an instruction to insert spaces between characters when creating sentences, "control"
is a command for instructing the desired control (for example, turning on/off lighting equipment or turning on/off the television) from a predetermined menu screen (not shown) that is separately presented by selecting this. ” is a command to print out the created text.

The screens shown in FIGS. 3A, 3B, and 4 show the case where an input mark 1O is displayed at the right end. In this case, it is assumed that the patient P's eyes are in a resting state (looking at Pola) and looking at the left half of the presentation monitor 8, and the character at a specific cursor position is This is a presentation screen in which the patient P is expected to gaze at the input mark 10 at the right end only when making a selection. In other words, if the patient's P's eyes are in a resting state and are always looking near the input mark IO, this is inconvenient and may cause input errors.

Therefore, in such a case, as shown in FIG. 5, the input mark 10 can be displayed at the left end of the screen so that the input mark 10 is not seen when the eyes are in a resting state.

Next, a more detailed explanation will be given of "eye movement detection J" which is an important operating principle of the present invention.

Due to the wooden nature of the eyeballs, it is extremely difficult to fixate on just one point, and the line of sight is constantly moving (fixation micromovement). Therefore, when the gaze component and other components that occur when viewing a still image are experimentally determined, the results shown in FIG. 6 are obtained. The horizontal axis shown in Figure 6 is the speed of eye movement (deg/5e) that occurs when viewing a certain still image.
e), and the vertical axis represents its occurrence rate (%). As is clear from this figure, most of the gaze components are included within 5 degrees/second. Therefore, the eye movement speed threshold vth is set to 5 degrees/
By setting the time to seconds, it is possible to clearly separate the gaze component and the movement component.

As is clear from the above-mentioned experimental results, even if the line of sight temporarily passes the point of fixation, it cannot be immediately determined that the eyeball has seen the point of fixation based on this fact. Therefore, even if the line of sight temporarily reaches the input mark 10 on the presentation monitor 8, it cannot be determined that the patient P has seen the input mark 10.

Conversely, even if the line of sight temporarily goes beyond the input mark IO due to visual fixation micromovements, if we assume that the eyeball position is detected every 50 milliseconds, then dth = Vt.
h X sample period = 5 (deg/5ee) X 50 (msec) =
If the line of sight is within a circle having a radius defined by 0.25 (deg), it is assumed that the eyeball is looking at the point of fixation (input mark 10). In other words, as shown in Fig. 7, when the next sample point is within the circumference defined by (threshold value of eye movement velocity vth) x (sample circumference M), look at the gaze point at the center of the circle. It is determined that there is. Therefore, when the line of sight moves to a point outside the circumference within the sample time, it is determined that the point of gaze has been missed.

In the present invention, based on such recognition, it is determined whether or not the patient P has gazed at the input mark 1o of the presentation monitor 8.

Thus, if the line of sight only passes the gaze point (input mark 10), the eye movement speed will be 5 degrees/second or more, so it will be outside the range of the circle surrounded by the radius dth shown in FIG. It is not determined that the characters surrounded by cursor C have been selected.

FIG. 8 is a schematic diagram showing the configuration of the eye movement sensor 2 (see FIG. 1). This figure schematically shows the eyeball viewed from the top of the head, where "12" is an infrared light emitting diode (LED), "14" and °゛]θ°°
is a phototransistor that detects infrared rays reflected from the cornea of the eyeball. The position of the iris (
That is, the position of the eyeball) is detected. As is clear from this figure, when the iris is in the center, the amount of light (reflected light) incident on the phototransistors 14 and 18 is the same, so that outputs of the same level are obtained.

FIG. 9 shows a hardware configuration for determining whether or not the patient P gazed at the input mark IO based on the output of the eye movement sensor shown in FIG. In order to facilitate understanding, the explanation below will be based on the assumption that the patient P is gazing at input mark 0 in FIG. 3(A) when the iris of the eye is in the center (see FIG. 8). However, in reality, as mentioned above, the input mark 10 is not the place that the patient P is looking at in a resting state, so when the patient P looks at the input mark 10, the output of the differential amplifier 24, which will be described later, becomes a zero level output. , it is necessary to take circuit considerations such as giving an offset to the amplifier.

In FIG. 9, the part shown on the left surrounded by a broken line shows the entire eye movement detector consisting of the eye movement detection sensor 2 and the eye movement analyzer 4. Here, as mentioned earlier,
"12" is an LED, and "14" and "18'" are phototransistors. Also, “18” and “2”
0” is sample/hold circuit, “22” is 2 kHz
An oscillator with an oscillation frequency of z, '°24''
+Increase dPjAN, '213'' H1 kHz or less (
7) A low-pass filter that passes the signal, °27'' is an amplifier for gain adjustment.

In the processing unit, "2B" is an A/B that sends out a 12-bit digital signal with a sample interval of 50 milliseconds.
A D converter, "30'" is a latch circuit that holds the previous sampling output for l sample periods, "32" is a digital arithmetic unit that subtracts the tB power of the A/Il converter 2B from the output of the latch circuit 32,'34''' is the reference value R from which the output of the digital arithmetic unit 32 is sent from the reference value circuit 36.
A comparator that sends out a pulse of level rl only when it is smaller than 1, 38° is a counter that cumulatively adds up and counts each time a pulse of level rlJ is sent out from the comparator 34, and 40 is the counted value of the counter 38. This is a comparator that sends out a read instruction signal when R2 exceeds the reference value R2 from the reference value circuit 42, and both are included in the processing device 6 (see FIG. 2).

Next, the operation of the above circuit will be explained.

When the patient P looks at the presentation monitor 8, a DC output signal corresponding to the position of the iris (position in the horizontal direction) is obtained from the phototransistor 14.18. These output signals are sampled and held by sample/hold circuits 18 and 20 and introduced into differential amplifier 24. Therefore, when the iris of the eyeball is perfectly centered, a zero level eye movement detection output is obtained from the differential amplifier 24.

The output of the differential amplifier 24 is passed through a low-pass filter 26 to remove unnecessary noise components, and then passed through a gain adjustment amplifier 27.
will be introduced in This amplifier 27 is adjusted in advance so that outputs of +5 volts and -5 volts are obtained when the iris moves by 15 degrees to the left and right, respectively. That is, the output voltage of the amplifier 27 is adjusted in advance to have a range of +5 volts to -5 volts.

Since the A/D converter 28 outputs a 12-bit digital signal, it is possible to determine the state of 4086 levels. In this example, the iris is 15 degrees to the left and right (
Since it is assumed that the distance is moved by a total of 30 degrees, by converting it into a digital signal, 30 (deg) + 4H
f3 = O, GO? (deg) of resolution can be obtained.

Also, as mentioned in relation to Figure 7, the radius dth of the circle defined by vth x sample period is 5 (deg/5ec) x 50 (msec) =
It becomes 0.25 (deg). Expressing this as a digital value, the resolution when digitized is 0.007 (d
eg), so 0.25 (deg) ÷ 0.007 (deg) = 34
. . . 1O decimal display - toooto . . . Binary display. Therefore, the reference value R introduced into the comparator 34
1 may be set to °゛3 (100010 in binary representation).

If the output of the digital arithmetic unit 32 is '34'' or less, it means that the eyeball has seen the gaze point (that is, the input mark lO; see FIG. 3(A)), so a pulse of level rlJ is output. Ru.

However, in this embodiment, the indicated position of the cursor is determined only when the patient P gazes at the input mark 10 for 200 milliJ seconds or more, so four output pulses are sent to the comparator 34. The time when it is sent from (2
(corresponding to the elapse of 00 milliseconds), the C of the processing device 6
The PU (not shown) is made to read the character, etc. at the cursor position. Therefore, the reference value R2 introduced into the comparator 40 is set to R2=4.

Mata, R2<4. In other words, the digital arithmetic unit 32
When the output exceeds 34゛, in this case, it is determined that the patient P has canceled the expression of intention (gazing at the input divided by 10) midway through, and the comparator 34 resets the signal.
- Apply a pulse to reset the count value of the counter 38 to zero.

In the embodiments described so far, the sensors were arranged to detect only the movement of the eyeball in the horizontal direction, but by adding a sensor that detects the movement of the eyeball in the downward direction, two-dimensional It is also possible to instantly identify the gaze position.

Furthermore, in the above embodiments, applications in the medical field have been explained, but it is also possible to use a person in normal health whose head position is fixed and only the black eye can move (for example, a rocket shooter). It is also possible to apply this to astronauts (of the time) to communicate and perform various controls using only eye movements.

Furthermore, even when the head position cannot be fixed, head movements can be detected by attaching an ultra-compact television camera (such as a CCO camera) to the top of the head or forehead and determining the video output. It is also possible to communicate intentions or perform various controls by compensating for the movement of the eyeballs.

Finally, we will outline examples of applications in such cases.

First, an example of application to a broadcasting system will be shown (see FIG. 10).

A cameraman, a person in charge of a VTR, or a person in charge of a graphic screen such as AMeDAS can know that the screen in question is on air by the lighting state of the tally lamp. However, when they are not on air, they do not know when their cameras will be switched on. Therefore, if it were possible to display the producer's or technical director's line of sight movements point by point on the camera's viewfinder, they would be able to quickly and accurately know when their own camera, etc., will be switched on, and they would be able to do so with peace of mind even during standby. This will allow you to move the camera and adjust the image.

In order to meet these demands, the producer or technical director is equipped with a single-ball motion sensor H and an ultra-compact camera■, which calculates the frame output of the camera to detect head movements and compare them with the movements of the eyeballs (black eyes). By taking the difference, it is possible to determine the relative movement amount of the line of sight and specify the object of gaze. In this case, the relationship between the position of the line of sight and the monitors (monitor TVs A to G) is stored in advance in the gaze target determination unit, and is used as a standard for determining which monitor the line of sight is gazing at. There is a need.

Second, an example of application to a monitoring system will be shown (see FIG. 11).

In this case as well, it is necessary to memorize in advance the movement of the line of sight of the monitoring expert with respect to the monitor television and meters. However, it is necessary to memorize not only the response when the monitored object is normal, but also the movement of the line of sight in an emergency. In this example, as shown on the right side of Figure 11, we configure an infinite surveillance system equipped only with surveillance cameras, reproduce the movement of the human gaze that has already been memorized, and thereby respond to various situations. Perform monitoring.

Third, an example of application to a monitoring robot that functions as a guard will be shown (see FIG. 12).

As in the case of the second application example, the movement of the robot is controlled by storing the movement of the line of sight of an expert in advance and giving that function to the robot's eyes (camera). By configuring such a system, it becomes an effective means for monitoring conditions in places where humans cannot enter, such as safety monitoring inside a nuclear reactor.

Effects As described above, the present invention not only tracks and detects the movement of the line of sight like an eye camera, but also quickly identifies whether or not the user is actually gazing at a specific gaze point. For example, it is possible to convey the will of a specific person to a third party without any mechanical operation such as pressing an electrical switch, or to perform various types of control on a controlled object according to the will of a specific person. It is possible to realize special functions that were not previously available, such as the ability to

[Brief explanation of drawings]

FIG. 1 is a diagram illustrating an example of the usage of an intention communication device to which the present invention is applied, FIG. 2 is a block diagram showing the overall configuration of an embodiment shown in FIG. 1, and FIGS. 3(A) and ( B), Figures 4 and 5 are diagrams showing the display screen of the presentation monitor, Figure 6 is a diagram showing the eye movement velocity distribution when viewing a still 1F image, and Figure 7 is a diagram showing the gaze point. A diagram explaining the definition, Fig. 8 is a configuration diagram of the eye movement sensor, Fig. 9 is a block diagram showing the hardware for determining whether or not the patient has gazed at the input mark on the presentation monitor, Fig. 1θ - Figures 12 are diagrams explaining the present invention in detail, and Figure 13 is a diagram showing the prior art. 2... Eye movement sensor, 4... Eye movement analyzer, 6... Processing device, 8... Presentation monitor, 10... Input mark, 12... Light emitting diode, 14.16. ...Phototransistor, 18.20...
・Sample/hold circuit, 22... oscillator (2kHz
z), 24...Differential amplifier, 26...Low pass filter (1kHz), 27...
Gain adjustment amplifier, 28... A/D converter, 30... Latch circuit, 32... Digital arithmetic unit, 34... Comparator, 36... Reference value circuit (R1), 38...・Counter, 40... Comparator, 42... Reference value circuit (R2), P... Patient, C... Cursor, vth... Dark value of eye movement speed (5 degrees/second), dth
...If the sample point exists within a circle whose radius is this dth in the range from a certain point of gaze to the next sample point (dtb = Vth x sample period), input mark lO is What you see and the judge Patent attorney Yoshi Tani - Figure 1 Figure 8 Figure 13

Claims (1)

[Scope of Claims] 1) Information presentation means for presenting information to a person who intends to convey an intention, and when the information presented to the information presentation means is information intended by the intention communicator, the intention communicator The information presentation device is provided with a gaze plane provided to be gazed and surrounded by a finite frame, and an eye movement detector for detecting whether or not the person communicating the intent is gazing at the gaze plane, and presented to the information presentation means. When the information conveyed is the information intended by the communicator, the output of the eye movement detector generated when the communicator gazes at the gaze plane is converted into information that can be recognized by humans or converted into an electrical signal. The information is extracted as information intended by the person communicating the intention, and is transmitted to the person to whom the intention is communicated or the controlled device, respectively, as the intention of the person communicating the intention or as a control signal for performing control based on the intention. An intention communication device. 2) In the communication device according to claim 1, the information presentation means is composed of a cathode ray tube, and a plurality of pieces of information are simultaneously displayed on the image display surface of the cathode ray tube in a form that allows one piece of information to be identified. When an output is generated from the eye movement detector because the communicator is gazing at the gaze plane in response to the information presented in the discernible form, the image of the cathode ray tube is displayed. An intention communication device characterized in that only the information presented in the discernible form is left on the display screen, and the other information is erased and presented. 3) In the intention communication device according to claim 1 or 2, in detecting whether or not the person communicating the intention is gazing at the gaze plane using the eye movement detector, a predetermined observation is performed. The point of gaze changes within a radius of approximately {threshold of eye movement speed (5 degrees/second)} x {time interval from when the eye is at a given point of gaze to when it becomes the next point of gaze} with the viewpoint as the center. Even if the change occurs, the intention communication device is configured to ignore the fluctuation and consider that the predetermined gaze point is being gazed at. 4) In the intention communication device according to claim 1, the eye movement detector is configured to compensate for the movement of the head of the person communicating the intention so as to detect only the true movement of the eyeballs. Characteristic communication device. 5) The intention communication device according to claim 1, wherein the information presentation means and the viewing surface are constructed by the same means.