JPH1186005A - Operation detecting device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation detecting device which can surely detect a head shake. SOLUTION: A CCD camera 3 picks up an image of the face of a driver 14. An entirety control part 13 converts the face image, obtained by the individual illumination with 1st front lighting 1 of the driver and 2nd oblique lighting under the control of a lighting light emission control part 4, from analog to digital by an A/D converter 5. A pupil extraction part 7 inputs face images from an image memory 6 and calculates their difference to extract an area of high density through binarization. Pupil areas are detected by area and circle detection as to respective areas. A nod detection part 8 detects displacement in the center of gravity of the pupils to detect a nod from reciprocal displacement in a specific time. A navigation display control part detects an information start operation switch 12 being operated through the entirety control part, reads information confirmed by the nod out of a navigation information data base 10 at a navigation display part and completes the display.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion detection device for performing an operation instruction in operating a device by swinging an operator.

[0002]

2. Description of the Related Art Vertical and horizontal swinging of a neck is one means of expressing intention in dialogue. Therefore, the idea of applying the vertical and horizontal swing of the operator's neck to the man-machine interface has been around for a long time. For example, in the vehicular warning device disclosed in Japanese Patent Application Laid-Open No. HEI 7-117593, when the driver does not notice the warning by looking at the reaction of the driver to the first warning in the vehicle distance warning performed in two stages, Give an alarm. Here, it is assumed that the face movement can be used as the driver's reaction. In many other cases, the operation burden is reduced to incorporate the swinging motion into the operation instruction.

[0003]

However, although the use of the swing operation is pointed out in any of the above, no specific method of detecting the swing is described.
At present, it is not clear that it is practically used.
SUMMARY OF THE INVENTION It is an object of the present invention to provide a motion detection device capable of detecting a vertical and horizontal swing of a neck by image recognition and operating the device as input information of the device.

[0004]

For this purpose, image input means for imaging the face of the subject, face feature point detecting means for detecting the face feature points of the subject by image processing, and the face feature point detecting means output. Comparing means for comparing the positions of the facial feature points of temporally different images; determining means for determining whether the positions of the facial feature points compared by the comparing means have reciprocated vertically or horizontally; and And a motion detecting means for detecting a swing motion based on the determination result.

The motion detecting means calculates the time required for one swing, and determines the timing of the next detection by the facial feature point detecting means of the previous time based on the calculated time. It is preferable that the motion detecting means detects the swing motion and then performs the next detection by the facial feature point detecting means at predetermined time intervals. It is preferable that the motion detection means calculates the time required for one swing and the time interval is the time required for swing detection. The motion detection means can determine whether the head swing is vertical or horizontal based on the displacement direction of the feature point determined by the determination means. It is preferable that the facial feature point detecting means determines the detection range of the next facial feature point detecting means by referring to the previous detection position of the facial feature point. Preferably, the facial feature point is a pupil.

[0006]

[Function] If the head is swung, the feature point is displaced in conjunction with the face. Since the displacement direction of the feature point coincides with the swing direction, the swing of the feature point can be recognized by detecting the reciprocal displacement of the feature point. By actually detecting the time required for swinging and determining the next detection time based on that time, the learning effect works and the detection time can be set accurately, preventing false detection. Is obtained. If the motion detection means detects the swing motion and then performs the next detection at a predetermined time interval, erroneous detection when the swing is reciprocated multiple times carelessly is prevented.

Since the displacement direction of the feature point coincides with the manner in which the head is swung, it is possible to determine whether the head is swung vertically or horizontally from the displacement direction. The facial feature point detecting means determines the next detection range by referring to the previous detection result of the feature points, so that the area for searching for the feature points is reduced, and the amount of data to be processed is reduced. By using the pupil as the feature point, it becomes possible to use reflection and non-reflection of the retina, and a pupil image is easily extracted by calculating a difference between the two images. Further, since the pupil image is circular or elliptical, it is easily distinguished from noise having an indefinite shape, and data indicating the pupil position is easily obtained by the center of gravity calculation.

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, an embodiment in which the present invention is applied to information detection of a vehicle-mounted navigation device will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. The driver 14 is a CCD camera 3 as image input means.
The first illumination 1, which is arranged so as to be able to image the face of the subject and illuminates the face, is located on the optical axis of the CCD camera
The second lighting 2 is installed at a predetermined position beside the first lighting 1 so as to illuminate the driver from an oblique direction. The first illumination and the second illumination are configured by LEDs that emit the same near-infrared light, and the illumination intensity is adjusted so as to illuminate the face with the same intensity.

The CCD camera 3 continuously captures an image of the face of the driver 14. The overall control unit 13 controls the A / D converter 5, and the first illumination 1 and the second illumination 1 are controlled by the illumination light emission control unit 4.
A / D is a face image when lighting 2 individually illuminates.
Let it be converted. The face image signal after the A / D conversion is stored in the image memory 6. The lighting of the first illumination 1 and the second illumination 2 is performed at short time intervals, and almost the same face is shown in each image. Hereinafter, the image when the first light 1 is turned on is referred to as an image A, and the image when the second light 2 is turned on is referred to as an image B.

The image memory 6 is connected to a pupil extracting section 7 as a facial feature point detecting means. The pupil extraction unit 7 performs a difference operation between the image A and the image B. The difference image is further subjected to a binarization process using a threshold to detect a region having a high density. Each area is subjected to labeling processing and numbering is performed. In the image A, since the first illumination 1 projects light from the lens direction of the CCD camera, the pupil is observed to be brightly illuminated by the reflected light from the retina. In the image B, light projection by the second illumination is performed from an oblique direction, and the pupil is observed dark. Accordingly, the result of the difference between the image A and the image B indicates that the pupil region has been enhanced, and is extracted by the above-described binarization processing.

In the difference image, in addition to the retinal reflection image, for example, a spectacle lens reflection image, a spectacle frame reflection image from a person wearing spectacles, and a part of the face appearing due to a change in external illumination. Noise may be included. Since these noises generally have an indeterminate shape and an indeterminate area, they can be identified by processing the shape and area.
The pupil extraction unit 7 calculates the area Ri of each area subjected to the labeling process, compares the area Ri with predetermined thresholds S1 and S2, and extracts an area satisfying S1 <Ri <S2. I do. Here, S1 and S2 are values corresponding to the upper and lower limits of the pupil area estimated from the photographing magnification of the CCD camera. For example, the pupil having a diameter of 2 to 8 mm is set according to how many pixels are observed on the image.

The extracted area is calculated as a pupil when the ratio F of the area of the area to the circumscribed rectangle is calculated to be a certain value or more. That is, since the retinal reflection image is observed in a circular or elliptical shape, the ratio F is large, whereas the reflection of the spectacle frame is an elongated area along the frame. Is shaken off from the detection target by the determination of. As for other noises, a ratio F as large as that of a circle cannot be obtained, so that they are not detected and removed.

Next, calculation is performed on the center of gravity of the pupil region. Since this center of gravity can represent the pupil, it is possible to detect the driver's swing motion based on the position change. Here, the center of gravity (Xg, Yg) of the pupil is calculated based on the following equation. Note that the binarized image is C (x, y),
C (x, y) = 1 is the extracted pupil region. Xg = Σx / [total number of pixels of C (x, y) = 1] where x is the x coordinate of the pixel of C (x, y) = 1 Yg = Σy / [total number of pixels of C (x, y) = 1 Here, y is the y coordinate of the pixel of C (x, y) = 1. The nodding detecting unit 8 connected to the pupil extracting unit 7 as a motion detecting unit detects the displacement of the center of gravity of the pupil, and nods from the displacement direction. Is detected. The nod detection method will be described in detail according to a flowchart described later.

The navigation display control unit 9 is connected to the general control unit 13, and the information search start switch 12 is connected to the general control unit. The navigation display control unit 9 detects that the information search start switch 12 has been operated through the overall control unit 13, performs information search, and inputs a nod detection result to perform predetermined search control. The navigation information database 10 stores information contents necessary for information search, and the navigation display unit 11 displays an icon for information search and the searched information contents.

When an information search is required, the driver 14 presses the information search start switch 12. When the switch information is detected by the navigation display control unit 9 via the overall control unit 13, the map information being displayed on the screen searches for facility information such as a parking lot, a gas station, a convenience store, and a restaurant. For this purpose, icons indicating the facility contents are first displayed on the navigation display unit 11 on the screen. Each icon is simultaneously displayed on the screen as shown in FIGS. 4 and 5, for example, and is controlled for a predetermined time T (for example, T = 2 s) by timer control.
It is expressed by changing the coloring for each time.

The driver communicates his / her intention by nodding while the icon indicating the information to be selected is colored.
The nodding is detected from the image captured by the CCD camera 3 by the processing of the nodding detecting unit 8. For example, when the driver nods to the colored parking lot icon, the navigation display control unit 9 reads out the parking lot information from the navigation information database 10 and displays it on the navigation display unit 11. The parking lot icon changes the coloring pattern together with the display of the parking lot information as shown in FIG. 6 to indicate that the information is selected. Thus, it can be confirmed that the nod is received. When a nod is detected again, the navigation display control unit 9 terminates the screen display assuming that the search for the navigation information is to be terminated. On the display screen, the searched information disappears and the original map screen is displayed as shown in FIG.

FIG. 2 shows an installation layout of a CCD camera, lighting, and an information search start switch that are related to the driver. That is, a monitor as the navigation display unit 11 is installed on the instrument panel below the windshield of the vehicle. An information search start switch 12 is provided at the lower right of the monitor. Immediately below the monitor, the CCD camera 3 and the first illumination 1 are installed so as to form a coaxial system. The second illumination 2 is the first
It is installed next to the lighting 1 at a certain interval. The CCD camera 3 can capture the face of the driver including both eyes.

Next, the flow and control of screen display will be described with reference to the flowchart of FIG. That is, when it is detected in step 101 that the information search start switch 12 has been pressed through the overall control unit 13, an icon indicating the information content is displayed in step 102.
In the present embodiment, information on a travel route such as a restaurant, a parking lot, a gas station, and a convenience store is called for the travel route information, and is superimposed on a display screen of the travel route information. The icons are controlled by a timer, and the coloring is sequentially updated and displayed at regular intervals. FIGS. 4 and 5 show the display screens, and (a) and (b) respectively show map information and an icon whose color is changing.

In step 103, the navigation display control section 9 checks whether or not the detection result of the nodding (vertical swing of the neck) is inputted from the nodding detecting section 8, and in step 104, the navigation information of the nodding colored icon information is displayed. Read from the information database and display on the monitor. At this time, the check for nodding is continued. FIG. 6 shows a display screen when the required information is a parking lot, for example. On the screen, a parking lot icon is simultaneously displayed with a different coloring pattern to express the selected content. Then, when it is checked in step 105 that there is nodding again, the information search is terminated in step 106. As shown in FIG. 7, the monitor display screen is a screen of only the map from which the searched information and icons disappear.

Next, the detection of a nod in the nod detector 8 will be described with reference to the flowchart of FIG. In step 201, when the timer for monitoring the lapse of the processing time is initialized (t = 0), the detection of the swing is started. First, in step 202, a pupil detection result is input from the pupil extraction unit 7. In step 203, it is determined whether the input has been started and the neck has started to swing in the vertical direction by calculation with the previous input result. If the shake has not begun, the process returns to step 202 so as to input the next pupil detection result after counting the elapse of the unit time in step 204, and proceeds to step 205 if it is determined that the shake has started. That is, step 20
3, the center of gravity position of the pupil input continuously in time is compared, and the center of gravity position (Xgt, Ygt) obtained at time t and time t
For the center of gravity (Xgt + 1, Ygt + 1) obtained at +1, the downward vertical displacement (Ygt + 1−Ygt) is larger than the fixed value Th1, and the horizontal displacement (Xgt + 1)
If (−Xgt) falls within the constant value Th2, it is determined that the neck has begun to swing in the vertical direction.

After storing the time ts at which the head is swung in the vertical direction at step 205 as a nod start time, the following flow is executed to detect an upward movement appearing in the latter half of the nod. In this flow, time ta is used instead of time t so as to be distinguished from the flow, and time ts is set as an initial value. That is, in step 206 to step 209
While monitoring the passage of time, the next pupil detection result is input. By comparing the positions of the pupil barycenters obtained at time ta + 1 and time ta, an upward vertical displacement | Ygta + 1−Yg
ta | exceeds a certain value Th1, and the horizontal displacement | Xgta
When + 1−Xgta | is within the constant value Th2, it is detected that the neck has moved upward. At this time, if it is determined in step 206 that the upward movement does not occur until a certain time tmax elapses (ta−ts) from the time ts when the neck starts to swing in the vertical direction, step 20
The vertical swing of the neck detected in step 3 is rejected because it is not caused by nodding. The speed of the nod varies from person to person, but the time tmax may be set as the longest possible nod, for example, 2 seconds. The pupil detection result input is continued during the above processing.

When the upward swing of the neck is detected, a flow for checking whether the movement of the neck has stopped is executed. This is a process for preventing a heading other than a nodding that is consciously recognized, such as a result of a vibration of the vehicle when the vehicle runs on an uneven road or a drowsiness, from being detected as a nodding.
Time t at which the upward direction was detected in step 210
a is set as an initial value of a time tb at which the following processing is performed. In steps 211 to 215, the pupil detection result is continuously input while monitoring the passage of time. If it is determined from the input result that the neck cannot be shaken by two detections,
It is detected as a nod that has been noticed. That is, the vertical displacement (Ygtb + 1−Ygtb) of the pupil center of gravity obtained at time tb + 1 and time tb is calculated, and the calculated value is within a constant value Dth, and the vertical displacement of the pupil center of gravity at time tb + 2 and time tb + 3 (Ygt + 3−3). If Ygt + 2) is also within the fixed value Dth, a nod is detected. If the head movement does not stop before the elapse of time (tb-ts) from the start of the head movement at a certain time tmax in step 211, the above detection result is rejected as nodling has not occurred. Step 203, step 207, and step 208 constitute a comparison unit and a determination unit.

The present embodiment is configured as described above. The pupil is detected as a facial feature point, and its motion is tracked to detect a nod that recognizes a displacement feature. can do. The operation is complicated, and functions that can only be performed while the vehicle is stopped are controlled by natural human movements, so that the functions can be handled even while the vehicle is traveling, thereby enabling advanced traveling control. In addition, since the pupil as a feature point has a predetermined size, measurement can be performed even if the image is blurred. This allows the driver to move in the depth direction. Further, the angle of view of the CCD camera can be set wide, and the movable range of the driver in the vertical and horizontal directions can be expanded. In the embodiment, the pupil was used as a feature of the face,
However, the present invention is not limited to this. For example, a nostril or eyelashes may be used.
In this embodiment, the nod detection has been described. However, it is needless to say that a lateral movement as a negative expression can be detected by a similar detection method.

Next, a second embodiment will be described. FIG.
FIG. 6 is a block diagram showing the configuration of the second embodiment. This embodiment is different from the first embodiment in FIG.
Nod detection and nod time calculation unit 19 instead of
The configuration is such that a nodding time prediction unit 20 is used, and a pupil position prediction unit 17 is added between the image memory 6 and the pupil extraction unit 7. Others are the same as the first embodiment.

The pupil position prediction unit 17 sets a new pupil search area near the previous pupil position. That is, in the first time, the difference image is binarized in the same manner as in the first embodiment, using the entire input image as a search area, to detect a high density area, and the pupil detection target is determined by labeling processing. From the second time, a small area including the previous pupil barycenter is set as a search area. That is, the previous pupil center of gravity (Xg
t, Ygt), the points (Xgt-Xo, Ygt-Y
o) and (Xgt + Xo, Ygt + Yo) are the upper left and lower right rectangles, respectively, as the search area. The sizes xo and yo of the search area are set in consideration of the amount of movement of the eyeball caused by nodding to the photographing magnification of the CCD camera. The pupil extraction unit 7 searches the search region set by the pupil position prediction unit 17 and extracts a pupil region.

Nodding detection and nodding time calculation unit 1
9 detects a nodding in the same manner as in the first embodiment, calculates a time tr from the start of the nodding to the end of the detection, and sends the time tr required for the nodding to the nodding time predicting section 20.
The past nodding time tr is stored in the nodding time predicting section 20, and the average value to and the standard deviation tstd are calculated by adding the newly measured nodding time tr. As a result, the predicted value of the maximum value of the time required for the nodding is updated to to + 3 × tstd and sent to the nodding detection and nodding time calculating unit 19, and the maximum nodding time tmax used in the nodding detection and the nodding time calculating unit 19 is set to the above value. Update to a value. When the prediction is performed, the actual value tr required for nodding in the past is used, so that meaningless reciprocating displacement is not used for statistical processing, and the data becomes highly reliable. The nodding detection and nodding time calculating unit 19 further detects the nodding, interrupts the nodding detecting process for the calculated time required for the nodding, and starts the next nodding detection.

FIG. 10 is a flowchart for interrupting the detection process. That is, when nod detection is performed according to the flowchart of FIG. 8, the time from the start of swinging to the stop of pupil movement is calculated in step 301, and the time required for nodding is obtained as tr. Step 30
At 2, the timer is initialized. Step 303
In, a weight per unit time is inserted. Step 30
At 4, the elapsed time is counted. Step 30
At 5, it is checked whether the elapsed time t has become tr. If the elapsed time t has not reached tr, the process returns to step 303 and the above processing is repeated. If the elapsed time t exceeds the time tr in step 305, the flowchart of FIG. 8 is executed so as to detect the next nod. Thus, the time t
During this period, nod detection is interrupted, so that nod, which is performed continuously as a single nod, is not detected as multiple nods, and careless navigation no longer hinders navigation control. Is prevented.

In this embodiment, the pupil search range is limited as described above, so that the amount of data to be handled is reduced, the speed of the pupil extraction process is increased, and the influence of noise is reduced. , A highly reliable extraction effect can be obtained. Also, since the time required for nodding is accumulated and stored, and the time required for nodding based on the average and variance is the maximum time for the next detection, even if the nodding time differs depending on the individual difference, the learning effect works and the stable detection effect is obtained. can get.

[0029]

According to the present invention, since the reciprocating displacement of the facial feature point is detected and the swing is recognized, it is not necessary to collect data other than the face image of the subject, and the detection can be performed without restricting the movement of the subject. This makes it possible for the subject to issue an operation command to the device in a natural state. For example, in driving an automobile, navigation and the like can be performed even during traveling by simply taking a face image of the driver, and advanced traveling control can be performed.

If the time required for swinging is actually detected, and the next detection time is determined based on the time, a learning effect is achieved, and the detection time can be set accurately. The effect of preventing is obtained. After detecting the swing motion, the motion detection means performs the next detection at predetermined time intervals, thereby preventing erroneous detection due to a difference in the number of times of neck motion.

If the motion detecting means determines whether the head swing is vertical or horizontal based on the displacement direction of the feature point, it is possible to issue both positive and negative operation commands. When the facial feature point detecting means determines the next detection range by referring to the detection result of the previous feature point, the detection range can be limited to a small range, the amount of data to be handled is reduced, and noise is reduced. The influence is reduced and the calculation load is reduced. When the feature point is a pupil, it becomes possible to use a reflection image of light, change the irradiation angle of illumination to create a reflection image and a non-reflection image of light, and a pupil image is easily extracted by a difference operation, A high quality extraction effect is obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of the present invention.

FIG. 2 is a diagram showing an installation layout of a CCD camera and the like.

FIG. 3 is a flowchart.

FIG. 4 is a navigation display screen for displaying icon coloring.

FIG. 5 is a navigation display screen for displaying icon coloring.

FIG. 6 is a navigation display screen for displaying parking lot information.

FIG. 7 is a navigation display screen.

FIG. 8 is a flowchart for detecting a nod.

FIG. 9 is a block diagram showing a second embodiment.

FIG. 10 is a flowchart for inserting a diet.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 1st illumination 2 2nd illumination 3 CCD camera 4 Illumination light emission control part 5 A / D converter 6 Image memory 7 Pupil extraction part 8 Nodding detection part 9 Navigation display control part 10 Navigation information database 11 Navigation display part 12 Information search start Switch 13 Overall control unit 14 Driver (subject) 17 Pupil position prediction unit 18 Pupil extraction unit 19 Nodding detection and nodding time calculation unit 20 Nodding time prediction unit

Claims (7)

[Claims] 1. An image input means for capturing an image of a face of a subject, a face feature point detecting means for detecting a face feature point of the subject by image processing, and a face of a temporally different image output by the face feature point detecting means. Comparing means for comparing the positions of the feature points; determining means for determining whether the positions of the facial feature points compared by the comparing means have reciprocated vertically or horizontally; and neck based on the determination result of the determining means. An operation detecting device, comprising: an operation detecting means for detecting a swing operation. 2. The method according to claim 1, wherein the motion detecting means calculates a time required for one head swing, and a timing at which detection is performed by the next previous facial feature point detecting means is determined based on the calculated time. The motion detection device according to claim 1. 3. The motion detecting apparatus according to claim 1, wherein said motion detecting means detects the head swinging motion and then performs detection by the next facial feature point detecting means at a predetermined time interval. apparatus. 4. The motion detecting apparatus according to claim 3, wherein said motion detecting means calculates a time required for one swing, and said time interval is a time required for swing detection. 5. The motion detection apparatus according to claim 1, wherein the motion detection means determines whether the head swing is vertical or horizontal based on the displacement direction of the feature point determined by the determination means. 6. The motion detecting apparatus according to claim 1, wherein the facial feature point detecting means determines a detection range of the next facial feature point detecting means by referring to a previous detection position of the facial feature point. 7. The motion detecting apparatus according to claim 1, wherein the facial feature point is a pupil.