JP5092776B2 - Gaze direction detection device and gaze direction detection method - Google Patents

Description

  The present invention relates to a gaze direction detection device and a gaze direction detection method for executing a calibration process for accurately detecting a gaze direction of a driver.

  In recent years, various systems (safe driving support systems) for supporting safe driving of vehicles have been proposed. In such a safe driving support system, for example, when a warning is given to the driver according to the situation outside the vehicle (existing situation of pedestrians, other vehicles, obstacles, etc.), the warning is given to the situation that the driver is aware of. When done, it becomes annoying. Therefore, a system that detects where the driver is looking (that is, the direction of the driver's line of sight) and issues a warning about a situation that the driver is not aware of is desired. In order to realize such a system, it is important to accurately detect (estimate) the driver's line-of-sight direction. For this purpose, it is necessary to perform a calibration process for detecting the line-of-sight direction.

For example, the gaze direction determination device described in Patent Document 1 uses a distance sensor (capacitance sensor or the like) built in the headrest and the seat, and calculates the head angle from the headrest sensor and the shoulder angle from the seat sensor. By detecting, the driver's gaze direction is estimated. In this device, the headrest is moved up and down while measuring the distance from the head when detecting that the driver is seated on the seat so that the gaze direction can be accurately detected regardless of the body shape of the driver. The headrest is stopped at the position where the distance is minimum. That is, the calibration process is executed by automatically setting the position of the headrest to the optimum position.
JP 2007-280352 A

  However, in the method described in Patent Document 1 described above, the driver is not always facing the front during the calibration process by the vertical movement of the headrest, and the driver is also facing the front. I can't figure out exactly. For this reason, there exists a problem that a calibration process is not performed appropriately.

  The present invention has been made in view of these problems, and an object of the present invention is to provide a gaze direction detection device and a gaze direction detection method capable of realizing appropriate calibration processing.

The gaze direction detecting device according to claims 1 , 4, 6 , and 11 of the present invention made to achieve the above object is used in a vehicle equipped with a photographing means for photographing a driver from the front. The calibration process for detecting the driver's line-of-sight direction based on the captured image is executed.

  In the gaze direction detection device, the gaze image display means displays a gaze image for causing the driver to gaze on the display means for displaying the image in front of the driver, and the captured image acquisition means performs the gaze image. A driver's photographed image photographed by the photographing means in a state where the gaze image is displayed by the display means is acquired. Further, the gaze direction detection means detects the driver's gaze direction based on the captured image acquired by the captured image acquisition means, and the calibration means detects the driver's gaze direction and the gaze direction detected by the gaze direction detection means. Based on the display position of the gaze image displayed by the image display means, the gaze direction detection calibration process by the gaze direction detection means is executed.

  According to such a gaze direction detection device, a driver's captured image in a desired gaze direction can be easily and accurately obtained by displaying a gaze image and causing the driver to gaze. Processing can be realized.

Here, the gaze image for gaze front to the driver, for example as described in claim 1, may be so as to display an image subjected to mirror image conversion to the driver of the captured image. In this way, since the driver turns to the front as if looking at the mirror, the driver can accurately grasp the state in which the driver has turned to the front.

In the line-of-sight direction detecting device according to any one of claims 2, 4, 7, and 12 , the gaze image display means sequentially displays the gaze image at a plurality of positions, and the captured image acquisition means comprises the gaze image display means. Thus, a driver's photographed image photographed by the photographing means in each state in which gaze images are sequentially displayed at a plurality of positions is acquired. The line-of-sight direction detection means detects the driver's line-of-sight direction based on the captured images in each state acquired by the captured image acquisition means, and the calibration means detects the driver's line-of-sight detected by the line-of-sight direction detection means. Based on the direction and the display position of the gaze image displayed by the gaze image display means in that state, the gaze direction detection calibration processing by the gaze direction detection means is executed.

  According to such a gaze direction detection device, it is possible to realize a more appropriate calibration process than when the calibration process is executed based on only one gaze direction.

In particular, as described in claims 3 and 4 , the display means is configured to be able to change the display distance of the image visually recognized by the driver, and the display distance is different at a plurality of positions for displaying the gaze image. Is included, it is possible to execute calibration processing in consideration of not only the vertical and horizontal directions but also the depth direction.

  By the way, when the driver does not look at the gaze image accurately by looking away at the time of the calibration process, it is conceivable that an inappropriate calibration process is executed.

Therefore, in the line-of-sight direction detection apparatus according to claims 5 and 6 , after the calibration processing is performed by the calibration unit, the confirmation image display unit displays the gaze image as the display position by the gaze image display unit. The images are displayed at different positions, and the confirmation image acquisition means acquires the driver's photographed image photographed by the photographing means in a state where the gaze image is displayed by the confirmation image display means. Then, the invalidation means detects the driver's direction detected by the gaze direction detection means based on the display position of the gaze image displayed by the confirmation image display means and the captured image acquired by the confirmation image acquisition means in that state. Based on the relationship with the line-of-sight direction, it is determined whether or not the calibration process by the calibration unit has been appropriately performed. If it is determined that the calibration process has not been performed properly, the calibration process is invalidated.

According to such a gaze direction detection device, it is possible to prevent an inappropriate calibration process from being executed.
Specifically, for example, in the gaze direction detecting device according to claim 8 , the invalid means is acquired by the confirmation image acquisition means in the display position of the gaze image displayed by the confirmation image display means and the state thereof. The driver's line-of-sight direction detected by the line-of-sight detection means based on the captured image is displayed on the display means in a state where it can be viewed at the same time, and whether or not the calibration process by the calibration means has been properly performed Let the person judge. According to such a gaze direction detection device, calibration processing can be executed according to the driver's intention.

On the other hand, in the gaze direction detecting device according to claim 9 , for example, the invalid means includes the display position of the gaze image displayed by the confirmation image display means, and the captured image acquired by the confirmation image acquisition means in that state. If the error from the driver's line-of-sight direction detected by the line-of-sight direction detection means satisfies a predetermined allowable condition, it is determined that the calibration process by the calibration means has been performed appropriately. According to such a gaze direction detecting device, it is possible to automatically determine whether or not the calibration process has been appropriately performed.

In the line-of-sight direction detection device according to claims 10 and 11 , the gaze image display means displays a grid for facilitating the perception of the position together with the gaze image. According to such a gaze direction detection device, the calibration process can be performed more accurately.

Next, the gaze direction detection method according to any one of claims 13 to 16 is used in a vehicle equipped with an imaging unit for imaging the driver from the front, and calibration for detecting the gaze direction of the driver based on a captured image by the imaging unit. This is for executing the process.

  In this gaze direction detection method, a gaze image display step for displaying a gaze image for causing the driver to gaze on a display means for displaying an image in front of the driver, and a gaze image display step. A captured image acquisition step for acquiring a driver's captured image captured by the imaging means in a state in which the image for display is displayed, and a line of sight for detecting the driver's line of sight based on the captured image acquired in the captured image acquisition step Calibration of gaze direction detection in the gaze direction detection step based on the direction detection step, the gaze direction of the driver detected in the gaze direction detection step, and the display position of the gaze image displayed in the gaze image display step A calibration step for executing processing.

The gaze direction detection methods according to claims 13 to 16 have the same features as those of claims 1, 4, 6, and 11, respectively. Therefore, according to the gaze direction detecting method according to claim 13 to 16, respectively claim 1, it is possible to obtain the same effect as the gaze direction detecting apparatus according to 4,6,11.

Embodiments to which the present invention is applied will be described below with reference to the drawings.
[1. overall structure]
FIG. 1 is a block diagram illustrating a schematic configuration of a detection ECU 10 according to the embodiment, and FIG. 2 is a schematic diagram illustrating an interior of a vehicle in which the detection ECU 10 is mounted.

  As shown in FIG. 1, the detection ECU 10 includes a camera interface (camera IF) 11, a face orientation detector 12, a line-of-sight direction detector 13, a head-up display interface (HUDIF) 14, a memory 15, and a controller. 16.

  The camera interface 11 is an interface for inputting a photographed image from the camera 20 (see FIG. 2) installed in the vehicle compartment. Here, the camera 20 is disposed at a position (for example, an instrument panel) where the driver of the vehicle is photographed from the front, and the driver's face can be photographed from the front.

  The face orientation detector 12 detects the driver's face orientation by analyzing the driver's photographed image by the camera 20 input via the camera interface 11. As the face orientation detection method, for example, “S. Baker et al.“ Lucas-Kanade 20 years on: A unifying framework: Part 4 ”Technical Report CMU-RI-TR-04-14, Robotics Institute, Carnegie Mellon University, The method disclosed in “February, 2004” can be used.

  The line-of-sight direction detector 13 detects the driver's line-of-sight direction by analyzing a photographed image of the driver by the camera 20 input via the camera interface 11. In addition, as a method of gaze direction detection, for example, “T.Ishikawa et al.“ Passive Driver Gaze Tracking With Active Appearance Model ”Technical Report CMU-RI-TR-04-08, Robotics Institute, Carnegie Mellon University, January, 2004.” Can be used.

  The head-up display interface 14 is an interface for outputting image information to be displayed on a head-up display (hereinafter simply referred to as “HUD”) 30 (see FIG. 2) installed in the vehicle compartment. Here, the HUD 30 is a display device that displays an image in front of the driver using the windshield (front glass) 1. Specifically, the image is visually recognized on a virtual image plane further forward than the windshield 1. Is done.

The memory 15 is a storage device for storing various information.
The controller 16 is mainly configured by a microcomputer including a CPU, a ROM, a RAM, and the like, and executes various processes. Further, the controller 16 receives operation information of the switch 40 installed in the vehicle compartment (in this embodiment, the steering wheel). Here, the switch 40 is an operation device for the driver to perform an operation with a finger, and includes at least an OK button and an NG button in the present embodiment.

[2. Processing executed by the controller]
The detection ECU 10 of the present embodiment executes a calibration process for detecting the driver's line-of-sight direction based on the driver's captured image by the camera 20. The calibration processing referred to in the present embodiment means that the algorithm for detecting the line-of-sight direction based on the captured image is corrected based on the actual line-of-sight direction and the captured image in that state, so that individual differences among the drivers can be obtained. This is a process for making it possible to detect the line-of-sight direction with high accuracy.

  Here, the main process executed by the controller 16 of the detection ECU 10 when the operation for starting the calibration process is performed will be described with reference to the flowchart of FIG. 3. Note that the calibration process is performed when the vehicle is stopped, and may be performed only when the vehicle is first boarded, or may be performed as many times as necessary.

When the main process is started, the controller 16 first acquires, as front information, an image captured by the camera 20 in a state where the driver is facing the front in S100.
Specifically, as shown in FIG. 4, the driver's face image being captured by the camera 20 is mirror-image converted and displayed on the virtual image plane ahead of the driver (displayed as a moving image in real time) by the HUD 30. Although not shown, a message (for example, “Look at the eye position”) for causing the driver to pay attention to the eye position in the display image is also displayed.

  Note that the message to the driver is not limited to being displayed as a character string on the virtual image plane ahead of the driver by the HUD 30, but may be transmitted by voice, for example, May be used in combination.

  Further, the driver's face image (mirror image) may be displayed as it is as shown in FIG. 5 (a). For example, as shown in FIG. 5 (b), each center point of both eyes (by image analysis). It is also possible to display the point of interest and the center of the eye) as the point of interest, and emphasize the point of interest by reducing the brightness of the other part (face image). It is also possible to display only the gazing point.

  Here, the driver's face image displayed by the HUD 30 is set so as to be displayed at a position that is substantially in front of the actual driver (position adjusted vertically and horizontally). If it sees, the image as if there is a mirror in the front will be displayed. In this manner, the driver gazes at the display image for a certain period of time, and the captured image (face image with the driver facing the front) input from the camera 20 during that time is the front information (calibration in the front direction). Information) (stored in the memory 15).

  Note that the processing of S100 as described above may proceed in order at a predetermined timing, or the driver may proceed at a desired timing by operating the switch 40.

  Subsequently, in S200, a face image in a state where the driver is facing a predetermined direction other than the front is also acquired as calibration information, and calibration processing is executed based on the acquired calibration information. Specifically, in this calibration process, as shown in FIG. 6A, the target T, which is a point for the driver to watch, is displayed on the virtual image plane in front of the driver by the HUD 30, and the target T A face image for each display position is acquired while changing the display position to a plurality of locations. In the example of FIG. 6A, the display position of the target T is changed to a plurality of locations at regular intervals on the circumference centered on the position directly in front, and a face image for each display position is acquired.

  The display position of the target T is not limited to the circumference. For example, it may be on a rectangle as shown in FIG. 6 (b), on a cross as shown in FIG. 6 (c), or on a grid as shown in FIG. 6 (d).

  Further, as shown in FIG. 6E, grid lines may be displayed so that the driver can easily perceive the position of the target T. In this case, the visibility of the target T can be made difficult to be impaired by the grid by setting the color and brightness of the grid line to be different from those of the target T.

Here, details of the calibration processing in S200 will be described with reference to the flowchart of FIG.
When the calibration process is started, first, in S201, the position for displaying the target T (the display position of the target T in S202 described later) is determined. In this calibration process, as will be described later, the processes of S201 to S204 are performed in accordance with predetermined display positions of the target T (in this embodiment, a plurality of positions on the circumference shown in FIG. 6A). As a result, the target T display process (S202) is finally performed for all display positions. Accordingly, in the process of S201, one of the display positions that have not been subjected to the display process is determined according to a predetermined rule (order).

Subsequently, in S202, the target T is displayed at the display position determined in S201, and a message for causing the driver to watch the target T is displayed.
Subsequently, in S203, an image photographed by the camera 20 while the driver is gazing at the target T is acquired (saved in the memory 15) as calibration information.

  Subsequently, in S204, it is determined whether the necessary number of pieces of calibration information has been acquired. That is, it is determined whether display processing of the target T has been performed for all display positions.

If it is determined in S204 that the necessary number of pieces of calibration information has not been acquired, the process returns to S201, and the process proceeds to an unprocessed display position.
On the other hand, if it is determined in S204 that the required number of pieces of calibration information has been acquired, the process proceeds to S205, where each display position is displayed based on each piece of calibration information (face image) stored in the memory 15. The driver's face orientation is calculated. Further, in S206, the gaze direction at each display position is detected based on each calibration information and the face orientation calculated in S205, and from the face image based on the relationship between the detected gaze direction and the display position of the target T. A gaze detection model for detecting the gaze direction is generated (corrected). In other words, the gaze direction detection calibration is executed. Thereafter, the calibration process is terminated, and the process proceeds to S300 in FIG.

In S300, a detection confirmation process for confirming that the line-of-sight detection model generated in the calibration process of S200 is appropriate is executed.
Here, details of the detection confirmation processing in S300 will be described with reference to the flowchart of FIG. Note that S310 is the same as the process shown in FIG.

  When the detection confirmation process is started, first, in S301, the position where the target T is displayed is determined as in S201 described above. However, in S301, a display position (a plurality of predetermined display positions) different from the display position used in S201 is used.

  Subsequently, in S302, similarly to S202 described above, the target T is displayed at the display position determined in S301, and a message for causing the driver to watch the target T is displayed.

  Subsequently, in S303, the captured image input from the camera 20 while the driver is gazing at the target T and the display position of the target T at that time are associated with each other (stored in the memory 15).

Subsequently, in S304, the driver's face orientation is calculated based on the face image acquired in S303 using the line-of-sight detection model generated in the calibration process.
Subsequently, in S305, the gaze direction of the driver is calculated based on the face image acquired in S303 and the face orientation calculated in S304 using the gaze detection model generated in the calibration process.

Subsequently, in S306, the calculation results in S304 and S305 are stored in the memory 15.
Subsequently, in S307, it is determined whether the required number of face images has been acquired. That is, it is determined whether display processing of the target T has been performed for all display positions.

If it is determined in S307 that the required number of face images has not been acquired, the process returns to S301, and the process proceeds to an unprocessed display position.
On the other hand, if it is determined in S307 that the required number of face images has been acquired, the process proceeds to S308, where the target T is displayed at all display positions where the target T is displayed in the main detection confirmation process, and for each target T. A calculated point corresponding to the calculated line-of-sight direction is displayed. That is, the target T at each display position and the calculated points corresponding to the line-of-sight direction calculated based on the captured image gazing at the target T are displayed simultaneously. Therefore, the closer the distance between the display position of the target T and the position of the calculation point is, the more appropriate the gaze detection model generated by the calibration process is.

  Subsequently, in S309, a message for causing the driver to determine whether there is no problem with the displayed result (for example, “Is this result OK?”) Is displayed on the virtual image plane ahead of the driver by the HUD 30.

Subsequently, returning to FIG. 3, in S <b> 310, a determination result by the driver as to whether there is no problem with the displayed result is determined based on whether the OK button of the switch 40 is pressed.
If it is determined in S310 that the OK button of the switch 40 has not been pressed (the NG button has been pressed), the process returns to S200 and starts again from the calibration process.

On the other hand, if it is determined in S310 that the OK button of the switch 40 has been pressed, the main process is terminated.
[3. effect]
As described above, the detection ECU 10 of the present embodiment causes the target T to be watched by the driver to be displayed on the virtual image plane ahead of the driver by the HUD 30 (S202), and the captured image acquired from the camera 20 in that state. The direction of the driver's line of sight is detected based on (S203, S205). Then, calibration processing is executed based on the detected line-of-sight direction and the display position of the target T (S206).

  According to such an ECU 10 for detection, the target T is displayed in the front and the driver gazes, so that the driver's photographed image in the desired line-of-sight direction (the photographed image in the driver's original driving posture). Can be obtained easily and accurately, and appropriate calibration processing can be realized.

  Further, the detection ECU 10 displays an image obtained by performing mirror image conversion on the driver's photographed image as a target T for causing the driver to gaze at the front. For this reason, the driver turns to the front as if looking at the mirror, and can accurately grasp the state in which the driver himself faces the front.

  Further, the detection ECU 10 sequentially displays the target T at a plurality of positions (S201), and obtains a driver's photographed image from the camera 20 in each state where the target T is sequentially displayed at the plurality of positions. And a driver | operator's gaze direction is detected based on the picked-up image in each state, and a calibration process is performed based on the detected gaze direction and the display position of the target T in the state. For this reason, more appropriate calibration processing can be realized as compared with the case where the calibration processing is executed based on only one line-of-sight direction.

  Further, after the calibration process is executed, the detection ECU 10 displays the target T at a display position different from the previous time (S301, S302), and acquires a photographed image of the driver from the camera 20 in that state ( S303). Then, based on the relationship between the display position of the target T and the driver's line-of-sight direction detected based on the captured image acquired in that state, it is determined whether or not the calibration process has been appropriately performed. If it is determined that it has not been performed, the calibration process is invalidated (S308 to S310). For this reason, it is possible to prevent an inappropriate calibration process from being executed.

  In particular, the display position of the target T and the calculation point corresponding to the line-of-sight direction in that state are displayed at the same time so that the driver can determine whether the calibration process has been performed appropriately. Therefore, the calibration process can be executed according to the driver's intention.

[4. Correspondence with Claims]
In the detection ECU 10 of the present embodiment, the camera 20 corresponds to a photographing unit, and the HUD 30 corresponds to a display unit. Further, the controller 16 that executes the processes of S100, S201, and S202 corresponds to the gaze image display means, and the controller 16 that executes the processes of S100 and S203 corresponds to the captured image acquisition means. Further, the controller 16 that executes the processes of S205 and S206 corresponds to the line-of-sight direction detecting means and the calibration means. The controller 16 that executes the processes of S301 and S302 corresponds to a confirmation image display unit, and the controller 16 that executes the process of S303 corresponds to a confirmation image acquisition unit and executes the processes of S308 to S310. The controller 16 corresponds to invalid means.

[5. Other forms]
As mentioned above, although embodiment of this invention was described, it cannot be overemphasized that this invention can take a various form, without being limited to the said embodiment.

  For example, in S301 of the detection confirmation process (FIG. 8) described above, a display position different from the display position used in S201 of the calibration process (FIG. 7) is used as the display position of the target T to be determined. However, the display position is not limited to this, and the same display position as that used in S201 may be used.

  In the detection ECU 10 of the above embodiment, the display position of the target T and the calculation point corresponding to the line-of-sight direction in that state are displayed in a state where they can be viewed simultaneously (S308), and the calibration process is appropriately performed. However, the present invention is not limited to this, and the detection ECU 10 may determine automatically. Specifically, for example, the line of sight generated by the calibration process when the distance between the display position of the target T and the position of the calculation point (all distances when there are a plurality of distances) is equal to or less than a predetermined determination threshold value It can be determined that the detection model is appropriate. In such a configuration, the switch 40 may be omitted.

  On the other hand, for example, as shown in FIG. 9, a stereo head-up display that can change the display distance (position in the depth direction of the virtual image plane) of the image visually recognized by the driver may be used. In this way, the display position of the target T can be changed not only in the vertical and horizontal directions but also in the depth direction (front and rear direction), so that the calibration process can be executed in consideration of the depth direction. In particular, in such a configuration, as described above, a grid line is displayed to make it easier for the driver to grasp the position of the target T (FIG. 6E), and further in the depth direction of the target T. If the grid width is changed according to the position, the position of the target T in the depth direction can be easily perceived.

It is a block diagram which shows schematic structure of ECU for detection of embodiment. It is a schematic diagram which shows the interior of the vehicle by which ECU for detection is mounted. It is a flowchart of a main process. It is a schematic diagram which shows the vehicle interior in the state which displayed the driver | operator's face image. It is explanatory drawing which shows the example of a display of a face image. It is explanatory drawing which shows the example of a display of a target. It is a flowchart of a calibration process. It is a flowchart of a detection confirmation process. It is a mimetic diagram showing the interior of a vehicle equipped with a stereo head-up display.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Wind shield, 10 ... ECU for detection, 11 ... Camera interface, 12 ... Face direction detector, 13 ... Gaze direction detector, 14 ... Head-up display interface, 15 ... Memory, 16 ... Controller, 20 ... Camera, 30 ... head-up display, 40 ... switch, T ... target

Claims (16)

A gaze direction detection device that is used in a vehicle equipped with an imaging unit that images a driver from the front, and that performs calibration processing for detecting the driver's gaze direction based on an image captured by the imaging unit,
Display means for displaying an image in front of the driver;
Gaze image display means for causing the display means to display a gaze image for gaze the driver;
Captured image acquisition means for acquiring a driver's captured image captured by the imaging means in a state where the gaze image is displayed by the gaze image display means;
Gaze direction detection means for detecting the driver's gaze direction based on the captured image acquired by the captured image acquisition means;
Based on the driver's gaze direction detected by the gaze direction detection means and the display position of the gaze image displayed by the gaze image display means, the gaze direction detection calibration processing by the gaze direction detection means is performed. Calibration means to perform,
Equipped with a,
The gaze image display means, as gaze image for gaze front driver, gaze direction detecting apparatus according to claim Rukoto to display the image subjected to mirror image conversion to the driver of the captured image. The gaze image display means sequentially displays the gaze image at a plurality of positions,
The captured image acquisition unit acquires a driver's captured image captured by the imaging unit in each state where the gaze image is sequentially displayed at the plurality of positions by the gaze image display unit,
The line-of-sight direction detection means detects the driver's line-of-sight direction based on the captured image in each state acquired by the captured image acquisition means,
The calibration means is based on the gaze direction of the driver detected by the gaze direction detection means and the display position of the gaze image displayed by the gaze image display means in that state. The gaze direction detection device according to claim 1, wherein calibration processing for gaze direction detection is performed. The display means is configured to change a display distance of an image visually recognized by the driver,
The gaze direction detection device according to claim 2 , wherein the plurality of positions include positions having different display distances. A gaze direction detection device that is used in a vehicle equipped with an imaging unit that images a driver from the front, and that performs calibration processing for detecting the driver's gaze direction based on an image captured by the imaging unit,
Display means for displaying an image in front of the driver;
Gaze image display means for causing the display means to display a gaze image for gaze the driver;
Captured image acquisition means for acquiring a driver's captured image captured by the imaging means in a state where the gaze image is displayed by the gaze image display means;
Gaze direction detection means for detecting the driver's gaze direction based on the captured image acquired by the captured image acquisition means;
Based on the driver's gaze direction detected by the gaze direction detection means and the display position of the gaze image displayed by the gaze image display means, the gaze direction detection calibration processing by the gaze direction detection means is performed. Calibration means to perform,
Equipped with a,
The gaze image display means sequentially displays the gaze image at a plurality of positions,
The captured image acquisition unit acquires a driver's captured image captured by the imaging unit in each state where the gaze image is sequentially displayed at the plurality of positions by the gaze image display unit,
The line-of-sight direction detection means detects the driver's line-of-sight direction based on the captured image in each state acquired by the captured image acquisition means,
The calibration means is based on the gaze direction of the driver detected by the gaze direction detection means and the display position of the gaze image displayed by the gaze image display means in that state. Execute gaze direction detection calibration process by
The display means is configured to change a display distance of an image visually recognized by the driver,
Wherein the plurality of the locations, gaze direction detecting apparatus characterized by different positions contains Rukoto of the display distance. After the calibration process is executed by the calibration means, a confirmation image display means for displaying the gaze image at a position different from the display position by the gaze image display means,
Confirmation image acquisition means for acquiring a driver's photographed image photographed by the photographing means in a state where the gaze image is displayed by the confirmation image display means;
The driver's gaze direction detected by the gaze direction detection means based on the display position of the gaze image displayed by the confirmation image display means and the captured image acquired by the confirmation image acquisition means in that state On the basis of the relationship between the calibration means and the invalidation means for invalidating the calibration process when it is determined that the calibration process is not properly performed,
The line-of-sight direction detection device according to claim 1, comprising: A gaze direction detection device that is used in a vehicle equipped with an imaging unit that images a driver from the front, and that performs calibration processing for detecting the driver's gaze direction based on an image captured by the imaging unit,
Display means for displaying an image in front of the driver;
Gaze image display means for causing the display means to display a gaze image for gaze the driver;
Captured image acquisition means for acquiring a driver's captured image captured by the imaging means in a state where the gaze image is displayed by the gaze image display means;
Gaze direction detection means for detecting the driver's gaze direction based on the captured image acquired by the captured image acquisition means;
Based on the driver's gaze direction detected by the gaze direction detection means and the display position of the gaze image displayed by the gaze image display means, the gaze direction detection calibration processing by the gaze direction detection means is performed. Calibration means to perform,
After the calibration process is executed by the calibration means, a confirmation image display means for displaying the gaze image at a position different from the display position by the gaze image display means,
Confirmation image acquisition means for acquiring a driver's photographed image photographed by the photographing means in a state where the gaze image is displayed by the confirmation image display means;
The driver's gaze direction detected by the gaze direction detection means based on the display position of the gaze image displayed by the confirmation image display means and the captured image acquired by the confirmation image acquisition means in that state On the basis of the relationship between the calibration means and the invalidation means for invalidating the calibration process when it is determined that the calibration process is not properly performed,
A line-of-sight direction detecting device comprising: The gaze image display means sequentially displays the gaze image at a plurality of positions,
The captured image acquisition unit acquires a driver's captured image captured by the imaging unit in each state where the gaze image is sequentially displayed at the plurality of positions by the gaze image display unit,
The line-of-sight direction detection means detects the driver's line-of-sight direction based on the captured image in each state acquired by the captured image acquisition means,
The calibration means is based on the gaze direction of the driver detected by the gaze direction detection means and the display position of the gaze image displayed by the gaze image display means in that state. The gaze direction detection apparatus according to claim 6 , wherein a gaze direction detection calibration process is performed by using the gaze direction detection method. The invalidation means is detected by the gaze direction detection means based on the display position of the gaze image displayed by the confirmation image display means and the captured image acquired by the confirmation image acquisition means in that state. The driver's line-of-sight direction is displayed on the display unit in a state in which the driver can view simultaneously, and the driver is allowed to determine whether or not the calibration process by the calibration unit has been appropriately performed. The gaze direction detecting device according to any one of claims 5 to 7 . The invalidation means is detected by the gaze direction detection means based on the display position of the gaze image displayed by the confirmation image display means and the captured image acquired by the confirmation image acquisition means in that state. any error between the viewing direction of the driver, when the predetermined allowable condition is satisfied, the claims 5 calibration process by the calibration means and judging to have been properly performed until claim 7 or gaze direction detecting apparatus according to item 1. The gaze direction detection according to any one of claims 1 to 9 , wherein the gaze image display means displays a grid for easily perceiving the position together with the gaze image. apparatus. A gaze direction detection device that is used in a vehicle equipped with an imaging unit that images a driver from the front, and that performs calibration processing for detecting the driver's gaze direction based on an image captured by the imaging unit,
Display means for displaying an image in front of the driver;
Gaze image display means for causing the display means to display a gaze image for gaze the driver;
Captured image acquisition means for acquiring a driver's captured image captured by the imaging means in a state where the gaze image is displayed by the gaze image display means;
Gaze direction detection means for detecting the driver's gaze direction based on the captured image acquired by the captured image acquisition means;
Based on the driver's gaze direction detected by the gaze direction detection means and the display position of the gaze image displayed by the gaze image display means, the gaze direction detection calibration processing by the gaze direction detection means is performed. Calibration means to perform,
Equipped with a,
The gaze image display means, line-of-sight direction detecting apparatus according to claim Rukoto display the grid for easy perceive the position together with the gaze image. The gaze image display means sequentially displays the gaze image at a plurality of positions,
The captured image acquisition unit acquires a driver's captured image captured by the imaging unit in each state where the gaze image is sequentially displayed at the plurality of positions by the gaze image display unit,
The line-of-sight direction detection means detects the driver's line-of-sight direction based on the captured image in each state acquired by the captured image acquisition means,
The calibration means is based on the gaze direction of the driver detected by the gaze direction detection means and the display position of the gaze image displayed by the gaze image display means in that state. The gaze direction detection apparatus according to claim 11 , wherein a gaze direction detection calibration process is performed. A gaze direction detection method that is used in a vehicle equipped with a photographing unit that photographs a driver from the front, and that performs calibration processing for detecting a gaze direction of a driver based on a captured image by the photographing unit,
A gaze image display step for displaying a gaze image for gaze the driver on the display means for displaying an image in front of the driver;
A captured image acquisition step of acquiring a driver's captured image captured by the imaging means in a state where the gaze image is displayed in the gaze image display step;
A line-of-sight direction detecting step for detecting a driver's line-of-sight direction based on the captured image acquired in the captured image acquisition step;
Calibration processing for gaze direction detection in the gaze direction detection step based on the gaze direction of the driver detected in the gaze direction detection step and the display position of the gaze image displayed in the gaze image display step A calibration step to perform
Equipped with a,
Wherein the gaze image display step, the gaze image for gaze front driver, gaze direction detecting method comprising Rukoto to display the image subjected to mirror image conversion to the driver of the captured image. A gaze direction detection method that is used in a vehicle equipped with a photographing unit that photographs a driver from the front, and that performs calibration processing for detecting a gaze direction of a driver based on a captured image by the photographing unit,
A gaze image display step for displaying a gaze image for gaze the driver on the display means for displaying an image in front of the driver;
A captured image acquisition step of acquiring a driver's captured image captured by the imaging means in a state where the gaze image is displayed in the gaze image display step;
A line-of-sight direction detecting step for detecting a driver's line-of-sight direction based on the captured image acquired in the captured image acquisition step;
Calibration processing for gaze direction detection in the gaze direction detection step based on the gaze direction of the driver detected in the gaze direction detection step and the display position of the gaze image displayed in the gaze image display step A calibration step to perform
Equipped with a,
In the gaze image display step, the gaze image is sequentially displayed at a plurality of positions,
In the captured image acquisition step, acquire a driver's captured image captured by the imaging unit in each state where the gaze image is sequentially displayed at the plurality of positions in the gaze image display step,
In the line-of-sight direction detection step, the line-of-sight direction of the driver is detected based on the captured image in each state acquired in the captured image acquisition step,
In the calibration step, the gaze direction detection step based on the driver's gaze direction detected in the gaze direction detection step and the display position of the gaze image displayed in the gaze image display step in that state. Execute gaze direction detection calibration process at
The display means is configured to change a display distance of an image visually recognized by the driver,
Wherein the plurality of positions, gaze direction detecting method characterized that you have contains different positions the display distance. A gaze direction detection method that is used in a vehicle equipped with a photographing unit that photographs a driver from the front, and that performs calibration processing for detecting a gaze direction of a driver based on a captured image by the photographing unit,
A gaze image display step for displaying a gaze image for gaze the driver on the display means for displaying an image in front of the driver;
A captured image acquisition step of acquiring a driver's captured image captured by the imaging means in a state where the gaze image is displayed in the gaze image display step;
A line-of-sight direction detecting step for detecting a driver's line-of-sight direction based on the captured image acquired in the captured image acquisition step;
Calibration processing for gaze direction detection in the gaze direction detection step based on the gaze direction of the driver detected in the gaze direction detection step and the display position of the gaze image displayed in the gaze image display step A calibration step to perform
After the calibration process is executed in the calibration step, a confirmation image display step for displaying the gaze image at a position different from the display position in the gaze image display step;
A confirmation image acquisition step of acquiring a driver's photographed image photographed by the photographing means in a state in which the gaze image is displayed in the confirmation image display step;
The gaze direction of the driver detected in the gaze direction detection step based on the display position of the gaze image displayed in the confirmation image display step and the captured image obtained in the confirmation image acquisition step in that state On the basis of the relationship between the invalidation step and the invalidation step for invalidating the calibration process when it is determined that the calibration process in the calibration step is appropriately performed. ,
A gaze direction detection method comprising: A gaze direction detection method that is used in a vehicle equipped with a photographing unit that photographs a driver from the front, and that performs calibration processing for detecting a gaze direction of a driver based on a captured image by the photographing unit,
A gaze image display step for displaying a gaze image for gaze the driver on the display means for displaying an image in front of the driver;
A captured image acquisition step of acquiring a driver's captured image captured by the imaging means in a state where the gaze image is displayed in the gaze image display step;
A line-of-sight direction detecting step for detecting a driver's line-of-sight direction based on the captured image acquired in the captured image acquisition step;
Calibration processing for gaze direction detection in the gaze direction detection step based on the gaze direction of the driver detected in the gaze direction detection step and the display position of the gaze image displayed in the gaze image display step A calibration step to perform
Equipped with a,
Wherein the gaze image display step, the line-of-sight direction detecting method comprising Rukoto display the grid for easy perceive the position together with the gaze image.