JP7145792B2 - Oversight detection device, oversight detection method, and program - Google Patents

Description

The present invention relates to an oversight detection device, an oversight detection method, and a program.

Conventionally, various driving assistance systems for assisting drivers have been researched and developed. Patent Literature 1 discloses an invention related to visual recognition support for reliably detecting a visually recognized object to be visually recognized by a driver. In the invention described in this document, a visible object to be viewed by the driver is determined based on an image showing the surrounding situation acquired by the camera. Then, a visual target detected by the eye camera that is not in the line of sight of the driver is determined as a visual target overlooked by the driver, and the driver's line of sight is guided in the direction of the overlooked visual target. do.

JP 2017-111649 A

In the conventional technology described above, an oversight is determined when the driver is not looking at the visual target to be visually recognized. There are times when The inventors of the present invention considered that one of the causes of such an oversight is that there is a situation in which it is difficult to perceive a visually recognized object, and thus came up with the present invention. SUMMARY OF THE INVENTION An object of the present invention is to provide an oversight detection device that appropriately detects a driver's oversight.

The oversight detection device of the present invention includes a camera that captures the image of the driver, a line-of-sight detection unit that detects the line of sight of the driver based on the image captured by the camera, and a visible light camera that captures the surroundings of the vehicle. an effective area size setting unit for setting the size of an effective area related to object recognition based on the degree of certainty when an object is recognized from the captured image; a determination unit that sets an effective area based on the size of the area and determines whether or not the driver recognizes the object based on whether the object is in the effective area. Prepare.

In the present invention, the size of the effective area for object recognition is set based on the certainty of object recognition based on the image of the visible light camera. The accuracy of object recognition based on images taken with a visible light camera is close to the accuracy with which humans can recognize objects. By determining whether the driver recognizes the object or has overlooked it, it can be appropriately detected.

The oversight detection device of the present invention includes a camera that captures the image of the driver, a line-of-sight detection unit that detects the line of sight of the driver based on the image captured by the camera, and object recognition based on the external environment of the vehicle. an effective area size setting unit for setting the size of the effective area for the line of sight and the effective area size set by the effective area size setting unit; and a determination unit for determining whether or not the driver recognizes the object based on whether or not the object detected by the surroundings monitoring sensor is present.

By setting the size of the effective area related to object recognition based on the vehicle's external environment in this way, the effective area can be set to reflect the easiness of recognizing an object that changes due to the external environment. can be properly detected.

In the oversight detection device of the present invention, the effective area size setting unit includes current time, illuminance detected by an illuminance sensor, vehicle speed, headlight lighting state, fog lamp lighting state, wiper operating state, and travel position information. and the size of the effective area may be set based on the estimated external environment.

With this configuration, it is possible to determine whether it is daytime or nighttime based on the current time. For example, during the daytime it is easy to recognize an object, so the effective area can be set large, and at nighttime it is difficult to recognize an object, so the effective area can be set small. In addition, by using the illuminance detected by the illuminance sensor, the external brightness condition can be reflected in the size of the effective area. Furthermore, the lighting status of headlights and fog lamps, the operating status of wipers, and the speed of the vehicle can be considered to be operations performed by the driver in response to the external environment, so the external environment can be estimated from these conditions. . Furthermore, it is also possible to estimate the external environment from a map based on the travel position information.

In the oversight detection device of the present invention, the determination unit determines that the driver recognizes the object when the time during which the object is in the effective area is equal to or longer than a predetermined threshold, and determines that the driver recognizes the object. It may be determined that the object is overlooked when the value is less than the threshold value of .

Even if an object is in the effective area of the driver's line of sight, if the time is too short, the object cannot be recognized. According to the present invention, when the time for which the object is in the effective area is equal to or longer than the predetermined threshold, it is determined that the object is recognized. can.

In the oversight detection device of the present invention, the effective area size setting section may set the size of the effective area based on the age of the driver.

Since the effective area in which an object can be recognized becomes narrower with age, by setting the size of the effective area based on the age of the driver, it is possible to perform age-appropriate oversight detection.

In the oversight detection device of the present invention, the effective area size setting section may set the effective area smaller when the object is stationary than when the object is moving.

A stationary object is more difficult to perceive than a moving object, but according to the present invention, by setting a small effective area for a stationary object, it is possible to maintain a high level of detection accuracy for overlooked objects. .

In the oversight detection device of the present invention, the effective area size setting unit may set the effective area smaller as the number of objects around the vehicle increases.

If there are many objects around the vehicle, it becomes difficult to recognize the objects. However, according to the present invention, the more objects around the vehicle, the smaller the effective area is set, thereby maintaining a high detection accuracy of overlooked objects.

An oversight detection method of the present invention is a method for detecting an oversight by a driver of an object around a vehicle, comprising the steps of: acquiring an image captured by a camera that captures the driver; a step of detecting the line of sight of a person; and a step of setting the size of an effective area for recognizing an object based on the degree of certainty when the object is recognized from an image captured by a visible light camera that captures the surroundings of the vehicle. setting an effective area based on the line of sight and the size of the set effective area, and determining whether or not the driver recognizes the object based on whether the object is within the effective area; and a step of determining whether

An oversight detection method of the present invention is a method for detecting an oversight by a driver of an object around a vehicle, comprising the steps of: acquiring an image captured by a camera that captures the driver; a step of detecting a line of sight of a person; a step of setting a size of an effective area for object recognition based on the external environment of the vehicle; and a step of setting an effective area based on the line of sight and the size of the set effective area. and determining whether or not the driver recognizes the object based on whether or not the object detected by the surroundings monitoring sensor is included in the effective area.

A program of the present invention is a program for detecting an oversight by a driver of an object around a vehicle, comprising the step of obtaining, in a computer, an image captured by a camera that captures the driver; and setting the size of the effective area for object recognition based on the degree of confidence when the object is recognized from the image captured by the visible light camera that captures the surroundings of the vehicle. and setting an effective area based on the line of sight and the size of the set effective area, and allowing the driver to recognize the object based on whether the object is within the effective area. and a step of determining whether or not there is.

A program of the present invention is a program for detecting an oversight by a driver of an object around a vehicle, comprising the step of obtaining, in a computer, an image captured by a camera that captures the driver; setting the size of an effective area for object recognition based on the external environment of the vehicle; and determining the size of the effective area based on the line of sight and the size of the set effective area. and determining whether or not the driver recognizes the object based on whether or not the object detected by the surrounding monitoring sensor is in the effective area. .

ADVANTAGE OF THE INVENTION According to this invention, the ease of recognition of an object can be considered, and a driver's oversight can be detected appropriately.

1 is a diagram showing a configuration of an oversight detection device according to a first embodiment; FIG. 4 is a flow chart showing the operation of the oversight detection device of the first embodiment; FIG. 10 is a diagram showing the configuration of an oversight detection device according to a second embodiment; FIG. 2 is a diagram showing devices that detect an external environment and information on environments that can be detected by each device; 9 is a flowchart showing the operation of the oversight detection device of the second embodiment;

An oversight detection device according to an embodiment of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram showing the configuration of an oversight detection device 1 according to the first embodiment. The oversight detection apparatus 1 of the first embodiment includes a communication unit 10, and communicates with a millimeter wave radar 20 for detecting objects around the vehicle and a visible light camera 21 for capturing an image around the vehicle. . The millimeter wave radar 20 is less susceptible to rain, fog, and the like, and can detect objects even at night. The visible light camera 21 has a wide field of view and can recognize pedestrians and preceding persons. The millimeter wave radar 20 and the visible light camera 21 are peripheral monitoring sensors that sense the situation around the vehicle.

In this embodiment, an example of using a millimeter wave radar to detect objects around the vehicle was given, but as a method of detecting other vehicles and pedestrians around the vehicle, A method of acquiring information on a certain object may be adopted. V2X is a general term for communication between vehicles and things, such as vehicles and networks (V2N), vehicles and vehicles (vehicle-to-vehicle communication: V2V), vehicles and infrastructure (vehicle-to-infrastructure communication: V2I), and vehicles and smartphones. including pedestrians (V2P) with

The oversight detection device 1 includes an in-vehicle camera 12 for capturing an image of the driver, and a line-of-sight detection unit 13 for detecting the driver's line of sight based on the image captured by the in-vehicle camera 12 . The in-vehicle camera 12 is a camera that irradiates the face of the driver with near-infrared rays and takes an image. Since near-infrared rays are used, the driver can be photographed even at night. The line-of-sight detection unit 13 detects the direction in which the line of sight of the driver is directed based on the movement of the driver's face and eyeballs.

The overlook detection device 1 also includes an effective area size setting unit 11 that sets the size of an effective area for object recognition. Here, the "effective area" will be explained. Humans can perceive objects not only in the direction in which the human line of sight matches the direction in which the object to be recognized is located, but also when the object to be recognized is in an area with a certain width with respect to the direction of the line of sight. For example, it is possible to perceive the object. In this specification, the effective area is the area related to the recognition of such an object.

The size of the effective area is not always constant, and varies depending on, for example, the relationship between the color, brightness, etc. of the object and the color, brightness, etc. of the background. For example, when the surroundings are bright, such as during the daytime on a sunny day, it is easy to recognize objects. In such situations, it is possible to perceive objects with peripheral vision, and the effective area is large. On the other hand, in the evening when the surroundings are dim or in the rain, it is difficult to perceive objects. Although it is possible to perceive objects by looking directly at them, it is difficult to perceive objects with peripheral vision, so the effective area is small.

The effective area size setting unit 11 has a function of setting the size of the effective area based on the degree of certainty when object recognition is performed based on the image captured by the visible light camera 21 . “Confidence” is the degree of certainty of the result of object recognition when object recognition is performed. Confidence may be calculated using, for example, the contrast ratio, brightness difference or color difference between an object such as a pedestrian and its background. The effective area can be set in three stages, for example, a range of 20 degrees around the line-of-sight direction, a range of 3 degrees around, and 0. 20 degrees is the state in which the object can be recognized with peripheral vision, 3 degrees is the state in which the object can be recognized by looking directly at it, and 0 is the state in which the object cannot be seen.

The effective area size setting unit 11 sets the effective area to 20 degrees when the certainty of object recognition based on the visible light image is equal to or higher than a predetermined threshold, and sets the effective area to 3 degrees when the certainty is less than the certain threshold. If the object cannot be recognized, the effective area is set to 0. It should be noted that the fact that the object recognition could not be performed is because the millimeter wave radar 20 knows that the object exists. In other words, even though the object is detected by the millimeter wave radar 20, if the object cannot be recognized from the image of the visible light camera 21, it is not that the object does not exist, but that the object can be recognized from the visible light image. It can be determined that there was no

In this embodiment, since the size of the effective area is determined by the certainty of object recognition, the size of the effective area may differ for each object. For example, even at night, the active area is large for shiny objects and small or zero for black objects.

The determination unit 14 sets an effective area based on the line of sight, and determines whether the driver recognizes or overlooks the object based on whether the object is within the set effective area. do. The effective area is an area of a size set by the effective area size setting unit 11 (for example, a range of 20 degrees or a range of 3 degrees) centering on the line of sight of the driver.

A determination part 14 determines that the object is recognized when the object is in the effective area for a predetermined time or more, and determines that the object is not recognized, that is, the object is overlooked when the predetermined time is not reached. . The reason why the condition is that the object is in the effective area for a predetermined time or longer is that it takes a certain amount of time for humans to recognize the object.

The output unit 15 has a function of outputting the result of the omission determination by the determination unit 14 to other equipment such as an alarm device. As a result, other equipment such as an alarm device can perform control according to the oversight. For example, an alarm device can alert the driver. Note that the output unit 15 may output only when it is determined that there is an oversight, and may not output when there is no oversight.

The configuration of the oversight detection device 1 according to the present embodiment has been described above. An example of the hardware of the oversight detection device 1 is a computer equipped with a CPU, RAM, ROM, hard disk, communication interface, and the like. . The above-described oversight detection device 1 is realized by storing a program having modules for realizing each of the functions described above in the RAM or ROM and executing the program by the CPU. Such programs are also included in the scope of the present invention.

FIG. 2 is a flow chart showing the operation of the oversight detection device 1 of the first embodiment. The oversight detection device 1 detects an object around the vehicle using the millimeter wave radar 20 (S10). In addition, the oversight detection device 1 captures an image of the vehicle surroundings with the visible light camera 21 (S11), and performs object recognition using the captured image (S12). At this time, the oversight detection device 1 calculates the reliability of object recognition (S12). Subsequently, the overlook detection device 1 sets the size of the effective area based on the certainty of object recognition (S13). When the certainty of object recognition is equal to or higher than a predetermined threshold, the size of the effective area is set to the range of 20 degrees, and when the certainty of object recognition is less than the predetermined threshold, the size of the effective area is set to the range of 3 degrees. , and the effective area is set to 0 when the object cannot be recognized. The certainty of object recognition is obtained for each object around the vehicle, and the size of the effective area is determined for each object.

The oversight detection device 1 detects the line of sight of the driver from the image captured by the in-vehicle camera 12 (S14). Subsequently, the overlook detection device 1 determines whether or not the driver has overlooked the object detected by the millimeter wave radar 20, that is, whether or not the driver recognizes the object (S15). Specifically, the oversight detection device 1 determines whether or not an object has been within the range of the effective area centered on the line-of-sight direction of the driver for a predetermined period of time. The size of the effective area used here varies depending on the object to be determined.

When the oversight detection device 1 determines that there is an oversight, for example, it outputs information to that effect to an alarm device (S16), and alerts the driver by issuing an alarm.

The configuration and operation of the oversight detection device 1 according to the first embodiment have been described above. The oversight detection device 1 of the first embodiment determines oversight using the effective area, not whether the line of sight is looking directly at the direction of the object, so it can appropriately detect oversight by the driver. . Also, since the effective area is set for each object based on the certainty of object recognition from the image of the visible light camera 21, the size can be set according to the ease of finding each object.

In this embodiment, an example in which data from the millimeter wave radar 20 is used is given, but a configuration in which the data from the millimeter wave radar 20 is not used is also possible. In this case, it is determined whether or not the driver recognizes the object recognized from the image captured by the visible light camera 21 .

(Second embodiment)
FIG. 3 is a diagram showing the configuration of an oversight detection device 2 according to the second embodiment. Although the oversight detection device 1 of the first embodiment sets the effective area based on the certainty of object recognition from the image of the visible light camera 21, the oversight detection device 2 of the second embodiment Set the size of the active area based on the external environment.

In the oversight detection device 2 of the second embodiment, the communication unit 10 communicates with various sensors 22 and vehicle equipment 23 that acquire information about the external environment. The effective area size setting unit 11 estimates the external environment of the vehicle based on information from various sensors 22 and vehicle equipment 23, and sets the size of the effective area for object recognition based on the external environment. Here, the "external environment" is an environment related to the recognition of objects by the driver, such as good visibility, traffic conditions, and the road on which the vehicle is traveling.

FIG. 4 is a diagram showing devices that detect the external environment and information on the environment that can be detected by each device. Devices for obtaining information about visibility include, for example, clocks, illuminance sensors, headlights, fog lights, windshield wipers, GPS and map databases. Based on the information from the clock, it is possible to distinguish between daytime, evening, and nighttime, and to estimate the brightness of the surroundings. The brightness can be measured directly from the sensing result of the illuminance sensor.

It is also possible to estimate the external environment based on the operating state of the vehicle equipment. For example, when the headlights are on, it can be estimated that the surroundings are dark. If the fog lights are lit, it can be assumed that the visibility is poor due to fog. If the wipers are operating, it can be assumed that it is raining and visibility is poor. If the wipers are operating at high speed, it can be assumed that the visibility is considerably poor due to heavy rain.

The external environment can also be estimated from the travel position information. In this example, by using GPS and a map database, it is possible to determine the road type of the road on which the vehicle is currently traveling. , can estimate good or bad prospects.

Devices for obtaining information about traffic conditions include, for example, vehicle speed sensors. Depending on the vehicle speed, it can be estimated whether the road is congested, congested, or quiet. When the road is congested or congested, there are many objects that the driver should pay attention to, and the amount of recognition for each object is small, so the effective area is narrow.

As other information, it is possible to obtain information as to whether the vehicle is traveling in a tunnel or in the vicinity of a tunnel from GPS and a map database. Because the tunnel is dark, visibility is poor. When driving at the entrance or exit of a tunnel, you will suddenly enter a dark place from a bright place, or suddenly go out from a dark place to a bright place, and you will suddenly be in a bright place. Since the brightness changes rapidly, the effective area narrows until your eyes get used to it.

The effective area size setting unit 11 sets the size of the effective area based on information acquired from various sensors 22 or vehicle equipment 23 . Note that what is shown in FIG. 4 is only an example, and it is also possible to set the size of the effective area using information from other sensors or vehicle equipment 23 . Further, the size of the effective area may be set by estimating the external environment by combining the information shown in FIG. 4 or considering the driver's habits. For example, when the time is night and the headlights are not lit, the size of the effective area may be set to 0. Also, when a driver who normally does not use the wipers is using the wipers, it may be determined that it is raining heavily and visibility is poor. If fog lamps that are not normally used are being used and the traveling speed is slow, it may be determined that the visibility is poor.

FIG. 5 is a flow chart showing the operation of the oversight detection device 2 of the second embodiment. The oversight detection device 2 detects an object around the vehicle using the millimeter wave radar 20 (S20). The oversight detection device 2 also acquires information from the various sensors 22 and the vehicle equipment 23 (S21), and sets the size of the effective area based on the acquired information (S22).

The oversight detection device 2 detects the line of sight of the driver from the captured image of the driver (S23). Subsequently, the overlook detection device 2 determines whether or not the driver recognizes the object detected by the millimeter wave radar 20, that is, whether or not the driver has overlooked the object (S24). Specifically, the oversight detection device 2 determines whether or not an object has been within the range of the effective area centered on the direction of the line of sight of the driver for a predetermined period of time. The size of the effective area used here is uniformly set regardless of the object.

When the oversight detection device 2 determines that there is an oversight, for example, it outputs information to that effect to an alarm device (S25), and issues an alarm to the driver to alert the driver.

The configuration and operation of the oversight detection device 2 according to the second embodiment have been described above. As with the oversight detection device 1 of the first embodiment, the oversight detection device 2 of the second embodiment uses the effective area to determine the oversight. detectable. The size of the effective area is set based on the external environment using information from various sensors 22 and vehicle equipment 23 .

Although the oversight detection device 2 of the present invention has been described in detail above with reference to the embodiments, the present invention is not limited to the above-described embodiments. The oversight detection device 2 of the embodiment described above may set the size of the effective area based on the age of the driver, and set the effective area smaller when the driver is older. Since the driver's cognitive ability is considered to decline with age, oversight detection can be performed more appropriately by setting the size of the effective area according to age.

Further, in the oversight detection device 2 of the embodiment described above, the effective area size setting unit 11 may set the effective area smaller when the object is stationary than when the object is moving. Since it is harder to perceive an object when it is stationary than when it is moving, by setting the effective area smaller when the object is stationary, it is possible to maintain a high degree of accuracy in detecting an oversight.

Further, in the oversight detection device 2 of the embodiment described above, the effective area size setting unit 11 may set the effective area smaller as the number of objects around the vehicle increases. When there are many moving objects, the amount of recognition of the driver decreases. Therefore, by reducing the effective area when there are many moving objects, it is possible to maintain a high degree of detection accuracy for overlooking. Also, when there are many moving objects, the oversight detection device 2 may set a longer time to determine that the driver has recognized the object.

INDUSTRIAL APPLICABILITY The present invention is useful as a device or the like that detects a driver's oversight and assists driving.

1, 2 Oversight detection device 10 Communication unit 11 Effective area size setting unit 12 In-vehicle camera 13 Line-of-sight detection unit 14 Judgment unit 15 Output unit 20 Millimeter wave radar 21 Visible light camera 22 Various sensors 23 Vehicle equipment

Claims (7)

a camera that captures the driver,
a line-of-sight detection unit that detects the line-of-sight of the driver based on the image captured by the camera;
an effective area size setting unit that sets the size of the effective area for recognizing each object based on the degree of confidence when recognizing each object from images captured by a visible light camera that captures the surroundings of the vehicle;
An effective area is set based on the line of sight and the size set by the effective area size setting unit, and the driver recognizes the object based on whether the object is within the effective area. A determination unit that determines whether or not
Oversight detection device. The determination unit determines that the driver recognizes the object when the time during which the object is in the effective area is equal to or longer than a predetermined threshold, and determines that the object is recognized by the driver when the time is less than the predetermined threshold. 2. The oversight detection device according to claim 1, wherein the oversight detection device determines that the oversight has occurred. 3. The oversight detection device according to claim 1, wherein the effective area size setting unit sets the size of the effective area based on the age of the driver. 4. The overlook detection device according to claim 1 , wherein the effective area size setting unit sets the effective area smaller when the object is stationary than when the object is moving. 5. The overlook detection device according to claim 1 , wherein the effective area size setting unit sets the effective area smaller as the number of objects around the vehicle increases. A method for detecting a driver's oversight of an object in the vicinity of a vehicle, comprising:
a step of acquiring an image captured by a camera that captures a driver;
detecting a line of sight of the driver based on the image;
setting the size of the effective area for recognizing each object based on the degree of confidence when recognizing each object from the image captured by the visible light camera that captures the surroundings of the vehicle;
An effective area is set based on the line of sight and the size of the set effective area, and whether or not the driver recognizes the object is determined based on whether the object is within the effective area. a step of determining
An oversight detection method comprising: A program for detecting an oversight by a driver of an object around a vehicle, comprising:
a step of acquiring an image captured by a camera that captures a driver;
detecting a line of sight of the driver based on the image;
setting the size of the effective area for recognizing each object based on the degree of confidence when recognizing each object from the image captured by the visible light camera that captures the surroundings of the vehicle;
An effective area is set based on the line of sight and the size of the set effective area, and whether or not the driver recognizes the object is determined based on whether the object is within the effective area. a step of determining
program to run.

Images