JP2013097456A - Driver face state notifying device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a driver face state notifying device allowing a person outside a vehicle to well recognize whether or not a driver checks outside the vehicle.SOLUTION: The driver face state notifying device comprises: a camera 11 (photographing means) for photographing a face of a driver 2; an image processing section 12 (face state detecting means) for detecting a face state of the driver 2 on the basis of a face image of the driver 2 photographed by the camera 11; a controlling section 13 and an image converting section 14 (notification information generating means) for generating notification information reflecting a change in the face state in real time on the basis of the face state of the driver 2 detected by the image processing section 12; and a display 15, and left and right lamps 16 and 17 (notifying means) for notifying the person outside the vehicle of the notification information generated by the controlling section 13 and the image converting section 14 in real time.

Description

  The present invention relates to a driver face condition notifying device, and more particularly to a driver face condition notifying device that supports prevention of accidents caused by driver's carelessness.

  When the driver neglects to display or visually check the blinker, an accident involving a bicycle or a pedestrian or a collision accident when changing lanes on a highway or the like has occurred. The cause of these accidents is often due to the driver's carelessness, but the driver's carelessness cannot be recognized by a person outside the vehicle such as a car, bicycle or pedestrian behind. In order to solve such a problem, for example, as described in Patent Document 1 below, whether the driver of the vehicle turning right or left recognizes the person outside the vehicle that exists on the side of the host vehicle. There is known a vehicular warning device capable of notifying whether or not. In addition, a technique for improving safety and comfort using the driver's line of sight is described in the following Patent Document 2, and the driving status of the vehicle can be confirmed later using a virtual image of the driver. The technology is described in Patent Document 3 below.

JP 2010-113601 A JP 2007-263931 A JP 2010-211163 A

  In the vehicle warning device described in Patent Document 1, a blue lamp, a red lamp, and a yellow lamp are provided on the left side mirror. When the driver's visual accuracy is high, the blue lamp is used, and when the visual accuracy is low, the red lamp is used. When the visibility accuracy is medium, the yellow lamp is turned on to enable the person outside the vehicle to recognize whether or not the driver has sufficiently confirmed the side. However, in this vehicle warning device, each lamp is controlled to be turned on under the condition that the vehicle turns right and left and that the pedestrian is walking in the blind spot of the driver. The warning conditions due to the lighting of each lamp were limited. In addition, even if each lamp is lit, there is a high risk that people outside the vehicle will miss or not notice.

  The present invention has been made to address the above-described problems, and an object of the present invention is to provide a driver face condition notification device that allows a person outside the vehicle to recognize well whether or not the driver has confirmed the outside of the vehicle. There is.

Means for Solving the Problems and Effects of the Invention

In order to achieve the above object, the driver face condition notifying device of the present invention includes:
Photographing means for photographing the driver's face;
Face state detecting means for detecting the driver's face state based on the driver's face image photographed by the photographing means;
Notification information creating means for creating notification information reflecting the change in the face state in real time based on the face state of the driver detected by the face state detection means;
Informing means for informing the person outside the vehicle of the notice information created by the notice information creating means in real time;
It is characterized by providing.

  In the driver face state notification device of the present invention, the notification information creation unit creates notification information that reflects changes in the face state in real time based on the driver's face state detected by the face state detection unit. Then, the notification means notifies the notification information created by the notification information creation means to a person outside the vehicle in real time.

  In this driver face state notification device, the notification information created by the notification information creation means is reported to a person outside the vehicle in real time. For this reason, people outside the vehicle can know the driver's attention state in real time, so that people outside the vehicle can detect in advance the dangers caused by the driver's carelessness and mistakes, effectively preventing accidents. It can be avoided.

  The notification information creation means creates a virtual face image virtualized based on the face orientation and the eyelid opening as the face state detected by the face state detection means, and is provided so as to be visible from the rear or side of the vehicle. The virtual face image can be displayed in real time on the display as the notification means. In this case, for example, when creating the virtual face image based on the eyelid opening degree as the face state detected by the face state detecting means, the notification information creating means has a predetermined ratio of the eyelid opening degree per time. Accordingly, it is preferable that the eye pattern in the virtual face image is determined.

  According to this, a virtual face image virtualized based on the face direction and the eyelid opening as the driver's face state is created and displayed in real time on a display provided so as to be visible from the rear or side of the vehicle. The For this reason, it is difficult for a person outside the vehicle to overlook the displayed virtual face image of the driver, and it can be immediately recognized from the virtual face image that the driver has not confirmed left and right. Accidents caused by mistakes can be avoided well.

  The notification information creating means is provided so as to be visually recognized from the rear or side of the vehicle by determining the driver's view in the left-right direction based on the face direction as the face state detected by the face state detecting means. Of the left and right lamps as notification means provided corresponding to the left and right directions, when the left side is determined to be visually confirmed, the left lamp is turned on in real time, and when the right direction is determined, the right lamp is turned on. It can also be configured to light up in real time. In this case, for example, the notification information creating unit determines whether the driver visually checks in the left-right direction based on the face orientation as the face state detected by the face state detection unit. It is preferable that the configuration is such that visual inspection in the left-right direction is determined according to the face orientation angle.

  According to this, the driver's visual inspection in the left-right direction is determined based on the face orientation as the driver's face state, and the vehicle is provided so as to be visible from the rear or side of the vehicle corresponding to the visual observation in each direction. Either the left or right lamp is lit. For this reason, in addition to or instead of notification by the display, it is possible to immediately recognize that the driver has not confirmed the left and right, so that accidents due to driver's carelessness or mistakes can be avoided even better. .

(A) The block diagram which shows the driver | operator face state alerting | reporting apparatus which concerns on this invention with the side surface schematic diagram and block diagram of a vehicle. (B) is explanatory drawing which shows the imaging | photography angle of view of a camera by planar view. The front view of the camera shown in FIG. 5 is a flowchart showing a first control program showing a flow of operation of the first embodiment. The flowchart which shows the subroutine of the face direction and the eyelid opening detection process in the 1st control program of FIG. The flowchart which shows the 2nd control program which shows the flow of an operation | movement of Example 1. FIG. The flowchart which shows the subroutine of the face direction detection process in the 2nd control program of FIG. Explanatory drawing which shows the process with respect to the picked-up image etc. in each step of FIG. Explanatory drawing which shows the specification in the case of determining the eaves opening degree in step S13, S17 of FIG. Explanatory drawing which shows the specification in the case of determining a face orientation in step S33 of FIG. (A)-(D) is explanatory drawing which shows an example of the virtual face image which virtualized the driver | operator's face, respectively. (A)-(D) is explanatory drawing which shows a notification aspect, respectively.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1A is a configuration diagram illustrating a driver face state notification device according to the present invention together with a schematic side view and a block diagram of a vehicle 1, and FIG. 1B is a plan view of a shooting angle of view of a camera 11. It is explanatory drawing shown. The driver face state notification device has a function of detecting the face state of the driver 2, such as the face direction and the eyelid opening, and notifying a person outside the vehicle of the face state, and includes a camera 11, an image processing unit 12, The controller 13 includes an image converter 14, a display 15, and left and right lamps 16 and 17.

  The camera 11 captures the face of the driver 2 and is attached on the dashboard 4 (or on the steering column) in front of the driver's seat 3. The camera 11 includes an image sensor such as a CCD image sensor or a CMOS image sensor, and converts an image of the face of the driver 2 imaged by the lens 11a into image data by the image sensor. The camera 11 has a vertical field of view set to θ1 (see FIG. 1A) and a horizontal field of view set to θ2 (see FIG. 1B). The face of the driver 2 falls within the shooting range determined by the shooting angles of view θ1 and θ2.

  In addition, as shown in FIG. 2, the camera 11 includes a near-infrared LED 11b that irradiates the driver's face with near-infrared light, and a visible light cut filter (not shown) that blocks visible light from outside light. It has. As a result, the face of the driver 2 can be photographed by the camera 11 even when the surroundings are dark, such as at night. The camera 11 corresponds to the photographing means of the present invention.

  The image processing unit 12 is composed of a microcomputer whose main components are a CPU, ROM, RAM, input / output unit (I / O), and the like, and analyzes an image photographed by the camera 11. Specifically, the image processing unit 12 performs, for example, binarization processing on the image data from the camera 11 and extracts a desired image portion (feature point). By performing these binarization processing and feature point extraction processing, the face image of the driver 2 is extracted, and based on the extracted face image, the face direction and the eyelid opening degree of the driver 2 are detected as a face state. The image processing unit 12 corresponds to the face state detection means of the present invention.

  The control unit 13 is composed of a microcomputer whose main components are a CPU, ROM, RAM, input / output unit (I / O), and the like, and the eye pattern of the driver 2 is determined based on the face state detected by the image processing unit 12. Based on the determined eye pattern, the image conversion unit 14 creates a virtual face image, and the created virtual face image is displayed on the display 15. Moreover, the control part 13 determines the visual observation of the driver | operator 2 to the left-right direction based on the face direction which the image process part 12 detected, and turns on the left-right lamp 16 based on the determination result.

  The image conversion unit 14 is composed of a microcomputer whose main components are a CPU, a ROM, a RAM, an input / output unit (I / O), and the like, and a virtual face image 2f corresponding to the face state detected by the image processing unit 12 (see FIG. 10), a known image conversion circuit is provided. Specifically, the ROM of the image conversion unit 14 stores a plurality of virtual face information (information for modeling, avatar information) for expressing facial expressions mainly including face orientation and eyelid opening. Has been. Then, the image conversion unit 14 creates (reads out) a virtual face image 2f corresponding to the face orientation detected by the image processing unit 12 and the eye pattern determined by the control unit 13 based on the virtual face information in real time. ). By virtualizing (modeling) the face state of the driver 2 in this manner, the face state of the driver 2 can be easily understood by a person outside the vehicle and can be grasped instantly. The control unit 13 and the image conversion unit 14 correspond to notification information creation means of the present invention.

  The display 15 is composed of, for example, a liquid crystal display, an organic EL display, a plasma display, and the like, and is provided in the rear portion 1a of the vehicle 1 so that a person outside the vehicle can see from the rear or side of the vehicle 1 ( FIG. 11). The display 15 is driven by turning on the ignition switch. The display 15 may be configured to be in a driving state on condition that the vehicle 1 is in a traveling state.

  The left and right lamps 16 and 17 are formed of, for example, LEDs, and are provided on the left and right side mirrors 5 and 6 so that a person outside the vehicle can visually recognize the vehicle 1 from the rear or side. The left and right lamps 16 and 17 are driven when the ignition switch is turned on. In addition, you may comprise so that the left-right lamps 16 and 17 may be in a drive state on condition that the vehicle 1 is a driving | running | working state. The display 15 and the left and right lamps 16 and 17 correspond to the notification means of the present invention.

  Next, the operation flow of the first embodiment configured as described above will be described based on the first control program of FIG. 3 and the second control program of FIG. First, the first control program in FIG. 3 will be described. In the first control program of FIG. 3, the image processing unit 12 executes the processes of steps S11 and S12, the control unit 13 executes the processes of steps S13, S14, and S17 to S19, and steps S15 and S16. This processing is executed by the image conversion unit 14.

  The image processing unit 12 acquires face image data from the camera 11 (S11), and analyzes the acquired face image (S12). In the analysis of the face image, as shown in the subroutine of FIG. 4, first, the position of the face area included in the photographed image is roughly detected (S21). FIG. 7A is a diagram illustrating an example of a captured image 101 acquired from the camera 11. The captured image 101 includes a face area 110 corresponding to the face of the driver 2 and a background area 102 other than that.

  FIG. 7B shows a luminance sum line 103 of the vertical pixel columns of the photographed image 101 corresponding to the positions of the pixel columns of the photographed image 101 of FIG. As indicated by the line 103, generally, the luminance sum of the face region 110 is made larger than the luminance sum of the background region 102. Therefore, by looking at the change in luminance sum of the line 103, it is possible to roughly grasp the position in the left-right direction of the face area 110 with respect to the captured image 101. In step 21, a rough position of the face area 110 in the horizontal direction and the vertical direction with respect to the captured image 101 is detected based on the luminance sum of the horizontal pixel columns in addition to the luminance sum of the vertical pixel columns of the captured image 101. .

  Next, a region corresponding to a facial part such as an eye or nose constituting the face is detected from the photographed image 101 (S22). FIG. 7C shows a photographed image 101. The face region 110 of the photographed image 101 includes a left eye region 111 corresponding to the left eye, a right eye region 112 corresponding to the right eye, a nose region 113 corresponding to the nose, and the like. The area corresponding to the face part is included. Each of these regions 111 to 113 has image feature amounts (edge information and luminance information) corresponding to each component. Therefore, in step S22, each image feature amount included in the captured image 101 is compared (matched) with a face component template prepared in advance to detect the region of each component constituting the face.

  Next, the final position of the face area 110 is detected based on the rough position of the face area 110 detected in step S21 and each face part detected in step S22 (S23). In this case, in steps S21 and S22, a plurality of regions that are candidates for the face region 110 and a plurality of regions that are candidates for the face component may be detected. Focus on the true facial parts, taking into account the nose and mouth below the nose. Further, since the face contour and the shape of the face part change depending on the face inclination, the detected face part is appropriately corrected in consideration of the shape change due to the face inclination.

  FIG. 7D shows the detection result in step S23, and specifically shows that the area 120 in the captured image 101 is detected as a face area. As described above, the face area 120 that matches the actual face area 110 can be detected by executing the processes of steps S21 to S23. In the first embodiment, a quadrangular face area 120 is detected.

  Next, the eyelid opening degree of the driver 2 is detected from the distance between the upper and lower eyelids of the eye area included in the captured image 101 (S24). Next, the face orientation (angle) of the driver 2 is detected from the symmetry of the facial parts included in the face area 120 detected in steps S21 to S23 (S25). Finally, the eyelid opening detected in step S24 and the face orientation detected in step S25 are output to the control unit 13 (S26). When the control unit 13 acquires the face state information including the face direction and the eyelid opening degree from the image processing unit 12, the control unit 13 executes the processes after step S13 in FIG.

  Steps S13 and S17 are based on whether the eyes are fully open (wakes up), when they are about to sleep (when they are sleepy), and when they are completely closed (when they are sleeping). The process which determines the eaves opening degree for distinguishing and notifying the person of this is shown. In this process, for example, as shown in FIG. 8, when the heel opening is 20% (second threshold D2), and when the heel opening is 50% larger than D2 (first threshold D1). And the ratio P% (for example, P = 80) of the eyelid opening degree D for a predetermined T seconds (for example, T = 30) is used as the second threshold value D2. The eye pattern corresponding to the eyelid opening degree is determined depending on whether it is smaller than, between the second threshold value D2 and the first threshold value D1, or larger than the first threshold value D1, respectively. Yes.

  Specifically, if both of the steps S13 and S17 are “NO”, that is, if the ratio of the soot opening exceeds the drowsiness threshold D1% (the soot opening> D1) is P% or more, the driver wakes up. The eye pattern of the virtual face image is set to the eye pattern 3 shown in FIGS. 10A and 10B, for example (S19).

  On the other hand, “NO” in step S13 and “YES” in S17, that is, the ratio that the soot opening exceeds the drowsiness threshold D2% and becomes the drowsiness threshold D1% or less (D2 <瞼 opening degree ≦ D1). If P is greater than or equal to P%, it is determined that the driver is sleepy, and the eye pattern of the virtual face image is set to, for example, eye pattern 2 shown in FIG. 10C (S18).

  Further, if “YES” in step S13, that is, if the percentage of the eyelid opening is equal to or less than the drowsiness threshold D2% (the eyelid opening ≦ D2) is greater than or equal to P%, it is determined that the driver is sleeping, and the virtual For example, the eye pattern of the face image is set to the eye pattern 1 shown in FIG. 10D (S17). If the eyelid opening as the face state is displayed as an image close to the actual face image, it is considered extremely difficult for a person outside the vehicle to know the size of the eyelid opening, but step S14, By executing the processing of S18 and S19, the saddle opening is assigned to one of the eye patterns 1 to 3 and set, so that the outside opening of the driver 2 can be made extremely easy to understand for a person outside the vehicle.

  The image conversion unit 14 creates a virtual face image (model) 2f in real time corresponding to the face orientation detected by the image processing unit 12 and the eye pattern determined by the control unit 13 (S15). The images are sequentially updated (S16). For example, as shown in FIG. 10, the created virtual face image 2 f expresses facial expressions mainly with the face orientation and the eyelid opening. The control unit 13 displays the created virtual face image 2f on the display 15 in real time.

  Next, the second control program in FIG. 5 will be described. In the second control program of FIG. 5, the image processing unit 12 executes the processes in steps S31 and S32, and the control unit 13 executes the processes in steps S33 to S36. The processing in steps S31 and S32 is the same as the processing in steps S11 and S12 in FIG. However, unlike step S12, step S32 is configured to acquire only face orientation information. Correspondingly, in the subroutine of FIG. 6 executed in the process of step S32, step S24 (the saddle opening degree detection process) is omitted from the subroutine shown in FIG. Regarding the subroutine of FIG. 6, the same reference numerals are assigned to the processes corresponding to those of the subroutine shown in FIG.

  When acquiring the face orientation information from the image processing unit 12, the control unit 13 executes the processing from step S33 onward in FIG. Step S33 shows a process of determining the degree of face orientation for distinguishing and notifying a person outside the vehicle when the driver 2 is facing the front, when it is in the left direction, and when it is in the right direction. In this process, for example, as shown in FIG. 9, the face direction is 0 °, the right direction is positive, the left direction is negative, and the right and left directions are threshold values. , The direction of the face (angle) is viewed to the right or left depending on whether it is larger than the right threshold (T1, for example 20 °) or smaller than the left threshold (T2, for example, −20 °) Whether or not is determined.

  Specifically, if T2 <face orientation <T1 in step S33, the face orientation is considered to be the front, and the control unit 13 maintains the left and right lamps 16 and 17 from the lit state to the unlit state or from the unlit state. (S35). On the other hand, if T1 <face orientation in step S33, the face orientation is regarded as the right direction, and the control unit 13 maintains the right lamp 17 from the unlit state to the lit state or the lit state (S34). If the face orientation is smaller than T2 in step S33, it is considered that the face orientation is the left direction, and the control unit 13 maintains the left lamp 16 from the unlit state to the lit state or the lit state (S36). It is possible to simplify the contents of the lighting control program by determining whether or not the left and right lamps 16 and 17 are lit using each threshold value.

  By executing the first control program in FIG. 3 and the second control program in FIG. 5, a person outside the vehicle, for example, a virtual face image 2 f on the display 15 as shown in FIG. The lighting state by 16 and 17 can be visually recognized. Specifically, if the driver is facing the front without falling asleep, the virtual face image 2 f facing the front is displayed on the display 15. At this time, the left and right lamps 16 and 17 are turned off (FIG. 11A).

  If the driver visually confirms the left rear at the time of a left turn at an intersection, a virtual face image 2f facing left is displayed on the display 15. At this time, the left lamp 16 is turned on (FIG. 11B). Further, when the driver visually confirms the right rear at the time of changing the lane from the driving lane to the overtaking lane, the virtual face image 2 f facing right is displayed on the display 15. At this time, the right lamp 17 is turned on (FIG. 11C).

  On the other hand, if the driver is sleepy, the display 15 displays a virtual face image 2f that is likely to sleep. At this time, the left and right lamps 16 and 17 are turned off (FIG. 11D). If the left turn signal of the vehicle 1 is in a blinking state, for example, before the intersection, the notification mode shown in FIG. Bicycles, pedestrians, and the like that are at the left rear of 1 can be predicted to be involved in the vehicle 1, and can pay sufficient attention to the vehicle 1.

  Further, for example, when the vehicle 1 is traveling on a driving lane on an expressway, when the right turn signal of the vehicle 1 is turned off, the face direction of the driver 2 on the display 15 repeatedly looks right. Then, the succeeding vehicle traveling in the overtaking lane can predict that the vehicle 1 is likely to change the lane to the overtaking lane, and can pay sufficient attention to the vehicle 1. . In addition, when the vehicle 1 is meandering, the display 15 can be used to determine whether it is caused by the driver's immaturity or whether the vehicle 1 is sleepy and swaying. it can.

  As is clear from the above description, in the first embodiment, the control unit 13 and the image conversion unit 14 (notification information creation unit) are the face states detected by the image processing unit 12 (face state detection unit). A virtual face image 2f virtualized based on the face orientation and the eyelid opening is created, and the virtual face image 2f is displayed in real time on a display 15 (notification means) provided so as to be visible from the rear or side of the vehicle 1. Display.

  A virtual face image 2f virtualized based on the face direction and the eyelid opening as the face state of the driver 2 is created and displayed in real time on a display 15 provided so as to be visible from the rear or side of the vehicle 1. . Accordingly, it is difficult for a person outside the vehicle to overlook the displayed virtual face image 2f of the driver 2, and the driver 2 can immediately recognize from the virtual face image 2f that the driver 2 has not confirmed left and right. Accidents due to carelessness or mistakes 2 can be avoided well.

  In the first embodiment, the control unit 13 (notification information creating unit) moves the driver 2 in the left-right direction based on the face orientation as the face state detected by the image processing unit 12 (face state detection unit). Visual observation is determined (S33). When visual observation in the left direction is determined, the left lamp 16 is turned on in real time, and when visual observation in the right direction is determined, the right lamp 17 is turned on in real time.

  The left and right sides of the vehicle 2 are determined so as to be visible from the rear or side of the vehicle 1 in accordance with the visual direction in each direction based on the face direction as the face state of the driver 2. One of the lamps 16 and 17 is turned on. Thereby, in addition to the notification by the display 15, it is possible to immediately recognize that the driver 2 has not confirmed the left and right, so that accidents caused by the carelessness or mistake of the driver 2 can be avoided even better. .

  In the first embodiment, the right and left lamps 16 and 17 are used as the notification means in addition to the display 15. However, either the display 15 or the left and right lamps 16 or 17 is used, and the corresponding figure is shown. It is good also as a structure which implements either the 1st control program of 3 and the 2nd control program of FIG.

  The left and right lamps 16 and 17 are not limited to being provided on the left and right side mirrors 5 and 6, respectively. For example, an image is displayed on the display 15 together with the virtual face image 2f, or in the vehicle width direction at the rear portion 1a of the vehicle 1. You may comprise so that the bar-shaped indicator light extended to may be used.

DESCRIPTION OF SYMBOLS 1 Vehicle 2 Driver 2f Virtual face image 5, 6 Left and right side mirror 11 Camera (photographing means)
12 Image processing unit (face state detection means)
13 Control unit (notification information creation means)
14 Image converter (notification information creation means)
15 Display (notification means)
16, 17 Left and right lamps (notification means)

Claims (5)

Photographing means for photographing the driver's face;
Face state detection means for detecting the driver's face state based on the driver's face image photographed by the photographing means;
Notification information creating means for creating notification information reflecting the change in the face state in real time based on the face state of the driver detected by the face state detection unit;
Informing means for informing the person outside the vehicle of the notice information created by the notice information creating means in real time;
A driver face state notification device comprising:   The notification information creation means creates a virtual face image virtualized based on the face orientation and the eyelid opening as the face state detected by the face state detection means, and is visually recognized from the rear or side of the vehicle. The driver face state notification device according to claim 1, wherein the virtual face image is displayed in real time on a display as the notification unit provided in a possible manner.   The notification information creating means is provided so as to determine the driver's view in the left-right direction based on the face orientation as the face state detected by the face state detecting means, and to be visible from the rear or side of the vehicle. In addition, of the left and right lamps serving as the notification means provided corresponding to the left and right directions, the left lamp is turned on in real time when the left direction is determined to be viewed, and the right side when the right direction is determined. The driver face state notification device according to claim 1 or 2, wherein the lamp is turned on in real time.   The notification information creating means creates a virtual face image based on the eyelid opening degree as the face state detected by the face state detecting means, according to a predetermined ratio of the eyelid opening degree per time. The driver face state notification device according to claim 2 or 3, wherein an eye pattern in the virtual face image is determined.   When the notification information creating means determines the driver's visual observation in the left-right direction based on the face orientation as the face state detected by the face state detection means, the face orientation in the left-right direction with respect to the front The driver face state notification device according to claim 3, wherein visual observation in the left-right direction is determined according to an angle.