JP4872245B2 - Pedestrian recognition device - Google Patents

Description

  The present invention relates to a pedestrian recognition device that is mounted on a vehicle and detects a pedestrian in front of the vehicle.

In recent years, devices that assist the recognition of pedestrians when traveling at night are starting to be put into practical use. The following [Patent Document 1] also discloses such a pedestrian recognition device. Such a system is called a night view system or a night vision system, and often uses an infrared camera. A pedestrian is recognized from an image captured by an infrared camera, and this is notified to a vehicle occupant (driver) by displaying it on a display (sometimes used together with a buzzer). A pedestrian may be highlighted in the image on the display. A system that uses an infrared camera and a distance sensor to increase recognition accuracy has also been studied ([Patent Document 1]).
JP 2001-108758 A

  However, in such a conventional night view system, pedestrians can be detected with high accuracy, but all pedestrians are notified to passengers (drivers). That is, a pedestrian that can be easily recognized visually by the driver is also notified on the display. Since all the pedestrians already recognized by the driver and pedestrians that are difficult for the driver to be presented are presented, unnecessary information is announced to the driver, which is why it is really emphasized and announced. The pedestrian who should (highlight) should be buried in unnecessary information.

  For example, consider a case where there are many pedestrians that are visible in the foreground and there are pedestrians that are not illuminated by headlights slightly ahead. In the conventional night view system, the pedestrian who is relatively in front (the size [area] is large because it is in front) is emphasized and displayed (highlighted), so this notification is important for the driver. It may be difficult to recognize a pedestrian's notice (display) that is difficult to see and a little information. Accordingly, an object of the present invention is to efficiently notify the driver of pedestrian information that is really necessary for pedestrian recognition in front of the vehicle, that is, only pedestrians that are difficult to see or have not yet recognized. An object of the present invention is to provide a pedestrian recognition device that can be recognized.

The pedestrian recognition apparatus according to claim 1 is a pedestrian recognition apparatus for recognizing a pedestrian in front of a vehicle using an imaging result of a visible light camera and a far infrared camera mounted on the vehicle, the far infrared camera A pedestrian with a low pedestrian specific recognition rate that has a high pedestrian specific recognition rate when the object detected from the imaging result is a pedestrian and a low pedestrian specific recognition rate with the pedestrian detected by the visible light camera. A pedestrian-specific recognition rate with an object that is detected from the far-infrared camera imaging results and a pedestrian-identified object that is detected from the visible-light camera imaging results. Based on the recognition rate, a visually difficult pedestrian rate of whether or not the pedestrian recognized by each camera is difficult to see is calculated, and when the visually difficult pedestrian rate is a predetermined value or more, the walking Display person It is characterized by displaying a visual difficulties pedestrian.

  In addition, the case where it is specified that only one camera is a pedestrian (particularly, when a visible light camera cannot specify a pedestrian) is included. In this case, the pedestrian specific recognition rate of the camera on which the pedestrian cannot be specified may be 0%.

The invention according to claim 2, in the pedestrian recognition device according to claim 1, far-infrared camera and in addition to the visible light camera, measurable distance the magnitude of the distance and said object with the vehicle in front of the object A sensor is provided, and the detection result of the distance sensor is also used to calculate the pedestrian rate difficult to see.

  According to the pedestrian recognition device according to claim 1, it is considered that the recognition rate (pedestrian specific recognition rate) for identifying a pedestrian with a visible light camera is low and the vehicle occupant (driver) is difficult to see. There is a high possibility that a pedestrian who is a pedestrian and has a high pedestrian specific recognition rate with a far-infrared camera is considered not to recognize that the vehicle occupant is a pedestrian. By notifying such a pedestrian to the vehicle occupant through the display, it is possible to recognize a pedestrian that is difficult for the vehicle occupant to recognize with high accuracy.

Also, for pedestrians with a high pedestrian specific recognition rate with a visible light camera, it seems that the vehicle occupant is visually recognizing, so no notification is made on the display and unnecessary information is not presented The information regarding the pedestrian who is difficult to see is surely recognized by the occupant. Incidentally, display serves and LCD display mounted on the instrument panel, such as head-up display in which an image is displayed on the front windscreen inner surface, an image (video: Video) What long as you can display But it ’s okay.

And the pedestrian specific recognition rate that the object detected from the imaging result of the far-infrared camera is a pedestrian is high and the pedestrian specific recognition rate that the object detected from the imaging result of the visible light camera is a pedestrian is A low pedestrian is notified to the vehicle occupant. At this time, based on the above-described two pedestrian specific recognition rates, a visually difficult pedestrian that is considered difficult for the vehicle occupant to view is notified by the display. By notifying pedestrians that are difficult for the vehicle occupants to see, pedestrians that the vehicle occupants seem to recognize visually are not notified and unnecessary information is not presented. Only the information regarding the above-mentioned pedestrians with difficulty in visual recognition is surely recognized by the occupant.

At this time, on the basis of the far-infrared camera according Pedestrian particular recognition rate and visible light camera according Pedestrian particular recognition rate, visual difficulties pedestrian rate (how much eye view which is hard or those obtained by digitizing: difficult to visually larger ) Is calculated, and the pedestrian with difficulty in seeing is determined to be a visually difficult pedestrian when the rate of pedestrian with difficulty in seeing is equal to or greater than a predetermined value. In this way, it is possible to efficiently determine a pedestrian with difficulty in viewing using two cameras.

According to the pedestrian recognition apparatus of the second aspect , a higher recognition rate can be realized by using the detection result of the distance sensor in addition to the far-infrared camera and the visible light camera.

  The block diagram which shows the structure of the pedestrian recognition apparatus of this invention is shown in FIG. This pedestrian recognition device is mounted on a vehicle, includes a far-infrared camera 1 and a visible light camera 2 that capture the front of the vehicle, and further includes a distance sensor 3 that measures a distance to an object in front of the vehicle. ing. These cameras and sensors are connected to a determination processing ECU 4 that performs arithmetic processing related to pedestrian recognition.

  The far-infrared camera 1 is a camera that projects heat distribution, and has a performance that can perceive a human being through an obstacle such as a bush that is planted. Further, the far-infrared camera 1 has a capability of sensing humans not only at daytime but also at night. The visible light camera 2 projects a vehicle front image projected by daylight sunlight or nighttime headlight light in accordance with human visual ability (such as visual acuity and bird eye condition). In order to match the human visual ability, the focus, dynamic range, alias, etc., are matched as camera specifications.

  The distance sensor 3 scans the front of the vehicle using a laser, a millimeter wave, or the like, and detects the distance from the object in front of the vehicle in the same range as the cameras 1 and 2 described above as a distribution. As for the scenery in front of the vehicle, it is easy to understand if the distance from the object can be grasped like a contour line. The detection results of these cameras 1 and 2 and the distance sensor 3 can be processed in a time series in the determination processing ECU 4. That is, it can be processed as a moving image, and the movement (speed and direction) of a pedestrian can be detected.

  The determination processing ECU 4 performs image recognition processing on the detection results (image: video) of these cameras 1 and 2 and the distance sensor 3, calculates the probability that the detected object is a pedestrian, and seems to be a detected pedestrian. The relative distance from the detected object is acquired from the detection result of the distance sensor 3. Furthermore, the determination processing ECU 4 also has a function of improving the determination accuracy of being a pedestrian using the estimated human area calculated from the calculated distance, and the distance image, time variation, and skin pedestrian recognition processing It also has a function of improving the determination accuracy of pedestrians using the result of having performed.

  A lane recognition system 5 and a navigation system 6 are also connected to the determination processing ECU 4. By using these systems in combination, more advanced pedestrian recognition can be performed. Details will be described later. The determination processing ECU 4 is also connected to a display 7 for notifying a pedestrian (a pedestrian who is difficult to see) finally identified to an occupant (driver: driver 8). The display 7 also has a buzzer that emits a warning sound when a pedestrian with difficulty in viewing is detected.

  Examples of detection by the cameras 1 and 2 and the distance sensor 3 described above are shown in FIGS. FIG. 2 is an example of an image in front of the vehicle at night acquired by the visible light camera 2. Here, a case where there are pedestrians on the right front side and the back left side will be described as an example. As can be seen from FIG. 2, the visible light camera 2 can detect a pedestrian in front of the right hand. However, the pedestrian in the back left part is assimilated with the darkness of the background, and cannot be detected as a pedestrian.

  In addition, the detection as a pedestrian detects whether it is a pedestrian from the distance to the object, the magnitude | size of the target object in the distance, and the shape of the object. Regarding the determination of whether or not the person is a pedestrian, since conventional logic can be used, further detailed explanation is omitted here.

  FIG. 3 is an image example obtained by the far-infrared camera 1. The higher the temperature, the thinner (white) the image is displayed. As can be seen from FIG. 3, in the far-infrared camera 1, both the pedestrian on the right front side and the pedestrian on the left back can be detected as a pedestrian. FIG. 4 is an image example of the distance sensor 3. The closer it is, the darker (black) it is displayed. As can be seen from FIG. 4, the distance sensor 3 can detect both a pedestrian on the right front side and a pedestrian on the left back as a pedestrian.

  An example of control for notifying a vehicle occupant via a display of only a pedestrian (a pedestrian who is difficult to see) that is difficult for the vehicle occupant (driver) to recognize will be described with reference to FIG. First, as described above, the far-infrared camera 1 performs object detection relating to the front of the vehicle (step 500). For each detected object, the coordinates (x1, y1), the motion vector (Vx, Vy), and the area Sa are calculated (step 505). Similarly, the object detection regarding the vehicle front is performed by the visible light camera 2 (step 510). For each detected object, a coordinate (x2, y2), a motion vector (Wx, Wy), and an area Sb are calculated (step 515).

  The coordinates of the result of step 505, the detection result by the distance sensor 3 (the distance d to the object corresponding to the coordinates, and the probability ζ [here, ζ = 80%] that the object corresponding to the coordinates is a pedestrian) Then, a pedestrian determination process is performed, and a pedestrian probability (pedestrian specific recognition rate) α by the far-infrared camera 1 is calculated (step 525). Here, α is calculated as 90%, and the probability that a specific object is a pedestrian is calculated as 90%.

  Similarly, the coordinates of the result of step 515, the detection result by the distance sensor 3 (the distance d to the object corresponding to the coordinates, and the probability ζ [here ζ = 80%] that the object corresponding to the coordinates is a pedestrian. ]), A pedestrian determination process is performed, and a pedestrian probability (pedestrian specific recognition rate) β by the visible light camera 2 is calculated (step 530). Here, β is calculated as 10%, and the probability that a specific object is a pedestrian is calculated as 10%. That is, in the visible light camera 2 (that is, the visual observation of the driver), the object is not easily recognized as a pedestrian.

  Thereafter, the specific pedestrian determination is performed based on the pedestrian specific recognition rate α by the far-infrared camera 1 in step 525 and the pedestrian specific recognition rate β by the visible light camera 2 in step 530. That is, the specific pedestrian probability (the difficulty in viewing pedestrians) γ as to whether or not the specified pedestrian is difficult to see is calculated (step 535). Here, the visually difficult pedestrian rate γ is calculated using a calculation formula of γ = α × (α−β) [here, 90% × (90% −10%) = 72%]. That is, the higher the value of γ, the more difficult it is to be seen by the driver, and the driver does not tend to recognize the pedestrian.

  A threshold value (for example, 50%) is set in advance for this visually difficult pedestrian rate γ, and if it is higher than this threshold value, a specific pedestrian (difficult to see visually) is given to the driver (driver 8). Pedestrian) (step 540). There are various methods of notification, but it is conceivable that the pedestrian is surrounded by a frame on the display, and at the same time, a notification is given by a buzzer.

  In the example described above, the case where an object (pedestrian) that is both the far-infrared camera 1 and the visible light camera 2 can be detected has been described. However, the visible light camera 2 may not be able to detect the object (pedestrian). For example, it is a case like a pedestrian in the back left of FIG. A control example in such a case is shown in FIG. The overall flow is almost the same as the control in FIG.

  First, as shown in FIG. 6, the far-infrared camera 1 detects an object in front of the vehicle (step 600). For each detected object, the coordinates (x1, y1), the motion vector (Vx, Vy), and the area Sa are calculated (step 605). Similarly, the object detection regarding the vehicle front is performed by the visible light camera 2 (step 610). Here, it is a case where an object could not be detected, and in step 610, a result of no object detection being output.

  The coordinates of the result of step 605, the detection result by the distance sensor 3 (the distance d to the object corresponding to the coordinates, and the probability ζ [here, ζ = 80%] that the object corresponding to the coordinates is a pedestrian) Then, a pedestrian determination process is performed, and a pedestrian probability (pedestrian specific recognition rate) α by the far-infrared camera 1 is calculated (step 625). Here, α is calculated as 90%, and the probability that a specific object is a pedestrian is calculated as 90%. On the other hand, for the visible light camera 2, since no object was detected in step 610, the pedestrian probability (pedestrian specific recognition rate) β = 0% by the visible light camera 2 when the pedestrian determination process is performed. Calculated (step 630).

  Thereafter, a specific pedestrian determination is performed from the pedestrian specific recognition rate α by the far-infrared camera 1 in step 625 and the pedestrian specific recognition rate β by the visible light camera 2 in step 630, and the visually difficult pedestrian rate γ is determined. Calculate (step 635). Here, the calculation formula γ = α × (α−β) is used. In this case, γ = 90% × (90% −0%) = 81%. That is, since the recognition rate by the visible light camera 2 is lower than that described above (unrecognized: undetected), the final visually difficult pedestrian rate γ is higher than that described above. Also in the case of this example, as in the above example, the driver (driver 8) is notified as a specific pedestrian (a pedestrian who is difficult to see) (step 640).

  Further, in the next example, a case will be described in which the visible light camera 2 can sufficiently detect an object (pedestrian). For example, this is the case of a pedestrian in front of the right side of FIG. A control example in such a case is shown in FIG. First, as shown in FIG. 7, the far-infrared camera 1 detects an object in front of the vehicle (step 700). For each detected object, the coordinates (x1, y1), the motion vector (Vx, Vy), and the area Sa are calculated (step 705). Similarly, the object detection regarding the vehicle front is performed by the visible light camera 2 (step 710). For each detected object, a coordinate (x2, y2), a motion vector (Wx, Wy), and an area Sb are calculated (step 515).

  The coordinates of the result of step 705, the detection result by the distance sensor 3 (the distance d to the object corresponding to the coordinates, and the probability ζ [here, ζ = 80%] that the object corresponding to the coordinates is a pedestrian) Then, a pedestrian determination process is performed, and a pedestrian probability (pedestrian specific recognition rate) α by the far-infrared camera 1 is calculated (step 725). Here, α is calculated as 90%, and the probability that a specific object is a pedestrian is calculated as 90%.

  Similarly, the coordinate of the result of step 715, the detection result by the distance sensor 3 (the distance d to the object corresponding to the coordinate, and the probability ζ [here ζ = 80%] that the object corresponding to the coordinate is a pedestrian ]), A pedestrian determination process is performed, and a pedestrian probability (pedestrian specific recognition rate) β by the visible light camera 2 is calculated (step 730). Here, β is calculated as 80%, and the probability that a specific object is a pedestrian is calculated as 80%. That is, the visible light camera 2 (that is, the visual observation of the driver) is sufficiently aware of the object as a pedestrian.

  Thereafter, the specific pedestrian determination is performed from the pedestrian specific recognition rate α by the far-infrared camera 1 in step 725 and the pedestrian specific recognition rate β by the visible light camera 2 in step 530, and the visually difficult pedestrian rate γ is determined. Calculate (step 735). Here, the calculation formula γ = α × (α−β) is used. In this case, γ = 90% × (90% -80%) = 9%. That is, since the recognition rate by the visible light camera 2 is high, the final difficult-to-view pedestrian rate γ is low, and it is determined that the driver will be viewing the pedestrian. In the case of this example, the driver (driver 8) is not notified as a specific pedestrian (a pedestrian with difficulty in viewing) (step 740).

  In this way, for pedestrians that the driver seems to be watching sufficiently, notifications on the display are not made as unnecessary information, and only pedestrians who are not seen (difficult to see) are notified. Thus, it is possible to notify the driver of information relating only to pedestrians to be notified. Such a notification method is very effective when running on a road with many pedestrians or when there are many pedestrians that can be seen in front and there are pedestrians that are difficult to see in the back.

  Further, as described above, the recognition device of the present embodiment is also connected to the lane recognition system 5 and the navigation system 6. Referring to the lane information of the lane recognition system and the map information of the navigation system 6, it is determined whether the detected pedestrian is close to the road or whether the pedestrian is moving with respect to the road. The degree of danger may be calculated for each pedestrian together with the distance. And based on this risk level, you may label (with rank of a risk level) for every pedestrian.

  It is preferable that the pedestrian who has a high possibility of colliding with the vehicle (high risk) is emphasized and notified. For example, a pedestrian with a low risk level may be displayed with a green frame on the display, and the color of the frame may be changed to orange or red as the risk level increases. Also, when using a buzzer together, it is better to be able to notify whether or not there is a pedestrian with a high degree of danger by changing the tone and size of the buzzer.

  In addition, this invention is not limited to embodiment mentioned above. For example, in the above-described embodiment, the distance sensor is used in combination, but the same can be performed with a near-infrared camera instead of the distance sensor. Alternatively, a far-infrared camera or a visible light camera may be provided as a stereo camera, and the distance can be obtained by their parallax, and this distance data may be used.

It is a block diagram of one Embodiment of the pedestrian recognition apparatus of this invention. It is an example of detection by a visible light camera. It is an example of detection by a far-infrared camera. It is an example of detection by a distance sensor. It is a flowchart which shows the 1st example of pedestrian recognition control. It is a flowchart which shows the 2nd example of pedestrian recognition control. It is a flowchart which shows the 3rd example of pedestrian recognition control.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Far-infrared camera, 2 ... Visible light camera, 3 ... Distance sensor, 4 ... Determination processing ECU, 5 ... Lane recognition system, 6 ... Navigation system, 7 ... Display (speaker), 8 ... Driver.

Claims (2)

A pedestrian recognition device for recognizing a pedestrian in front of the vehicle using imaging results of a visible light camera and a far-infrared camera mounted on the vehicle,
The pedestrian specific recognition rate that the object detected from the imaging result of the far-infrared camera is a pedestrian is high, and the pedestrian specific recognition rate that the object detected from the imaging result of the visible light camera is a pedestrian is It has a display to notify the vehicle occupants of low pedestrians ,
Based on the pedestrian specific recognition rate that the object detected from the imaging result of the far-infrared camera is a pedestrian and the pedestrian specific recognition rate that the object detected from the imaging result of the visible light camera is a pedestrian When the pedestrian rate that is difficult to see for pedestrians recognized by each camera is calculated, and the pedestrian rate that is difficult to see is greater than or equal to a predetermined value, the pedestrian is displayed on the display. A pedestrian recognition device, characterized in that the vehicle occupant is displayed as a pedestrian who is difficult to see . In addition to the far-infrared camera and the visible light camera, a distance sensor capable of measuring the distance to the object in front of the vehicle and the size of the object is provided. The pedestrian recognition device according to claim 1 , wherein a detection result is also used.

Images