JP2022188532A - Road information sign recognition device - Google Patents

Abstract

To provide a road information indication object recognition device capable of accurately recognizing road information of a road information indication object.SOLUTION: A road information indication object recognition device 1 comprises a camera 2 which images a front side of a vehicle 10, a LiDAR 3 including an imaging range of the camera 2 in a detection range, and an information processing ECU 4. In a case where an indication object candidate detected by the LiDAR 3 is determined to be a road information indication object, the information processing ECU 4 performs image recognition on a range corresponding to the road information indication object in an image picked up by the camera 2 and recognizes road information of the road information indication object.SELECTED DRAWING: Figure 1

Description

The present invention relates to a road information sign recognition device.

A road information marker recognition device is also called TSR (Traffic Sign Recognition), and as a technology related to the road information marker recognition device, for example, a road sign determination device described in Patent Literature 1 is known. The road sign determination device described in Patent Document 1 includes an image recognition unit that detects a first road regulation sign by executing image recognition processing on a captured image in front of a vehicle captured by an imaging unit, and map data storage. a map information acquisition unit that acquires a road link including a second road control sign at the vehicle's location from a map information acquisition unit that, when the road link changes, the current first road control sign detected on the current road link; a road sign determiner that determines a final road control sign on the current road link, also utilizing at least one of the second road control signs.

JP 2015-102898 A

By the way, road information markers (eg, signs, signboards, traffic signals, etc.) are likely to be newly installed, relocated, or removed, so the freshness of the information in the map information tends to be low. Therefore, in the above technology for recognizing road information of road information markings based on map information, there is a possibility that the accuracy of the recognition will be degraded.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a road information sign recognizing device capable of accurately recognizing road information on road information signs.

A road information marker recognition device according to an aspect of the present invention includes a camera that captures at least a partial area around a vehicle, an active sensor that includes at least a part of the imaging range of the camera as a detection range, and an active sensor. When it is determined that the detected sign candidate is a road information sign, a process of recognizing the range corresponding to the road information sign in the image captured by the camera and recognizing the road information of the road information sign. a device;

This road information marker recognizing device can recognize the road information of the road information marker by effectively cooperating the camera and the active sensor without using map information including information on the road information marker. can. Even if the road information sign is newly installed, relocated or removed, it is possible to recognize the corresponding road information. Therefore, according to the road information marker recognition device according to one aspect of the present invention, it is possible to accurately recognize the road information of the road information marker.

In the road information marker recognition device according to one aspect of the present invention, the processing device extracts marker candidates from the detection result of the active sensor, and based on at least one of the shape, position, and reflectance of the marker candidate, It may be determined whether or not the candidate sign is a road information sign. In this case, it is possible to determine whether or not the candidate marker detected by the active sensor is a road information marker by using at least one of the shape, position and reflectance of the candidate marker.

In the road information marker recognition device according to one aspect of the present invention, the active sensor may be LiDAR. In this case, the LiDAR can be used as an active sensor to accurately recognize the road information of the road information sign.

ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the road information sign recognition apparatus which can recognize the road information of a road information sign with high precision.

FIG. 1 is a block diagram showing the configuration of a road information marker recognition device according to an embodiment. FIG. 2 is a flow chart showing an example of road information recognition of a road information sign carried out by the road information sign recognizing device according to the embodiment. FIG. 3 is a diagram showing an example of an image acquired by a camera. FIG. 4 is a diagram showing an example of point group information acquired by LiDAR. FIG. 5 is a diagram showing an example of point group information acquired by LiDAR. FIG. 6 is a diagram showing an example of point group information acquired by LiDAR. FIG. 7 is a diagram showing an example of an image acquired by a camera. FIG. 8 is a block diagram showing the configuration of a road information marker recognition device according to a modification.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same reference numerals are used for the same or corresponding elements, and overlapping descriptions are omitted.

FIG. 1 is a block diagram showing the configuration of a road information marker recognition device 1. As shown in FIG. As shown in FIG. 1 , the road information marker recognition device 1 is mounted on a vehicle 10 . A road information marker recognition device 1 recognizes road information of a road information marker. The vehicle 10 is not particularly limited, but here assumes a vehicle having a limited automatic driving function, excluding fully automatic driving.

Road information markers are, for example, signs, billboards, traffic lights, and the like. The road information sign is not particularly limited, and includes any sign that indicates road information. The road information markings targeted by the present embodiment are, for example, from the viewpoint of effectiveness of recognition, circular road signs and inverted triangular road signs. The road information marker recognition device 1 includes a camera 2 , LiDAR [Light Detection and Ranging] 3 , information processing ECU [Electronic Control Unit] 4 and HMI [Human Machine Interface] 5 .

The camera 2 is imaging equipment that images the external environment of the vehicle 10 . The camera 2 is provided, for example, behind the windshield of the vehicle 10 and captures an image of the front of the vehicle 10 . The cameras 2 may be provided on the rear and side surfaces of the vehicle 10 . The camera 2 acquires an image in front of the vehicle and transmits the acquired image to the information processing ECU 4 . The camera 2 may be a monocular camera or a stereo camera. The camera 2 is not particularly limited, and various cameras can be used. The camera 2 may image at least a partial area around the vehicle 10 .

LiDAR 3 is an active sensor that uses light to detect objects around vehicle 10 . The LiDAR 3 detects objects by outputting light for detection around the vehicle 10 and receiving light reflected by objects around the vehicle 10 . The LiDAR 3 here is mounted toward the front of the vehicle and detects objects in front of the vehicle. The mounting direction of the LiDAR 3 is not particularly limited, and it may be mounted toward the rear of the vehicle or toward the left and right sides of the vehicle. The objects are, for example, road information markings, structures, curbs on the road, and the like. The LiDAR 3 includes at least part of the imaging range of the camera 2 in its detection range. In other words, the detection range of LiDAR 3 includes the imaging range of camera 2 , or the detection range of LiDAR 3 is included in the imaging range of camera 2 .

The LiDAR 3 acquires point cloud information about a three-dimensional point cloud in front of the vehicle, and transmits the acquired point cloud information to the information processing ECU 4 . The point group information is information that includes at least a plurality of points (position data) forming an object. The point group information is information including three-dimensional distance and reflectance for each point. The camera 2 and the LiDAR 3 are aligned (calibrated), and acquire an image and point cloud information in front of the vehicle at the same constant cycle and in synchronization. The LiDAR 3 may be a flash type that acquires a point cloud in front of the vehicle at once, or a rotary type that performs a 360-degree scanning operation while rotating.

The information processing ECU 4 is an electronic control unit having a processor such as a CPU [Central Processing Unit] and a memory such as a ROM [Read Only Memory] and a RAM [Random Access Memory]. The information processing ECU 4 implements various functions by, for example, loading programs recorded in the ROM into the RAM and executing the programs loaded into the RAM by the CPU. The information processing ECU 4 may be composed of a plurality of electronic control units. The information processing ECU 4 constitutes a processing device.

The information processing ECU 4 extracts a marker candidate from the detection result (point group information) of the LiDAR 3 . As an extraction method, various known methods can be used. The information processing ECU 4 recognizes at least one of the shape, position, and reflectance of the extracted candidate marker, and determines whether the candidate marker is a road information sign based on at least one of the shape, position, and reflectance of the candidate marker. It is determined whether or not it is an object. When the information processing ECU 4 determines that the marker candidate is a road information marker, the information processing ECU 4 performs image recognition of the range corresponding to the road information marker in the image captured by the camera 2, and determines the road information of the road information marker. to recognize The information processing ECU 4 notifies the occupants (including the driver) of the recognized road information via the HMI 5 . The details of the processing of the information processing ECU 4 will be described later.

The HMI 5 is an interface for inputting/outputting information with the passenger. HMI5 has a display part (display) and a speaker, for example. The HMI 5 performs image output from the display section and audio output from the speaker according to the control signal from the information processing ECU 4 . HMI5 may be provided with HUD [Head Up Display]. The HMI 5 notifies the occupant of the road information of the road information sign recognized by the information processing ECU 4, for example, by image output from the display unit and audio output from the speaker.

Next, an example of road information recognition performed by the road information marker recognition device 1 will be described with reference to the flowchart of FIG.

For example, when the vehicle 10 starts running, the road information marker recognition device 1 executes the following road information recognition processing. That is, an image G (see FIG. 3) in front of the vehicle is acquired by the camera 2, and point group information T (see FIG. 4) in front of the vehicle is acquired by the LiDAR 3 (step S1). In the point cloud information T in FIG. 4, the left and right point clouds are reflected by the curb, and the center line is not shown due to the white line. In the point group information T of FIG. 4, the point group reflected by the road surface is removed in advance.

Subsequently, the information processing ECU 4 detects the left and right lane markers in the image G in front of the vehicle, and based on the detection results of the lane markers, determines the existence range (usually, on the sidewalk and on the roadway) where the road information marking is thought to exist. side) is specified (step S2). The information processing ECU 4 sets the existence range of the specified road information marking object in the point group information (range W1 of the dotted line in FIG. 5), and determines whether the road information marking object is located in the existence range of the road information marking object and is assumed to be the road information marking object. A point group of a candidate sign is extracted (step S3). In the example of FIG. 6, the point group of the marker candidates is the point group within the dotted rectangular range W2.

Subsequently, the information processing ECU 4 determines whether or not the extracted sign candidate is a road information sign (step S4). In step S4, the extracted candidate sign is evaluated using its shape, position (height position), and reflectance. For example, the information processing ECU 4 stores in advance a data table that associates shapes, positions, and reflectances for each type of a plurality of road information markings. In the above step S4, the extracted sign candidate can be evaluated by scoring using the data table and can be judged. Note that the shape of the sign candidate can be recognized using known image processing such as pattern matching, for example. The position of the marker candidate can be based on, for example, any one of the center position, the upper end position, the lower end position, the left end position, and the right end position of the marker candidate. The reflectance of a marker candidate can be based on, for example, any one of the maximum value, minimum value, and average value of the reflectance of each point group that constitutes the marker candidate.

In the example of this embodiment, since the road information marking is the speed sign 9, the shape of the road information marking is circular, and the diameter is designed to fall within a certain range (several tens of centimeters to a little less than 1.5 m). . Also, the installation height of the speed sign 9 falls within a range of about 1 m to 2 m. Since the speed sign 9 is provided with minute glass beads or the like that are invisible to the naked eye in order to improve the visibility of the sign under the headlight, the reflectance of the speed sign is stably above a certain level. Therefore, in step S4, the shape of the extracted candidate marker is a circle with a certain range, the position is in the range of 1 m to 2 m, and the reflectance is above a certain level. is determined to be

If it is determined that there is a road information sign, the information processing ECU 4 cuts out a range corresponding to the road information sign in the image G in front of the vehicle (dotted-line rectangular range W3 in FIG. 7) (YES in step S5; step S6). The range cut out from the image G in front of the vehicle is subjected to preprocessing (tilt correction, enlargement/reduction, etc.) and image recognition to recognize the road information of the road information sign (step S7). In this example, it recognizes that the road information sign is a speed sign and the speed limit value is "60 km/h". The method of preprocessing and image processing performed in step S7 is not particularly limited, and various known processing may be used.

The HMI 5 informs the occupant of the road information of the recognized road information sign (step S8). After step S8 or in the case of NO in step S5, if the vehicle 10 is not stopped, the process returns to step S1. If the vehicle 10 is stopped, the image recognition process ends (step S9). .

As described above, the road information marker recognition device 1 effectively cooperates with the camera 2 and the LiDAR 3 to recognize the road information of the road information marker without using the map information including the information on the road information marker. be able to. Even if the road information sign is newly installed, relocated or removed, it is possible to recognize the corresponding road information. Therefore, according to the road information marker recognition device 1, it is possible to accurately recognize the road information of the road information marker. It is possible to suppress the need to prepare map information on the vehicle 10 side and record the position of the road information sign in advance. Robust recognition becomes possible even for newly installed, relocated, and removed (disappeared) road information signs.

In the road information marker recognizing device 1, the information processing ECU 4 extracts marker candidates from the detection result of the LiDAR 3, and identifies the marker candidate as the road information marker based on at least one of the shape, position, and reflectance of the marker candidate. It is determined whether or not it is an object. In this case, it is possible to determine whether or not the marker candidate detected by the LiDAR 3 is a road information marker by using at least one of the shape, position and reflectance of the marker candidate.

The road information marker recognition device 1 includes LiDAR 3 as an active sensor. As a result, the LiDAR 3 can be used as an active sensor to accurately recognize the road information of the road information sign.

While various exemplary embodiments have been described above, various omissions, substitutions, and modifications may be made without being limited to the exemplary embodiments described above.

FIG. 8 is a block diagram showing the configuration of a road information marker recognition device 101 according to a modification. A road information marker recognizing device 101 according to a modification includes a GNSS [Global Navigation Satellite System] receiving unit 6, an internal sensor 7, and a map database 8 in addition to the above-described road information marker recognizing device 1 (see FIG. 1). more ready.

The GNSS receiver 6 constitutes a satellite positioning device that measures the position of the vehicle 10 (for example, the latitude and longitude of the vehicle 10) by receiving signals from four or more GNSS satellites. The GNSS receiver 6 transmits the measured position information of the vehicle 10 to the information processing ECU 4 . Internal sensors 7 include a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. A vehicle speed sensor is a detector that detects the speed of the vehicle 10 . The acceleration sensor is a detector that detects acceleration of the vehicle 10 . The yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) around the vertical axis of the center of gravity of the vehicle 10 . The internal sensor 7 transmits the detection result to the information processing ECU 4 .

The map database 8 is a database that records map information. The map database 8 is formed in a recording device such as an HDD (Hard Disk Drive) mounted on the vehicle 10, for example. The map information includes, for example, road position information, road shape information (for example, number of lanes, lane width, curvature information), position information of intersections and branch points, and the like. The map database 8 may be configured in a server that can communicate with the vehicle 10 .

The information processing ECU 4 of the road information marker recognizing device 101 estimates the positions of the left and right lane markers in the image G in front of the vehicle while interpolating them using the GNSS receiver 6, the internal sensor 7, and the map database 8. be able to. Therefore, even if the lane marker is blurred and difficult to distinguish from the camera 2, the location where the road information sign exists (the existence range of the road information sign (the range W1 of the dotted line in FIG. 5)) can be estimated satisfactorily. Therefore, more stable operation can be expected.

Although the LiDAR 3 is used as the active sensor in the above embodiment, the active sensor is not limited, and other active sensors may be used. In the above embodiment, image recognition by machine learning may be used. In the above embodiment, for example, when recognizing a road information sign such as a speed control sign, simple pattern matching may be used because the format is known and the numerical values to be displayed are limited.

In the above embodiment, an example in which one road information marker (marker candidate) exists in the image G and the point cloud information T has been described, but the number of road information markers in the image G and the point cloud information T is limited. There may be more than one. In addition, when a plurality of road information markings of different types exist in the image G and the point group information T, the road information for each may be recognized and notified to the occupant. On the other hand, when a plurality of road information markings of the same type exist in the image G and the point group information T, the road information about any one of them (for example, the one closest to the vehicle 10) is recognized. You can notify the crew.

DESCRIPTION OF SYMBOLS 1... Road information sign recognition apparatus, 2... Camera, 3... LiDAR (active sensor), 4... Information processing ECU (processing device), 9... Speed sign (road information sign), 10... Vehicle, G... Image, T: point cloud information.

Claims (3)

a camera that images at least some area around the vehicle;
an active sensor whose detection range includes at least part of the imaging range of the camera;
When it is determined that the marker candidate detected by the active sensor is a road information marker, the range corresponding to the road information marker in the image captured by the camera is image-recognized, and the road of the road information marker is recognized. and a processor for recognizing information. The processing device is
extracting a sign candidate from the detection result of the active sensor;
2. The road information marker recognition device according to claim 1, wherein whether or not said marker candidate is a road information marker is determined based on at least one of the shape, position and reflectance of said marker candidate. 3. The road information marker recognition device according to claim 1 or 2, wherein said active sensor is LiDAR.

Images