JP7613278B2 - Road information sign recognition device - Google Patents

Description

The present invention relates to a road sign recognition device.

The road sign recognition device is also called TSR [Traffic Sign Recognition], and as a technology related to the road sign recognition device, for example, the road sign judgment device described in Patent Document 1 is known. The road sign judgment device described in Patent Document 1 includes an image recognition unit that detects a first road regulation sign by performing image recognition processing on an image of the front of the vehicle captured by an imaging unit, a map information acquisition unit that acquires a road link including a second road regulation sign at the vehicle position from a map data storage unit, and a road sign judgment unit that determines the final road regulation sign on the current road link by using at least one of the current first road regulation sign and the second road regulation sign detected on the current road link when the road link changes.

JP 2015-102898 A

However, road information markers (such as road signs, billboards, and traffic lights) are easily installed, relocated, and removed, and therefore the information on them tends to be stale in map information. Therefore, the above technology that recognizes road information on road information markers based on map information may have poor recognition accuracy.

The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a road information sign recognition device that can accurately recognize road information of road information signs.

A road information sign recognition device according to one aspect of the present invention includes a camera that captures at least a portion of the area around the vehicle, an active sensor that includes at least a portion of the imaging range of the camera in its detection range, and a processing device that, when it is determined that a sign candidate detected by the active sensor is a road information sign, performs image recognition of the area corresponding to the road information sign in the image captured by the camera, and recognizes the road information of the road information sign.

This road information sign recognition device can recognize road information of road information signs by effectively using a camera and an active sensor in cooperation with each other without using map information that includes information about the road information signs. Even if a road information sign is newly installed, relocated, or removed, it is possible to recognize road information corresponding to the installation, relocation, or removal. Therefore, the road information sign recognition device according to one aspect of the present invention makes it possible to accurately recognize road information of road information signs.

In a road information sign recognition device according to one aspect of the present invention, the processing device may extract sign candidates from the detection results of the active sensor, and determine whether or not the sign candidates are road information signs based on at least one of the shape, position, and reflectance of the sign candidates. In this case, whether or not the sign candidates detected by the active sensor are road information signs can be determined using at least one of the shape, position, and reflectance of the sign candidates.

In a road information sign recognition device according to one aspect of the present invention, the active sensor may be a LiDAR. In this case, the LiDAR can be used as an active sensor to accurately recognize road information of road information signs.

The present invention makes it possible to provide a road information sign recognition device that can accurately recognize road information of road information signs.

FIG. 1 is a block diagram showing a configuration of a road information sign recognition device according to an embodiment. FIG. 2 is a flowchart showing an example of recognition of road information of a road information sign performed by the road information sign recognition device according to the embodiment. FIG. 3 is a diagram showing an example of an image captured by the camera. FIG. 4 is a diagram showing an example of point cloud information acquired by LiDAR. FIG. 5 is a diagram showing an example of point cloud information acquired by LiDAR. FIG. 6 is a diagram showing an example of point cloud information acquired by LiDAR. FIG. 7 is a diagram showing an example of an image captured by the camera. FIG. 8 is a block diagram showing the configuration of a road information sign recognition device according to a modified example.

Embodiments of the present invention will be described below with reference to the drawings. In the following description, the same or equivalent elements will be designated by the same reference numerals, and duplicate descriptions will be omitted.

Figure 1 is a block diagram showing the configuration of a road information sign recognition device 1. As shown in Figure 1, the road information sign recognition device 1 is mounted on a vehicle 10. The road information sign recognition device 1 recognizes road information of road information signs. The vehicle 10 is not particularly limited, but here it is assumed to be a vehicle with limited autonomous driving capabilities, excluding fully autonomous driving.

Road information signs include, for example, signs, billboards, and traffic lights. There are no particular limitations on road information signs, and they include any sign that indicates road information. From the viewpoint of recognition effectiveness, the road information signs targeted by this embodiment are, for example, road signs that are circular and inverted triangular in shape. The road information sign recognition device 1 includes a camera 2, a LiDAR [Light Detection and Ranging: LiDAR] 3, an information processing ECU [Electronic Control Unit] 4, and an HMI [Human Machine Interface] 5.

The camera 2 is an imaging device that captures images of the external environment of the vehicle 10. The camera 2 is provided, for example, on the back side of the windshield of the vehicle 10, and captures images in front of the vehicle 10. The camera 2 may be provided on the back and sides of the vehicle 10. The camera 2 captures images of the area in front of the vehicle, and transmits the captured images to the information processing ECU 4. The camera 2 may be a monocular camera or a stereo camera. There are no particular limitations on the camera 2, and various cameras can be used. It is sufficient that the camera 2 captures images of at least a portion of the area around the vehicle 10.

The LiDAR3 is an active sensor that uses light to detect objects around the vehicle 10. The LiDAR3 detects objects by outputting light for detection to the surroundings of the vehicle 10 and receiving light reflected by objects around the vehicle 10. The LiDAR3 here is mounted facing the front of the vehicle and detects objects in front of the vehicle. The mounting direction of the LiDAR3 is not particularly limited, and it may be mounted facing the rear, left or right side of the vehicle. The objects are, for example, road information signs, structures, and curbs on the road. The detection range of the LiDAR3 includes at least a portion of the imaging range of the camera 2. In other words, the detection range of the LiDAR3 includes the imaging range of the camera 2, or the detection range of the LiDAR3 is included in the imaging range of the camera 2.

The LiDAR3 acquires point cloud information relating to a three-dimensional point cloud in front of the vehicle, and transmits the acquired point cloud information to the information processing ECU4. The point cloud information is information that includes at least a plurality of points (position data) that make up an object. The point cloud information is information that includes three-dimensional distance and reflectance for each point. The camera 2 and the LiDAR3 are aligned (calibrated), and acquire images and point cloud information in front of the vehicle at the same fixed period and in synchronization. The LiDAR3 may be a flash type that acquires a point cloud in front of the vehicle at once, or a rotation type that performs a 360-degree scan while rotating.

The information processing ECU 4 is an electronic control unit having a processor such as a CPU [Central Processing Unit] and memories such as a ROM [Read Only Memory] and a RAM [Random Access Memory]. In the information processing ECU 4, for example, a program recorded in the ROM is loaded into the RAM, and the program loaded into the RAM is executed by the CPU, thereby realizing various functions. The information processing ECU 4 may be composed of multiple electronic control units. The information processing ECU 4 constitutes a processing device.

The information processing ECU 4 extracts sign candidates from the detection results (point cloud information) of the LiDAR 3. Various known methods can be used as the extraction method. The information processing ECU 4 recognizes at least one of the shape, position, and reflectance of the extracted sign candidate, and determines whether the sign candidate is a road information sign based on at least one of the shape, position, and reflectance of the sign candidate. If the information processing ECU 4 determines that the sign candidate is a road information sign, it performs image recognition of the range corresponding to the road information sign in the image captured by the camera 2, and recognizes the road information of the road information sign. The information processing ECU 4 notifies the recognized road information to the occupants (including the driver) via the HMI 5. Details of the processing by the information processing ECU 4 will be described later.

The HMI 5 is an interface for inputting and outputting information to and from the occupant. The HMI 5 has, for example, a display unit and a speaker. The HMI 5 outputs images from the display unit and audio from the speaker in response to a control signal from the information processing ECU 4. The HMI 5 may also be equipped with a HUD [Head Up Display]. The HMI 5 notifies the occupant of road information about road information signs recognized by the information processing ECU 4, for example, by outputting images from the display unit and outputting audio from the speaker.

Next, an example of road information recognition performed by the road information sign recognition device 1 will be described with reference to the flowchart in FIG. 2.

When the vehicle 10 starts traveling, for example, the road information sign recognition device 1 executes the following road information recognition process. That is, an image G (see FIG. 3) of the area in front of the vehicle is acquired by the camera 2, and point cloud information T (see FIG. 4) of the area in front of the vehicle is acquired by the LiDAR 3 (step S1). In the point cloud information T in FIG. 4, the point clouds on the left and right are reflected by the curbstones, and the center line is not visible because it is a white line. In the point cloud information T in FIG. 4, the point cloud reflected by the road surface has been removed in advance.

Next, the information processing ECU 4 detects left and right lane markers in the image G in front of the vehicle, and identifies the range in which the road information sign is likely to exist (usually the range on the sidewalk closer to the roadway) based on the detection result of the lane markers (step S2). The information processing ECU 4 sets the range in which the identified road information sign is likely to exist (the dotted range W1 in FIG. 5) in the point cloud information, and extracts a point cloud of candidate signs that are located in the range in which the road information sign is likely to exist and are candidates that are likely to be road information signs (step S3). In the example of FIG. 6, the point cloud of the candidate signs is the point cloud within the dotted rectangular range W2.

Then, the information processing ECU 4 judges whether the extracted sign candidate is a road information sign (step S4). In the above step S4, the extracted sign candidate is evaluated using its shape, position (height position), and reflectance. For example, the information processing ECU 4 stores in advance a data table in which the shape, position, and reflectance are associated with each type of road information sign. In the above step S4, the extracted sign candidate can be scored and evaluated using the data table to be judged. The shape of the sign candidate can be recognized using known image processing such as pattern matching. The position of the sign candidate can be based on, for example, the center position, upper end position, lower end position, left end position, or right end position of the sign candidate. The reflectance of the sign candidate can be based on, for example, the maximum value, minimum value, or average value of the reflectance of each point group that constitutes the sign candidate.

In this embodiment, the road information sign is a speed sign 9, and so the road information sign is designed to be circular in shape and have a diameter within a certain range (several tens of centimeters to just under 1.5 meters). The installation height of the speed sign 9 is within a range of approximately 1 to 2 meters. The speed sign 9 has tiny glass beads or the like that are invisible to the naked eye on its surface to improve the visibility of the sign by the headlights, so the reflectance of the speed sign is stable and above a certain level. Therefore, in step S4 above, the extracted sign candidate is determined to be a road information sign because its shape is circular within a certain range, its position is in the range of 1 to 2 meters, and its reflectance is above a certain level.

If it is determined that a road information sign is present, the information processing ECU 4 cuts out an area (a dotted rectangular area W3 in FIG. 7) corresponding to the road information sign in the image G ahead of the vehicle (YES in step S5, step S6). The area cut out from the image G ahead of the vehicle is pre-processed (tilt correction, enlargement/reduction, etc.) and image recognition is performed, and the road information of the road information sign is recognized (step S7). In this example, it is recognized that the road information sign is a speed sign and the speed limit value is "60 km/h". The method of pre-processing and image processing performed in the above step S7 is not particularly limited, and various known processes may be used.

The HMI 5 notifies the occupant of road information for the recognized road information sign (step S8). After step S8 or if step S5 is NO, if the vehicle 10 is not stopped, the process returns to step S1, whereas if the vehicle 10 is stopped, the image recognition process ends (step S9).

As described above, the road information sign recognition device 1 can recognize road information of road information signs by effectively cooperating the camera 2 and the LiDAR 3 without using map information including information about the road information signs. Even when a road information sign is newly installed, relocated, or removed, it is possible to recognize the corresponding road information. Therefore, the road information sign recognition device 1 makes it possible to accurately recognize the road information of the road information sign. This reduces the need to prepare map information on the vehicle 10 side and record the positions of the road information signs there in advance. Robust recognition is possible even when a road information sign is newly installed, relocated, or removed (disappeared).

In the road information sign recognition device 1, the information processing ECU 4 extracts sign candidates from the detection results of the LiDAR 3 and determines whether the sign candidates are road information signs based on at least one of the shape, position, and reflectance of the sign candidates. In this case, it is possible to determine whether the sign candidates detected by the LiDAR 3 are road information signs by using at least one of the shape, position, and reflectance of the sign candidates.

The road information sign recognition device 1 is equipped with a LiDAR 3 as an active sensor. This makes it possible to use the LiDAR 3 as an active sensor to accurately recognize road information of road information signs.

Although various exemplary embodiments have been described above, various omissions, substitutions, and modifications may be made without being limited to the above-described exemplary embodiments.

Figure 8 is a block diagram showing the configuration of a road information sign recognition device 101 according to a modified example. The road information sign recognition device 101 according to the modified example further includes a GNSS [Global Navigation Satellite System] receiver 6, an internal sensor 7, and a map database 8 in addition to the above-mentioned road information sign recognition device 1 (see Figure 1).

The GNSS receiver 6 constitutes a satellite positioning device that measures the position of the vehicle 10 (e.g., the latitude and longitude of the vehicle 10) by receiving signals from four or more GNSS satellites. The GNSS receiver 6 transmits the measured position information of the vehicle 10 to the information processing ECU 4. The internal sensor 7 includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the vehicle 10. The acceleration sensor is a detector that detects the acceleration of the vehicle 10. The yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) around the vertical axis of the center of gravity of the vehicle 10. The internal sensor 7 transmits the detection results to the information processing ECU 4.

The map database 8 is a database that records map information. The map database 8 is formed, for example, in a recording device such as a HDD [Hard Disk Drive] mounted on the vehicle 10. The map information includes, for example, road position information, road shape information (e.g., number of lanes, lane width, curvature information), intersection and branch point position information, etc. The map database 8 may be configured in a server that can communicate with the vehicle 10.

The information processing ECU 4 of this road information sign recognition device 101 can estimate the positions of the left and right lane markers in the image G in front of the vehicle by complementing them using the GNSS receiver 6, internal sensor 7, and map database 8. Therefore, even if the lane markers are difficult to identify from the camera 2 due to blurring or the like, the location where the road information sign exists (the range where the road information sign exists (the range W1 indicated by the dotted line in Figure 5)) can be accurately estimated, and more stable operation can be expected.

In the above embodiment, LiDAR3 is used as the active sensor, but the active sensor is not limited, and other active sensors may be used. In the above embodiment, image recognition using machine learning may be used. In the above embodiment, for example, when recognizing road information signs such as speed limit signs, simple pattern matching may be used because the format is known and the values to be displayed are limited.

In the above embodiment, an example was described in which one road information sign (marker candidate) exists in the image G and the point cloud information T, but the number of road information signs existing in the image G and the point cloud information T is not limited and may be multiple. When multiple road information signs of different types exist in the image G and the point cloud information T, road information for each of them may be recognized and notified to the occupant. On the other hand, when multiple road information signs of the same type exist in the image G and the point cloud information T, road information for any one of them (for example, the one closest to the vehicle 10) may be recognized and notified to the occupant.

1...Road information sign recognition device, 2...Camera, 3...LiDAR (active sensor), 4...Information processing ECU (processing device), 9...Speed sign (road information sign), 10...Vehicle, G...Image, T...Point cloud information.

Claims (3)

a camera that captures an image of at least a portion of an area around the vehicle;
an active sensor having a detection range that includes at least a part of the imaging range of the camera;
a processing device that, when it is determined that the sign candidate detected by the active sensor is a road information sign, performs image recognition on a range corresponding to the road information sign in the image captured by the camera, and recognizes road information of the road information sign ;
The processing device includes:
extracting sign candidates from the detection results of the active sensor;
determining whether the sign candidate is a road information sign based on at least one of a shape, a position, and a reflectance of the sign candidate;
The processing device includes:
Detecting left and right lane markers in the image, and specifying an area on the sidewalk as an area in which the road information sign is present based on the detection result of the lane markers;
setting an existence range of the identified road information sign in point cloud information relating to a three-dimensional point cloud in front of the vehicle detected by the active sensor, and extracting the sign candidate located within the existence range of the road information sign;
and a road information sign recognition device that determines whether the extracted sign candidate is the road information sign based on at least one of the shape, position, and reflectance of the sign candidate. The processing device includes:
2. The road information sign recognition device according to claim 1, wherein, when it is determined that a plurality of road information signs of the same type are present in the point cloud information, a corresponding range in the image for any one of the road information signs is image-recognized, and the road information of one of the road information signs is recognized. The road information sign recognition device according to claim 1 or 2, wherein the active sensor is a LiDAR.

Images