JP2570378B2 - Vehicle sign recognition device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a vehicle sign recognition device for recognizing a sign such as a traffic sign or a traffic signal from an image outside the vehicle around the vehicle.

[Prior Art] In order to enhance the safety of traveling of a vehicle, an image of a field of view in the front-rear direction of the vehicle is imaged using an imaging means such as a vehicle-mounted television camera.
Several devices that take out and use various information included in the image have been proposed. For example, JP-A-59-127200
Japanese Patent Application Laid-Open Publication No. H11-209,086 discloses an "in-vehicle traffic sign notification device" that recognizes road recognition from an image. Also, there has been proposed a device for recognizing a traffic sign or a traffic light by providing a transmitter for emitting a modulated light or the like for identification in a traffic light or the like, and mounting a device for identifying the signal on a vehicle (for example, Japanese Patent Application Laid-Open No. 62-95698). Publication "Automatic identification device for traffic signs" etc.).

[Problems to be Solved by the Invention] However, since traffic signs, traffic lights, and the like do not always have a fixed positional relationship with a running vehicle, it is difficult to directly recognize traffic signs and the like. There was a problem that is. In addition, in a configuration for recognizing a signal transmitted from a traffic sign, a traffic signal, or the like, a means for transmitting a signal for each sign must be attached in addition to a device mounted on a vehicle.
In addition, it is extremely difficult to realize this because the signal must be transmitted to all vehicles within a certain range while preventing interference. Further, when the transmitter fails, there is a problem that the influence is exerted on all vehicles passing through an intersection or the like where the traffic light is installed.

An object of the present invention is to solve the above problems and to appropriately recognize a sign such as a traffic sign or a traffic light from an image outside the vehicle.

Configuration of the Invention The configuration of the present invention that achieves the above object will be described below.

[Means for Solving the Problems] As shown in FIG. 1, the vehicle sign recognition device of the present invention has an image pickup means M2 for picking up an image of the outside of a vehicle around a vehicle M1, and an image picked up by the image pickup means M2. A road area recognizing means M3 for recognizing a road area from the image, searching a search area having a predetermined positional relationship with respect to the recognized road area,
And a sign recognition means M4 for recognizing a sign such as a traffic light.

Here, the means M3 for recognizing the road area is for recognizing the road area from the captured image. For example, the means M3 performs predetermined image processing or the like on the captured image and performs some processing on the image. Various types of image recognition, such as a configuration for extracting a region corresponding to a road region from among these regions and a configuration for storing the color of a region designated as a road and tracking the region of that color, etc. It can be configured to implement the technique.

The sign recognition means M4 searches for a traffic sign in the search area,
It is for recognizing a sign such as a traffic signal, and for example, can be configured to perform recognition based on a traffic sign or a shape or color of a traffic signal which is predetermined as a recognition target.

[Operation] The vehicle road recognition device of the present invention having the above-described configuration recognizes a road area by the road area recognition means M3 from an image outside the vehicle around the vehicle M1 captured by the imaging means M2. Then, a search area having another positional relationship is searched in advance, and the sign recognition means M4 recognizes a traffic sign, a traffic signal, or the like related to the traveling existing in this area. As described above, the vehicle sign recognition device of the present invention sets the search area for the sign to be recognized as having a specific positional relationship with the road area, and performs the sign recognition in this search area. Therefore, the recognition of the sign is realized with high accuracy and easily.

Embodiment In order to further clarify the configuration and operation of the present invention described above, a preferred embodiment of the vehicle sign recognition device of the present invention will be described below. FIG. 2 is a schematic configuration diagram of a vehicle control device incorporating a vehicle sign recognition device as a first embodiment of the present invention, and FIG. 3 is a block diagram showing a schematic configuration mainly of an electronic control device.

As shown in the figure, the control device for a vehicle includes a full-color television camera 2 using a CCD for imaging the front of the vehicle 1.
An electronic control unit 3 that performs processing of an image captured by the television camera 2 and processing of an alarm output, a vehicle speed sensor 4 that detects a vehicle speed, and a brake sensor 5 that detects an amount of depression of a brake pedal. It is composed of a color monitor 7 for displaying the image after the image is visible to the occupant, and an alarm device 10 for outputting an alarm sound.

The electronic control unit 3 is configured as an arithmetic and logic operation circuit including a well-known CPU 11, a ROM 12, a RAM 13, an image memory (hereinafter, referred to as a VRAM) 15, and the like. In addition to the CPU 11 and the like, the electronic control unit 3 receives image signals input ports 18, an input circuit 19, a CRTC 20, an output circuit 21, which are interconnected via a bus 17, and RGB signals from the television camera 2. An A / D converter 22 for converting a digital signal for each color is provided.

The input circuit 19 is connected to a vehicle speed sensor 4, a brake sensor 5, and a confirmation switch 25. This switch 25
Is used when a final determination by the occupant is required in a road recognition process described later. CPU 11 has an input circuit 19
, It is possible to read at any time whether or not the switch 25 has been operated, together with the vehicle speed V and the brake operation amount B.

The image input port 18 is connected to the A / D converter 22, and sequentially reads image signals from the TV camera 2,
It is developed in a predetermined area of VRAM15. Therefore, the CPU 11 can freely process the image outside the vehicle by accessing the predetermined address of the VRAM 15. The image processed in this way is displayed on the final color monitor 7, and the display on the color monitor 7 is performed by the CRTC 20. Upon receiving an instruction from the CPU 11, the CPTC 20 reads out image data after image processing developed in a predetermined area in the VRAM 15, and displays this on the color monitor 7.

The alarm device 10 is connected to the output circuit 21. The alarm device 10 is a device that receives and receives an external trigger signal and reproduces and outputs one of several pre-registered sounds.
Therefore, the CPU 11 can output a predetermined trigger signal to the alarm device 10 via the output circuit 21 to notify a vehicle occupant of an alarm sound or the like.

The sign recognition process and the warning process in the vehicle control device of the present embodiment having the above-described configuration will be described. The apparatus of this embodiment executes a sign recognition / warning processing routine shown in FIG. After turning on the power, first, the image signal input port
A process for inputting an image outside the vehicle captured by the television camera 2 is performed by 18 (step 100). The input image data is developed in a predetermined area of the VRAM 15 as described above. An example of the captured image is shown in FIG. Subsequently, the captured image data is binarized by applying a well-known spatial filter such as a Laplacian operator, and a process of extracting an edge from the captured image is performed (step 110). By edge extraction, the image is binarized into edges and other parts. After edge extraction, edge enlargement / reduction processing is performed to perform processing for connecting portions where edges that should have occurred are missing in the process of capturing images (step 120). FIG. 5B shows an example of the result of such edge extraction and processing.

After the edge is repaired, a process of extracting the portion surrounded by the edge as one region is performed (step 130 in FIG. 4). The extraction of the area surrounded by the edge is performed in order from the area existing at the bottom of the screen in consideration of the probability of the existence of the road, and by tracing the edge clockwise, the area of the area is extracted. The calculation is also performed. Subsequently, it is determined whether or not the extracted region corresponds to a road. In this embodiment, regarding the extracted area, the area calculated in step 130 is equal to or more than 1/10 of the entire screen (step 140), and the position where the area exists is a position corresponding to the road. When it is determined (step 150), it is determined that the road corresponds to the area. The area occupied by the road with respect to the entire screen depends on which part of the vehicle 1 is provided with the television camera 2 and its elevation angle, magnification, change in road gradient, and the like. Determined experimentally. In this embodiment, the determination of the position of the region is made based on whether or not the region exists from the bottom to the center of the screen. This is because, when the television camera 2 is provided at the front of the vehicle 1, the road is usually imaged at this position.

If the extracted area does not correspond to a road, it is determined whether all the areas existing on the screen have been extracted and determined (step
160) If the determination of all the areas has not been completed, the process returns to step 130 to continue the determination of the next area. On the other hand, when the determination for all the areas is completed (step 160), it is determined that no area corresponding to the road has been found, and the above-described processing is executed again from the input of a new image (step 100).

When it is determined that the extracted area is a road by the processing of steps 130 to 150, processing for displaying this on the color monitor 7 is performed (step 170). In this case, if the area extracted as a road is subjected to a process with high visibility (for example, yellow color or blinking of the area) and displayed superimposed on the captured original image, this can be recognized at a glance. It is preferred. This example is shown in FIG. 5 (C). In the figure, the hatched area is colored yellow on the color monitor 7.

In this manner, it is determined whether the area shown on the color monitor 7 can be finally determined as the road area by operating the confirmation switch 25 (step 180). If the confirmation switch 25 is operated within a certain period, the area displayed on the color monitor 7 may be determined to be a road. The attributes of this area, in this embodiment, the road width L and its color, are stored in the RAM 13. (Step 190).

With the above processing, it is determined that the initial recognition of the road has been completed, and the recognition of the road using the attributes of the road area is continued in step 200 and subsequent steps. First, similarly to steps 100, 110, and 120 in the initial recognition processing, input of an image captured by the television camera 2 (step 200) and extraction of edges (step 21)
0), repair the edge (step 220), and then RA
The detection point is set using the road width L which is one of the attributes stored in M13 (step 230). The detection point SP is set as a half point of the road width L as shown in FIGS. 6 (A) and 6 (B). Although the shape of a road changes when it approaches a curve, the width at the bottom of the screen does not change significantly.
Is set, and the extraction of the area is started from the area where the detection point SP exists. If the extracted area cannot be recognized as a road, the detected point SP is successively moved to an upper area and a new area is extracted. When the detected point SP reaches the top of the screen, ( (Step 240), assuming that the road area cannot be recognized in the image input this time, the above-described processing is repeated again from the input of the image (Step 200).

If the detection point SP does not reach the upper part of the screen, an area where the detection point SP exists is extracted (step 250), and it is determined whether or not the area of the area is 1/10 or more of the entire screen (step 250). Step 260). If the area of the area is equal to or smaller than the size corresponding to the road, the above-described processing is repeated from the setting of the detection point SP (step 230).

If the area of the extracted area occupies 1/10 or more of the screen, the color which is one of the attributes of the area is determined (step 270). If the color of the area can be regarded as the same as the color of the road stored in the RAM 13 in step 190, it is determined that the area is a road, and it is determined whether or not this area is continuous to the center of the screen (continuous). Sex determination) (step 280). If the continuity is satisfied, the area may be recognized as a road area, and the process proceeds to step 400 and subsequent steps described later. An example of the road area thus recognized is shown in FIG.

When imaging on a real road, an edge is generated by the shadow of an overpass, a building, a telephone pole, or the like reflected on the road, and the road area is divided, or an edge is generated by a white line such as a pedestrian crossing or a center line. The area may be divided. Such an example is shown in FIG.

When the road area is divided, even the area determined to have the area and the color corresponding to the road may not be continuous to the center of the screen. Therefore, when it is determined that there is no continuity (step 280), the detection point SP is moved to the upper or lower part of the divided area to extract the area (step 300), and the area and the road ahead are extracted. To compare the color with the area determined to be
310). If the colors of the two areas can be regarded as the same, the two areas are combined (step 320), and the continuity is determined again (step 280). If the area is not continuous to the center of the screen even after combining, it is further determined whether the area above and below the area cannot be regarded as the same color. Repeat until the sex determination is satisfied (steps 280, 300, 310, 320). If there is no region of the same color without the continuity being satisfied, it is determined that no region corresponding to the road has been found in the image input this time, and the image input processing (step 20) is repeated.
From step 0), the above-described processing (steps 200 to 320) is repeated.

If the continuity is satisfied (step 280), it is determined that the area is a road, and it is determined whether or not the recognized road includes an intersection based on the obtained shape of the road area (step 280). 400). As shown in FIG. 7 (A),
When an intersection is included, the road area GR does not have a monotonous shape as illustrated in FIG. 5 (D) but has a unique shape having an area extending from a central area.
Therefore, the intersection is discriminated by paying attention to this shape, and if the intersection is included, a process of recognizing the signal line is performed based on the position of the intersection (step 410). In addition, as shown in FIGS. 8A and 8B, it is also possible to recognize a crosswalk and determine an intersection. The traffic light is
Since the probability of existence at a specific position with respect to an intersection is extremely high, a search area GS is set diagonally above the intersection, as shown in FIG. 7 (B) or FIG. 8 (B). An object having the shape and color of the traffic light is specified in the search area.

After thus recognizing the traffic light, it is determined whether the traffic light is a red light (step 420). When it is determined that the color of the lit lamp among the three lamps of the traffic light is red, the signals from the vehicle speed sensor 4 and the brake sensor 5 are read through the input circuit 19, and the vehicle speed V and the brake operation are read. The quantity B is input (step 430). Based on the vehicle speed V and the brake operation amount B thus input, it is determined whether or not an alarm is required (step 440). If the signal is a red light, the vehicle speed V is equal to or higher than a predetermined value. If the brake is not operated, it is determined that the driving vehicle may not be aware of the traffic signal, and the alarm device 10 is operated via the output circuit 21 (step 450). The alarm device 10 uses a built-in voice synthesizing device to notify a voice such as "Red light. Please reduce the speed." Since the distance to the traffic light can be known from the horizon on the image from the height of the traffic light, the size of the signal lamp on the image, and the like, it may be possible to detect this and control the alarm output. If the vehicle speed V does not decrease due to the alarm, vehicle speed reduction control such as reducing the output of the internal combustion engine or automatically applying a brake may be performed.

After activating the alarm device 10 in this way, or if it is not an intersection (step 400) or if the signal is not a red light (step 420) or if no alarm is required (step 440)
Then, the process returns to step 200 described above, and the above-described processing is repeated.

According to the vehicle control device incorporating the vehicle sign recognition device of the present embodiment configured as described above, the road is continuously and accurately recognized based on the color of the area determined to be the road by the initial processing. The traffic signal can be recognized very quickly, easily and accurately by detecting an intersection included in the road area and searching for an area set for the intersection. That is, since the search area GS is set for the intersection and recognition is performed, it is possible to improve the accuracy of signal recognition and shorten the recognition time without erroneously recognizing a tail lamp, a neon sign, or the like of the preceding vehicle. It is. As a result, an alarm can be output from the alarm device 10 using the recognized traffic light, and safety in driving the vehicle can be further improved.

Next, a second embodiment of the present invention will be described. The vehicle control device incorporating the vehicle sign recognition device of the second embodiment has the same hardware configuration as that of the first embodiment (see FIGS. 2 and 3). In the second embodiment, as shown in the flowchart of FIG.
An initial recognition process corresponding to 0 to step 180 is executed (see FIG. 4). Further, as attributes of the area finally determined to be a road, the roughness of the area is stored in addition to the color (step 185). The roughness of a region is a so-called texture in image processing. In the present embodiment, the density of a region is defined as a ratio of black pixels when the density of the region is represented by a binarized pixel distribution (dither method). .

After the initial recognition of the area corresponding to the road and the storage of its attributes are performed in this way, the flag FR for red signal recognition used in the processing described later is reset to a value 0, and the input is performed in the same manner as in the first embodiment. Processing such as extracting an area from the obtained image is performed, and processing for recognizing an area that may correspond to a road is performed (steps 200 to 270). afterwards,
The vehicle speed signal from the vehicle speed sensor 4 is read to determine whether or not the vehicle 1 is running (step 275). If the vehicle is running, as in the first embodiment, processing such as determination of continuity of the area and combination / correction of the area based on color and roughness are performed (steps 280 to 320).

When the continuity condition is satisfied by such processing, determination is made as to whether or not an intersection is included, and determination and processing of an alarm output based on the vehicle speed V and the brake operation amount B are performed as in the first embodiment. (Steps 400-450). In this embodiment, in addition to the processing of the first embodiment, step 420
If the red signal is recognized, the flag FR is set to 1 (step 423). If the red signal is not recognized, a process of resetting to 0 is performed (step 426). ).

According to the above-described processing, when the intersection is recognized during traveling and the traffic light is a red light, the flag FR is set to 1 and the vehicle speed V is set based on the brake operation amount B according to the first embodiment. Similarly, if an alarm is required, an alarm
Processing for driving 10 is performed. Further, in this embodiment,
If the vehicle 1 is stopped by the red light, step 275
Is "NO", and the processing of step 500 and thereafter is executed.

First, it is determined whether or not the flag FR has a value of 1 (step 500). If the value is 1, that is, if the signal recognized by the immediately preceding process is red, a process of recognizing a traffic signal from an input image is performed. (Step 510). This processing is the same as the recognition of the signal during traveling (step 410), but when the vehicle is stopped, the continuity processing (steps 280 to 280) is performed because the inter-vehicle distance is small and continuity is not likely to be established. Recognition is performed from the image before performing 320). Therefore, the search area GS of the traffic light is the search area determined at the time when the continuity was established before that and the area around the search area.

This area is searched to determine whether or not the traffic signal has been recognized (step 520). If the traffic signal has been recognized, the traffic signal further determines whether or not it is a green light (step 530). If it is a green signal, it can be determined that the red signal has been changed to a green signal, so the alarm device 10 is driven (step 54).
0), and a process of resetting the flag FR to the value 0 is performed (step 550). Since the alarm device 10 can store a plurality of types of sounds, unlike step 450, the alarm device 10 outputs a sound such as "the signal has turned green" here.

After resetting the flag FR, or when at least one of the above conditions is not satisfied (steps 500, 520,
530), the processing is executed again from the recognition processing of the road area (step 200).

As in the first embodiment, the vehicle control device of the present embodiment configured as described above can recognize a traffic light and output a warning before an intersection as necessary, and further stop at an intersection to stop. When the traffic light changes to a green light, the occupant can be notified of the change. Therefore, the occupant does not need to constantly monitor the traffic light while the vehicle is stopped, and can perform other operations, such as checking a map and cleaning glasses, calmly. As a result, the safety of driving the vehicle can be improved.

Although several embodiments of the present invention have been described above,
The present invention is not limited to such embodiments at all, for example, a configuration in which the recognition of a road area is performed by a reflection plate or the like installed on the road, a configuration using other features obtained by image processing as attributes of the road, Alternatively, it is needless to say that the present invention can be implemented in various modes without departing from the gist of the present invention, such as a configuration for recognizing a traffic sign indicating a speed limit instead of a traffic light.

Effects of the Invention As described in detail above, according to the vehicle sign recognition device of the present invention, a search area for recognizing a sign using a road area is set, so that a traffic light, a traffic sign, or the like can be extremely accurately and easily. It has an excellent effect that it is possible to recognize the sign of. Further, since there is no need to install a transmitter or the like on the side of a traffic light or a traffic sign, it becomes easy to actually mount a sign recognition device on a vehicle. Therefore, it is possible to facilitate and improve the accuracy of the processing of another control device mounted on the vehicle, for example, a device that issues a warning based on the recognized signal and the vehicle speed. The realization of such a device can contribute to the improvement of driving safety of a vehicle and is extremely useful.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating the basic configuration of the present invention, FIG. 2 is a schematic configuration diagram of a vehicle control device incorporating a vehicle sign recognition device as a first embodiment of the present invention, and FIG. FIG. 4 is a flow chart showing a road recognition / warning process routine executed by the electronic control unit of the first embodiment, and FIGS. 5 (A), (B), (C), (D), and FIG.
(E) is an explanatory view exemplifying an image processed by the road recognition processing, FIGS. 6 (A) and (B) are explanatory views showing the state of detection point SP setting, and FIGS. 7 (A) and (B) ) Is an explanatory diagram showing how an intersection is recognized and a search area GS is set, FIGS. 8A and 8B are explanatory diagrams showing how a traffic light is recognized from an actual image example, and FIG. FIG. 9 is a flowchart showing a signal recognition / warning processing routine according to the second embodiment;
It is. DESCRIPTION OF SYMBOLS 1 ... Vehicle 2 ... TV camera 3 ... Electronic control device 4 ... Vehicle speed sensor 5 ... Brake sensor 7 ... Color monitor 10 ... Alarm device 15 ... VRAM SP ... Detection point GS ... Search area

Claims (1)

(57) [Claims] An image pickup means for picking up an image of the outside of a vehicle around a vehicle, a road area recognizing means for recognizing a road area from an image picked up by the image pickup means, and a predetermined positional relationship with respect to the recognized road area. A sign recognition device for a vehicle, comprising: sign recognition means for searching in a search area having a sign for recognizing a sign such as a traffic sign or a traffic signal involved in traveling.