JP7647676B2 - Traffic light recognition device - Google Patents

Description

The present invention relates to a traffic light recognition device.

A signal recognition system, such as that described in Patent Document 1, is known in the past. In the signal recognition system described in Patent Document 1, signal detection information of a target traffic light around a vehicle is obtained based on camera image information from a camera mounted on the vehicle. Light pattern information is obtained that indicates the relative positional relationship between the multiple light parts of the traffic light and the appearance of each of the multiple light parts when lit. The light state of the target traffic light is recognized by comparing the signal detection information with the light pattern information.

JP 2021-2275 A

Unlike lights other than arrows (e.g., circular red, blue, or yellow lights), blue light arrows can be recognized not only based on whether the light is on, but also on the direction the arrow is pointing. In the technical fields of driving assistance and autonomous driving systems related to advanced safety technologies, there is a demand for more accurate recognition of the direction the arrow is pointing, including traffic lights located far away from the vehicle in dark environments such as at night.

A traffic light recognition device according to one aspect of the present invention includes a lighting state recognition unit that recognizes the lighting state of the light parts of traffic lights around a vehicle based on an image captured by an on-board camera, and an arrow direction recognition unit that recognizes the direction of a blue light arrow at a traffic light based on the captured image and the lighting state. When both a first light part corresponding to a light other than the arrow and a second light part corresponding to the blue light arrow are on, the lighting state recognition unit recognizes the lighting state of the first light part and the second light part being simultaneously on, and when simultaneous lighting is recognized by the lighting state recognition unit, the arrow direction recognition unit recognizes the direction of the blue light arrow based on the comparison result between the luminance distribution of the first flare generated in the captured image with the first light part as a light source and the luminance distribution of the second flare generated in the captured image with the second light part as a light source.

In a traffic light recognition device according to one aspect of the present invention, when the lighting state recognition unit recognizes simultaneous lighting, for example, in a dark environment such as at night, there is a high possibility that a first flare and a second flare are occurring in the captured image. The first flare occurs in the captured image with the first light portion corresponding to the light other than the arrow as a light source, so the luminance distribution on the captured image is concentric with the first light portion. The second flare occurs in the captured image with the second light portion corresponding to the blue light arrow as a light source, so the luminance distribution on the captured image is biased with respect to the concentric circle corresponding to the second light portion. Therefore, when the lighting state recognition unit recognizes simultaneous lighting, the direction indicated by the blue light arrow can be more accurately recognized based on the bias in the luminance distribution of the second flare based on the luminance distribution of the first flare. This makes it possible to more accurately recognize the direction indicated by the arrow, including traffic lights located far from the vehicle in a dark environment such as at night.

In one embodiment, the arrow direction recognition unit does not need to recognize the direction of the blue light arrow when it is determined that blurring due to vehicle shaking is occurring based on the captured image and the detection results of the vehicle's acceleration sensor. In this case, it is possible to prevent a decrease in the accuracy of recognition of the direction the arrow is pointing in a situation where motion blurring due to vehicle shaking is occurring in the captured image.

The traffic light recognition device according to one aspect of the present invention makes it possible to more accurately recognize the direction of the arrow, including traffic lights located far away from the vehicle in a dark environment, such as at night.

1 is a block diagram showing a schematic configuration of a traffic light recognition device according to an embodiment. FIG. 13 is a diagram showing an example of a situation in which a candidate area is estimated. FIG. 11 is a diagram showing another example of a situation in which a candidate area is estimated. FIG. 4 is a diagram showing an example of a lighting state of a first lighting portion. FIG. 4 is a diagram showing an example of the luminance distribution of a first lamp portion. FIG. 4 is a diagram showing an example of a state in which the first and second lamp portions are simultaneously turned on; FIG. 4 is a diagram showing an example of the luminance distribution of first and second lamp portions; 2 is a flowchart showing an example of processing performed by the traffic light recognition device of FIG. 1 .

The following describes an embodiment of the present invention with reference to the drawings. In the following description, the same or equivalent elements are designated by the same reference numerals, and duplicate descriptions are omitted.

Figure 1 is a block diagram showing the schematic configuration of a traffic light recognition device according to an embodiment. The traffic light recognition device 100 shown in Figure 1 is mounted on a vehicle such as a passenger car, and recognizes the illumination state of traffic lights included in an image captured by a camera of the vehicle.

As shown in FIG. 1, the traffic light recognition device 100 includes an ECU [Electronic Control Unit] 10 that provides overall control of the device. The ECU 10 is an electronic control unit that includes a CPU [Central Processing Unit], a ROM [Read Only Memory], and a RAM [Random Access Memory]. The ECU 10 loads programs stored in the ROM into the RAM and executes them with the CPU to perform various vehicle controls. The ECU 10 may be composed of multiple electronic control units.

The ECU 10 is connected to a GPS receiver 1, a camera (vehicle-mounted camera) 2, a radar sensor 3, an acceleration sensor 4, and a map database 5.

The GPS receiver 1 measures the vehicle's position (e.g., the latitude and longitude of the vehicle) by receiving signals from three or more GPS satellites. The GPS receiver 1 transmits the measured vehicle position information to the ECU 10.

Camera 2 is an imaging device that captures images of the external conditions of the vehicle. As an example, camera 2 is provided behind the windshield of the vehicle and captures images of the area in front of the vehicle. Camera 2 transmits captured images of the area in front of the vehicle to ECU 10. Camera 2 may be a monocular camera or a stereo camera. Furthermore, the location where camera 2 is attached is not limited to the back side of the windshield, as long as it is capable of capturing images of at least the area in front of the vehicle. Camera 2 is not limited to the area in front of the vehicle, and may be composed of multiple cameras that capture images of multiple directions of the vehicle.

The radar sensor 3 is a detection device that uses radio waves (e.g., millimeter waves) or light to detect objects (electric poles, traffic lights, etc.) around the vehicle. The radar sensor 3 includes, for example, millimeter wave radar or LiDAR (Light Detection and Ranging). The radar sensor 3 detects objects by transmitting radio waves or light to the vicinity of the vehicle and receiving the radio waves or light reflected by the objects. The radar sensor 3 transmits the detection results regarding the detected objects to the ECU 10.

The acceleration sensor 4 is a detector that detects the acceleration of the vehicle. The acceleration sensor 4 includes, for example, a vertical acceleration sensor that detects vertical acceleration, which is the acceleration in the up and down direction of the vehicle, and a lateral acceleration sensor that detects left and right acceleration, which is the acceleration in the left and right direction (lateral direction) of the vehicle. The acceleration sensor 4 may also include a vertical acceleration sensor that detects vertical acceleration, which is the acceleration in the front and rear direction (vertical direction) of the vehicle. The acceleration sensor 4 transmits vehicle acceleration information to the ECU 10.

The map database 5 is a database that stores map information. The map database 5 is formed, for example, in a HDD [Hard Disk Drive] mounted on the vehicle. The map information includes road position information, road shape information (e.g., curves, types of straight sections, curvature of curves, etc.), intersection and branch point position information, and traffic light position information on the map. The traffic light position information on the map is information on the position coordinates of the traffic light on the map. The map information may include surrounding structure data associated with the traffic light position information on the map. Note that the map database 5 does not necessarily need to be mounted on the vehicle, and may be formed on a server that can communicate with the vehicle.

The surrounding structure data includes position information on the map of surrounding structures that exist around the traffic light and shape information of the surrounding structures. Surrounding structures are structures that exist around the traffic light and are likely to house the traffic light. Examples of surrounding structures may include pillars, utility poles, poles, pedestrian bridges, and inner walls of tunnels. Note that the surrounding structures do not necessarily need to include the entire structure, and may include only a portion of the structure within a certain range from the traffic light. The position information of the surrounding structures on the map is information on the three-dimensional position of the surrounding structures on the map. The shape information of the surrounding structures is information on the three-dimensional shape of the surrounding structures.

Next, the functional configuration of the ECU 10 will be described. The ECU 10 has a vehicle position recognition unit 11, an object recognition unit 12, a candidate area estimation unit 13, a lighting state recognition unit 14, and an arrow direction recognition unit 15. Note that some of the functions of the ECU 10 described below may be executed by a server capable of communicating with the vehicle.

The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the position information of the GPS receiver 1 and the map information of the map database 5. The vehicle position recognition unit 11 may also recognize the position of the vehicle on the map using SLAM [Simultaneous Localization and Mapping] technology, using position information of fixed obstacles such as utility poles included in the map information of the map database 5 and the image captured by the camera 2 or the detection results of the radar sensor 3. The vehicle position recognition unit 11 may use traffic light data and surrounding structure data stored in the map database 5 in addition to the map information of the map database 5. The vehicle position recognition unit 11 may also recognize the position of the vehicle on the map using other well-known methods.

The object recognition unit 12 acquires an image of the area in front of the vehicle, including color information, based on the image captured by the camera 2. The object recognition unit 12 recognizes objects around the vehicle, for example, based on the image captured by the camera 2 and the detection results of the radar sensor 3. The object recognition unit 12 may perform object recognition by integrating the image captured by the camera 2 and the detection results of the radar sensor 3 using a well-known sensor information integration process. The object recognition unit 12 performs sensor information integration, for example, by adding color information captured by the camera 2 to three-dimensional detection points (reflection points of light or millimeter waves) detected by the radar sensor 3. The object recognition unit 12 recognizes the relative position of the object with respect to the vehicle and the shape of the object.

Note that the object recognition unit 12 does not necessarily need to use both the captured image of the camera 2 and the detection result of the radar sensor 3. The object recognition unit 12 may use only the captured image of the camera 2 for object recognition. If only the captured image of the camera 2 is used for object recognition, there is no need to include the radar sensor 3.

The candidate area estimation unit 13 estimates a candidate area based on the position of the vehicle on the map recognized by the vehicle position recognition unit 11, the recognition result of the object recognition unit 12, and the map information of the map database 5. The candidate area is an area in the captured image that is a candidate for a traffic light that faces the vehicle ahead in the direction of travel of the vehicle. The candidate area may be a search area [ROI: Region of Interest] set in the captured image. The search area is an area in the captured image where image recognition processing is performed to recognize the lighting state of the traffic light. By using the candidate area, it is possible to identify the traffic light to be recognized as a target for lighting state recognition even in a situation where light sources other than traffic lights are present around the vehicle in a dark environment such as at night.

2 is a diagram showing an example of a situation in which a candidate area is estimated. FIG. 2 is a captured image captured by a camera of a vehicle traveling on a one-lane road with a crossroad with traffic lights ahead. FIG. 3 is a diagram showing another example of a situation in which a candidate area is estimated. FIG. 3 is a captured image captured by a camera of a vehicle traveling on a one-lane road with a preceding vehicle stopped in front of a crosswalk with traffic lights ahead. The candidate area estimation unit 13 may, for example, use the machine learning results of an image including a traffic light to estimate a portion of the captured image including an object similar to the outline of a traffic light as a candidate area (area A1 in FIG. 2). For example, when the candidate area estimation unit 13 recognizes a group of red lights specific to a stopped preceding vehicle in the captured image, it may estimate a portion of the captured image including the preceding vehicle and the area above it as a candidate area (area A2 in FIG. 3). For example, the candidate area estimation unit 13 may estimate a portion of the captured image including surrounding structures as a candidate area (area A3 in FIGS. 2 and 3). The candidate area estimation unit 13 may, for example, recognize a portion of the captured image that includes a pair of traffic lights facing each other in a direction that intersects with the vehicle's traveling direction (area A4 in FIG. 2), and estimate the portion that includes the traffic light between the pair of traffic lights as the candidate area (area A1 in FIG. 2). The candidate area estimation unit 13 may, for example, recognize a portion of the captured image that corresponds to an intersection located near the vehicle position recognized by the vehicle position recognition unit 11 (for example, an intersection when the vehicle position reaches a white line and a sign a predetermined distance before the intersection ahead in the vehicle's traveling direction) as the candidate area (area A5 in FIG. 2).

The candidate area estimation unit 13 recognizes the candidate area so as to include the position of the traffic light facing the vehicle in the captured image. In the examples of Figures 2 and 3, the candidate area is set as a rectangular region. Note that the shape of the candidate area is not limited to a rectangular shape, and may be an ellipse, a circle, a polygon, or other shape. Furthermore, the method of estimating the candidate area is not limited to the above method.

The lighting state recognition unit 14 recognizes the lighting state (described below) of the light parts of the traffic lights in front of (around) the vehicle based on the image captured by the onboard camera. Generally, traffic lights have multiple light parts. A light part is a part that is turned on and off. A light part is composed of, for example, a light bulb, an LED, a light-emitting device, a display, etc.

When a candidate area is present in the captured image, the lighting state recognition unit 14 recognizes the lighting state of the traffic light in the candidate area. When a candidate area estimated by the candidate area estimation unit 13 (e.g., at least one of the examples in Figures 2 and 3 described above) is present in the captured image, the lighting state recognition unit 14 recognizes the position of the traffic light in the captured image of the camera 2 based on the result of matching the traffic light and surrounding structures extracted from the map database 5 with the recognition result of the object recognition unit 12. The lighting state recognition unit 14 can recognize the position of the traffic light in the captured image by a well-known method. The lighting state recognition unit 14 recognizes the lighting state of the traffic light by performing image recognition (e.g., pattern matching that takes color into consideration) in the candidate area. The content of the image recognition process is not particularly limited, and it may be a process that can recognize the lighting state of the traffic light.

The lighting state of a traffic light includes a state in which the first light portion, which corresponds to a light other than an arrow, is lit. The first light portion is typically a circular signal, and is the main light source of, for example, a yellow or red light. FIG. 4 is a diagram showing an example of the lighting state of the first light portion. In FIG. 4, for example, three examples (40 m, 80 m, and 120 m) of different distances from the vehicle to the signal are shown as captured images representing a lighting state in which the yellow light is lit alone. The top row of FIG. 4 is a life-size captured image, and the bottom row is an enlarged view of the first light portion in the top row.

As shown in FIG. 4, when the yellow light is turned on, a high-luminance area corresponding to the light source and a flare area spreading around the light source appear on the captured image. Flare is a light fogging phenomenon that occurs when noise light is reflected off the lens surface or the like when the lens of camera 2 is pointed at the lighted part of the traffic light. In the flare area, part of the image becomes whitish, reducing sharpness and contrast. Flare is likely to occur in images captured at night, for example. Furthermore, the greater the distance from the vehicle to the traffic light, the more image crushing occurs in the high-luminance area of the yellow light in the captured image, and the tendency is for the boundary between the high-luminance area and the flare area to become blurred.

Figure 5 shows an example of the luminance distribution of the first light portion. For ease of explanation, Figure 5 shows the color information of the 80m enlarged view of Figure 4 inverted to grayscale. As shown in Figure 5, a two-dot chain line LS and a dashed line FL are drawn.

The two-dot chain line LS defines a circular area corresponding to the area where the light source of the first light portion is present. The two-dot chain line LS may be obtained, for example, as a high-luminance area when the luminance distribution of the captured image is binarized using a predetermined light source threshold value. The shape of the two-dot chain line LS may be used, for example, to determine the direction in which the arrow of the second light portion is pointing by comparing the shapes of the second light portion and the light source described below.

The dashed line FL defines an area corresponding to the first flare that occurs in the captured image with the first light portion as the light source. In the example of FIG. 5, the flare occurs isotropically from a circular light source, and the dashed line FL circle is approximately concentric with the two-dot chain line LS circle and has a larger radius than the two-dot chain line LS circle. The dashed line FL may be obtained, for example, as a medium luminance portion when the luminance distribution of the captured image is binarized with a predetermined flare threshold value. The flare threshold value is a threshold value having a lower luminance value than the light source threshold value. The shape of the dashed line FL may be used, for example, to determine the direction indicated by the arrow of the second light portion by comparing the spread of the flare of the second light portion described below.

The lighted state of a traffic light includes a state in which the second light portion corresponding to the blue light arrow is on. The second light portion is typically an arrow signal, and is used as an auxiliary light source to supplement, for example, a red light. The second light portion has a smaller light source area than the first light portion. There is no particular restriction on the direction in which the arrow of the second light portion points. The lighted state of a traffic light also includes a simultaneous light state in which both the first light portion and the second light portion are on. Simultaneous light here means a state in which both the red light and the green arrow signal are on.

The lighting state recognition unit 14 recognizes the lighting state of the first lighting portion and the second lighting portion when both the first lighting portion corresponding to the light other than the arrow and the second lighting portion corresponding to the blue light arrow are lit. The lighting state recognition unit 14 does not recognize the lighting state of the first lighting portion and the second lighting portion when either the first lighting portion corresponding to the light other than the arrow or the second lighting portion corresponding to the blue light arrow is not lit.

Figure 6 is a diagram showing an example of a simultaneous illumination state of the first and second light portions. In Figure 6, three examples (40 m, 80 m, and 120 m) of different distances from the vehicle to the signal are shown as captured images representing a simultaneous illumination state in which, for example, a red light and a green arrow signal are illuminated at the same time. The top row of Figure 6 is a life-size captured image, and the bottom row is an enlarged view of the first and second light portions in the top row.

As shown in Figure 6, when a red light and a blue arrow signal are both illuminated at the same time, a high-brightness portion corresponding to the light source and a flare portion extending around the light source appear in the captured image for the red light and the blue arrow signal, respectively. As with the example in Figure 4, the greater the distance from the vehicle to the traffic light, the more likely it is that the high-brightness portions of the red light and the blue arrow signal in the captured image will be crushed, and the boundaries between the high-brightness portions and the flare portions will tend to become blurred. Therefore, for a blue arrow signal, the greater the distance from the vehicle to the traffic light, the more difficult it becomes to recognize the direction it is pointing.

The arrow direction recognition unit 15 recognizes the direction in which the blue light arrow is pointing at the traffic light based on the captured image and the lighting state. For example, when simultaneous lighting is recognized by the lighting state recognition unit 14, the arrow direction recognition unit 15 recognizes the direction in which the blue light arrow is pointing based on the result of comparing the luminance distribution of the first flare generated in the captured image with the first light portion as a light source and the luminance distribution of the second flare generated in the captured image with the second light portion as a light source. The comparison of the luminance distribution of the first flare with the luminance distribution of the second flare means, for example, whether or not the luminance distribution in the captured image with the blue light arrow as a light source is biased relative to the luminance distribution with lights other than the arrow as a light source.

Figure 7 is a diagram showing an example of the luminance distribution of the first and second light portions. For ease of explanation, Figure 7 shows a grayscale image in which the color information of the 80m enlarged view of Figure 6 has been inverted. Note that when recognizing that the direction is the direction indicated by the blue arrow signal, recognition including color information may also be performed.

As shown in FIG. 7, dashed lines FL1 and FL2 are drawn. Dashed line FL1 defines an area corresponding to the first flare that occurs in the captured image when the red light in FIG. 6 is used as a light source. In the case of the red light in FIG. 6, flare occurs isotropically from a circular light source, so the circle of dashed line FL1 can be equivalent to the circle of dashed line FL of the yellow light in FIG. 5. The luminance distribution of the first flare is approximately uniform within the circle of dashed line FL1.

The dashed line FL2 is illustrated for convenience in order to compare the luminance distribution of the second flare with the luminance distribution of the first flare. The dashed line FL2 is a circular area including the second flare that occurs in the captured image with the blue arrow signal in FIG. 6 as a light source, and is shown in the same shape as the dashed line FL1 as an example. In the blue arrow signal in FIG. 6, the flare occurs from an arrow-shaped light source with a smaller light-emitting area than the circular red signal, so the luminance distribution of the second flare is not uniform in the dashed line FL2, and the luminance is higher on the tip side (pointing direction) of the blue arrow and lower on the base side (opposite to the pointing direction) of the blue arrow. In the example of FIG. 7, the luminance distribution of the second flare has dark parts DZ1 and DZ2 that are low in the upper left and lower left parts within the circle of the dashed line FL2. In this case, the arrow direction recognition unit 15 can recognize that the right side of the paper in FIG. 7, which is opposite to the dark parts DZ1 and DZ2 with respect to the second flare, is the direction pointed by the blue arrow signal.

In addition, the arrow direction recognition unit 15 may, for example, calculate a luminance histogram in the vertical and horizontal directions for each pixel in a portion of the luminance distribution of an image captured using the red and blue arrow signals in FIG. 6 as light sources where the luminance is equal to or greater than the above-mentioned flare threshold, and recognize that the direction is that indicated by the blue arrow signal based on the direction in which the peak position on the luminance histogram of the blue arrow signal is biased relative to the peak position on the luminance histogram of the red signal.

Alternatively, the arrow direction recognition unit 15 may recognize the dark portions DZ1 and DZ2 based on whether the color of the pixels in the captured image using the red signal and blue arrow signal in Fig. 6 as light sources is red or blue (or green). When the arrow direction recognition unit 15 recognizes the dark portions DZ1 and DZ2, it may recognize the high brightness portions adjacent to the dark portions DZ1 and DZ2 in the captured image using a predetermined brightness threshold, and recognize the opposite side of the high brightness portions from the dark portions DZ1 and DZ2 as the direction indicated by the blue arrow signal.

For example, when the distance is 40 m in FIG. 6, the shape of the high-luminance part corresponding to the light source is such that it is possible to recognize the direction indicated by the blue arrow signal, compared to when the distance is 80 m or more. In this case, the arrow direction recognition unit 15 may obtain an area along the shape of the light source itself, such as the two-dot chain line LS in FIG. 5, for each light source of the red light and blue arrow signal, and recognize the direction indicated by the blue light arrow by comparing these shapes.

Incidentally, the arrow direction recognition unit 15 may not need to recognize the direction of the blue light arrow when it is determined that blurring due to vehicle swaying has occurred based on the captured image and the detection result of the vehicle acceleration sensor 4. For example, the arrow direction recognition unit 15 may determine that both the first blur and the second blur are caused by vehicle swaying when, based on the captured image, a first blur occurring in the captured image with the first light portion as a light source and a second blur occurring in the captured image with the second light portion as a light source extend in the vehicle up-down direction in the captured image with an equal amount of blur. The arrow direction recognition unit 15 may determine that both the first blur and the second blur are caused by vehicle swaying based on the detection result of the vehicle acceleration sensor when the vertical acceleration of the vehicle in the up-down direction is equal to or greater than a predetermined vertical acceleration threshold, or when the lateral acceleration of the vehicle in the lateral direction is equal to or greater than a predetermined lateral acceleration threshold.

Next, an example of the traffic light recognition process of the traffic light recognition device 100 will be described with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the process of the traffic light recognition device of FIG. 1. The process of the flowchart shown in FIG. 8 is repeated at a predetermined interval, for example, while the vehicle is traveling.

As shown in FIG. 8, in step S01, the ECU 10 recognizes the vehicle position by the vehicle position recognition unit 11. The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on, for example, the position information of the GPS receiver unit 1 and the map information of the map database 5.

In step S02, the ECU 10 recognizes objects using the object recognition unit 12. The object recognition unit 12 acquires an image of the area in front of the vehicle, including color information, based on an image captured by the camera 2. The object recognition unit 12 recognizes objects around the vehicle, for example, based on the image captured by the camera 2 and the detection results of the radar sensor 3.

In step S03, the ECU 10 estimates the candidate area using the candidate area estimation unit 13. The candidate area estimation unit 13 estimates the candidate area based on, for example, the position on the map of the vehicle recognized by the vehicle position recognition unit 11, the recognition result of the object recognition unit 12, and the map information of the map database 5.

In step S04, the ECU 10 determines whether or not a candidate area is present in the captured image using the lighting state recognition unit 14. If a candidate area is estimated in the captured image by the candidate area estimation unit 13, the lighting state recognition unit 14 determines that a candidate area is present in the captured image. If the ECU 10 determines that a candidate area is present in the captured image (step S04: YES), the ECU 10 proceeds to step S05. On the other hand, if the ECU 10 determines that a candidate area is not present in the captured image (step S04: NO), the ECU 10 ends the current process of FIG. 8.

In step S05, the ECU 10 recognizes the lighting state by the lighting state recognition unit 14. The lighting state recognition unit 14 recognizes the lighting state of the lights of the traffic lights in front of (around) the vehicle based on an image captured by the onboard camera. The lighting state recognition unit 14 recognizes the lighting state of the traffic lights in the candidate area, for example, by performing image recognition within the candidate area (e.g., pattern matching that takes color into consideration).

In step S06, the ECU 10 determines whether the lighting state with simultaneous lighting has been recognized by the lighting state recognition unit 14. When both the first light portion corresponding to the light other than the arrow and the second light portion corresponding to the blue light arrow are on, the lighting state recognition unit 14 determines that the lighting state with simultaneous lighting of the first light portion and the second light portion has been recognized. When either the first light portion corresponding to the light other than the arrow or the second light portion corresponding to the blue light arrow is not on, the lighting state recognition unit 14 determines that the lighting state with simultaneous lighting has not been recognized. When the ECU 10 determines that the lighting state with simultaneous lighting has been recognized (step S06: YES), it proceeds to step S07. On the other hand, when the ECU 10 determines that the lighting state with simultaneous lighting has not been recognized (step S06: NO), it ends the current process of FIG. 8.

In step S07, the ECU 10 determines whether or not blur (first blur and second blur) caused by vehicle shaking has occurred by the arrow direction recognition unit 15. The arrow direction recognition unit 15 may determine that the first blur and the second blur caused by vehicle shaking have occurred when the first blur caused in the captured image with the first lamp portion as a light source and the second blur caused in the captured image with the second lamp portion as a light source extend in the vehicle up-down direction in the captured image with an equal amount of blur based on the captured image. Alternatively, the arrow direction recognition unit 15 may determine that the first blur and the second blur caused by vehicle shaking have occurred based on the detection result of the acceleration sensor 4 of the vehicle when the vertical acceleration of the vehicle in the up-down direction is equal to or greater than a predetermined vertical acceleration threshold, or when the horizontal acceleration of the vehicle in the left-right direction is equal to or greater than a predetermined horizontal acceleration threshold. When the ECU 10 determines that blur caused by vehicle shaking has not occurred (step S07: YES), it proceeds to step S08. On the other hand, if the ECU 10 determines that blurring is occurring due to vehicle shaking (step S07: NO), it ends the current process in FIG. 8.

In step S08, the ECU 10 recognizes the arrow direction using the arrow direction recognition unit 15. When simultaneous illumination is recognized by the illumination state recognition unit 14, the arrow direction recognition unit 15 recognizes the direction indicated by the blue light arrow based on the comparison result between the luminance distribution of the first flare occurring in the captured image with the first light portion as a light source and the luminance distribution of the second flare occurring in the captured image with the second light portion as a light source. After that, the ECU 10 ends the current process of FIG. 8.

As described above, when the lighting state recognition unit 14 recognizes simultaneous lighting, for example, in a dark environment such as at night, there is a high possibility that the first flare and the second flare are occurring in the captured image. Since the first flare occurs in the captured image with the first light portion corresponding to the light other than the arrow as a light source, the luminance distribution on the captured image is concentric with the first light portion. Since the second flare occurs in the captured image with the second light portion corresponding to the blue light arrow as a light source, the luminance distribution on the captured image is biased with respect to the concentric circle corresponding to the second light portion. Therefore, according to the traffic light recognition device 100 described above, when the lighting state recognition unit 14 recognizes simultaneous lighting, the arrow direction recognition unit 15 can more accurately recognize the direction of the blue light arrow based on the bias of the luminance distribution of the second flare based on the luminance distribution of the first flare. This makes it possible to more accurately recognize the direction of the arrow, including traffic lights located far from the vehicle in a dark environment such as at night.

In the traffic light recognition device 100, the arrow direction recognition unit 15 does not recognize the direction of the blue light arrow when it is determined that blurring due to vehicle swaying has occurred based on the captured image and the detection results of the vehicle's acceleration sensor 4. This makes it possible to prevent a decrease in the accuracy of recognition of the direction the arrow is pointing in a situation where motion blurring due to vehicle swaying has occurred in the captured image.

[Modification]
Although the embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment. The present invention can be embodied in various forms including the above-mentioned embodiment and various modifications and improvements based on the knowledge of those skilled in the art.

In the traffic light recognition device 100 described above, the arrow direction recognition unit 15 is configured not to recognize the direction of the blue light arrow if it is determined that blurring due to vehicle swaying is occurring, but this process (step S07 in FIG. 8) may be omitted.

In the above embodiment, the first light portion corresponding to a light other than the arrow is a red light in terms of the lighting state when the lights are simultaneously turned on, but this is not limited to the above. The first light portion that is turned on simultaneously with the second light portion (blue light arrow) may be a light of a color other than red, such as blue or yellow. For moving objects such as trams, where the ability to proceed is determined by a light arrow of a color other than blue (e.g., yellow), the present invention can be applied by replacing the second light portion with a third light portion corresponding to a light arrow of a color other than blue.

In the above embodiment, a traffic light in front of the vehicle has been described, but traffic lights around the vehicle may also be imaged. Also, the radar sensor 3 does not necessarily have to be provided. For example, a TOF (Time Of Flight) sensor, a Far Infrared Rays camera, etc. may be used instead of the radar sensor 3.

2... camera (vehicle-mounted camera), 4... acceleration sensor, 14... lighting state recognition unit, 15... arrow direction recognition unit, 100... traffic light recognition device.

Claims (2)

a lighting state recognition unit that recognizes lighting states of lights of traffic signals around the vehicle based on an image captured by an in-vehicle camera;
and an arrow direction recognition unit that recognizes a direction indicated by a blue light arrow at the traffic light based on the captured image and the lighting state,
The lighting state recognition unit recognizes the lighting state of the first lighting portion and the second lighting portion when both the first lighting portion corresponding to the light other than the arrow and the second lighting portion corresponding to the blue light arrow are turned on,
When the simultaneous lighting is recognized by the lighting state recognition unit, the arrow direction recognition unit recognizes the direction indicated by the blue light arrow based on a comparison result between a luminance distribution of a first flare generated in the captured image using the first light portion as a light source and a luminance distribution of a second flare generated in the captured image using the second light portion as a light source. 2. The traffic light recognition device according to claim 1, wherein the arrow direction recognition unit does not recognize the direction indicated by the blue light arrow when it determines, based on the captured image and the detection results of the vehicle's acceleration sensor, that blurring due to the swaying of the vehicle is occurring.

Images