CN113928335B - Method and system for controlling a vehicle having an autonomous driving mode - Google Patents

Abstract

Aspects of the present disclosure relate to controlling a vehicle having an autonomous driving mode. For example, a current state of a traffic light may be determined. One of the plurality of yellow light durations may be selected based on a current state of the traffic light. It is possible to predict when a traffic light will turn red based on the selected one. The prediction may be used to control the vehicle in an autonomous driving mode.

Description

Method and system for controlling a vehicle having an autonomous driving mode

Technical Field

The present invention relates to a method and system for controlling a vehicle having an autonomous driving mode, in particular, enabling an autonomous vehicle to respond to a yellow traffic light.

Background Art

Autonomous vehicles, e.g., vehicles that do not require a driver when operating in an autonomous driving mode, can be used to help transport passengers or items from one location to another. An important component of an autonomous vehicle is a perception system that allows the vehicle to perceive and interpret its surroundings using sensors, such as cameras, radars, lidar sensors, and other similar devices. For example, the perception system and/or the vehicle's computing device can process data from these sensors to identify objects and their characteristics, such as location, shape, size, orientation, direction, acceleration or deceleration, type, etc. This information is critical for the vehicle's computing system to make appropriate driving decisions for the vehicle.

Summary of the invention

Aspects of the present disclosure provide a method for controlling a vehicle with an autonomous driving mode. The method includes one or more processors of one or more computing devices of the vehicle receiving a current state of a traffic light; the one or more processors selecting one of a plurality of yellow light durations based on the current state of the traffic light; predicting when the traffic light will turn red based on the selected one; and the one or more processors controlling the vehicle in the autonomous driving mode using the prediction.

In one example, the method includes one or more processors receiving an updated state of a traffic light, and one or more processors selecting a second of a plurality of yellow light durations based on the updated state of the traffic light, wherein the selected one and the second are different durations. In this example, the second duration is selected without relying on any state of the traffic light before the updated state of the traffic light. In another example, the method also includes determining the time when the traffic light was last observed in a particular state, and wherein the prediction of when the traffic light will turn red is further based on determining the time when the traffic light was last observed in the particular state. In this example, the particular state is a flashing yellow light. Alternatively, the particular state is a flashing green light. In another example, the plurality of yellow light durations include a first yellow light duration for a constant green circle state of the traffic light and a second yellow light duration for a green arrow state of the traffic light, and the first yellow light duration and the second yellow light duration are different durations. In another example, the plurality of yellow light durations include a first yellow light duration for a constant green arrow state of the traffic light and a second yellow light duration for a flashing yellow arrow state of the traffic light, and the first yellow light duration and the second yellow light duration are different durations. In another example, a plurality of yellow light durations are stored in a table associated with the traffic light, and selecting one of the plurality of yellow light durations includes accessing the table. In this example, the method further includes one or more processors determining a duration of the yellow light of the traffic light and storing the determined duration for later use. In addition, the method also includes sending the determined duration to a remote computing device.

Another aspect of the present disclosure provides a system for controlling a vehicle with an autonomous driving mode. The system includes one or more processors configured to receive a current state of a traffic light; select one of a plurality of yellow light durations based on the current state of the traffic light; predict when the traffic light will turn red based on the selected one; and control the vehicle in the autonomous driving mode using the prediction.

In one example, the system also includes a vehicle. In another example, the one or more processors are further configured to receive an updated state of the traffic light; and based on the updated state of the traffic light, select a second of the plurality of yellow light durations, wherein the selected one and the second are different durations. In this example, the one or more processors are further configured to select the second yellow light duration without relying on any state of the traffic light before the updated state of the traffic light. In another example, the plurality of yellow light durations include a first yellow light duration for a solid green circle state of the traffic light and a second yellow light duration for a green arrow state of the traffic light, wherein the first yellow light duration and the second yellow light duration are different durations. In another example, the plurality of yellow light durations include a first yellow light duration for a green arrow state of the traffic light and a second yellow light duration for a flashing yellow arrow state of the traffic light, and the first yellow light duration and the second yellow light duration are different durations. In another example, the system also includes storing a table including a plurality of yellow light durations, and the one or more processors are further configured to access the table to select one of the plurality of yellow light durations. In this example, the one or more processors are also used to determine the duration of the yellow light of the traffic light; and store the determined duration for later use. In another example, the one or more processors are further configured to send the determined duration to a remote computing device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of an example vehicle according to aspects of the present disclosure.

FIG. 2 is an example of map information according to aspects of the present disclosure.

FIG. 3 is an example diagram of a vehicle according to aspects of the present disclosure.

4 is an example visual diagram of a system according to aspects of the present disclosure.

5 is an example functional diagram of a system according to aspects of the present disclosure.

6 is an example aerial view of a geographic area according to aspects of the present disclosure.

7 is an example camera image in accordance with aspects of the present disclosure.

FIG. 8 is an example flow chart according to aspects of the present disclosure.

DETAILED DESCRIPTION

Overview

The technology involves enabling an autonomous vehicle to respond to a yellow light. For example, when a perception system of an autonomous vehicle observes a yellow light at a traffic light intersection, a computing device of the vehicle must decide whether the vehicle should stop or continue through the intersection. Current approaches for such determinations may involve pre-storing default traffic light durations based on the speed of the lanes at the intersection, the geometry of the lanes, and/or other formulas used by cities or other agencies to set the duration of yellow lights. However, in practice, there may be a lack of consistency in how these durations are set across the country or even within the same city. As a result, in many cases, the yellow light may be much shorter or longer than expected, which may cause the autonomous vehicle to run a red light or stop immediately when there is sufficient time to pass through the intersection, resulting in an emergency braking event. Other approaches that may include communicating with or receiving information from traffic lights about the duration of yellow lights may not be suitable for widespread use.

To address these shortcomings, rather than simply pre-storing a default yellow light duration for a traffic light, multiple default yellow light durations may be stored based on different possible transitions of the traffic light. For example, while some traffic lights may only have transitions from green to yellow to red to green, other types of traffic lights may have different types of transitions. Thus, a different default yellow light duration may be stored for each of the different possible transitions of the traffic light.

The yellow light duration can be determined in various ways. For example, a human labeler can be asked to label the duration of the yellow light from the video. Alternatively, the duration can be automatically determined by an onboard traffic light detection system software module of the autonomous vehicle.

In use, when an autonomous vehicle approaches a traffic light, the vehicle's traffic light detection system software module will attempt to detect the current state of the traffic light. Based on the current state of the traffic light, the vehicle's computing device can determine the duration of the next or current yellow light of the traffic light. In this regard, the vehicle's computing device can access a pre-stored table of traffic light durations for the traffic light and select a yellow light duration. The selected yellow light duration can then be used to predict when the traffic light will turn red, and in turn determine whether the vehicle should stop for the traffic light.

This information may also be shared with a remote computing device, such as a backend server computing device. If there is enough difference between the duration of the table and the additional duration, it can be used as a signal to mark the traffic light for additional analysis. Additionally or alternatively, the server computing device may broadcast this information to other nearby vehicles or all vehicles in the fleet. These differences may also be used to determine if there is any bimodal distribution for a particular traffic light. In other words, the yellow light duration may vary at different times of the day and/or different days of the week (e.g., longer during rush hour or less busy times).

The features described in this article can enable autonomous vehicles to recognize and respond to different yellow light durations at a single traffic light. This can reduce the likelihood that such vehicles will run red lights or brake suddenly at inappropriate times.

Example System

As shown in FIG1 , a vehicle 100 according to one aspect of the present disclosure includes various components. Although certain aspects of the present disclosure are particularly useful for specific types of vehicles, the vehicle can be any type of vehicle, including but not limited to automobiles, trucks, motorcycles, buses, recreational vehicles, etc. The vehicle can have one or more computing devices, such as a computing device 110 that includes one or more processors 120, a memory 130, and other components typically found in a general-purpose computing device.

Memory 130 stores information accessible by one or more processors 120, including instructions 132 and data 134 that can be executed or used by processor 120. Memory 130 can be of any type capable of storing information accessible by a processor, including computing device readable media, or other media that stores data that can be read by an electronic device, such as a hard drive, memory card, ROM, RAM, DVD or other optical disk, and other writable and read-only memory. Systems and methods may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.

Instructions 132 may be any set of instructions that are executed by a processor directly (such as machine code) or indirectly (such as a script). For example, instructions may be stored as computing device code on a computing device readable medium. In that regard, the terms "instructions" and "programs" may be used interchangeably herein. Instructions may be stored in an object code format to be processed directly by a processor, or may be stored in any other computing device language including a script or a collection of independent source code modules that are interpreted on demand or compiled in advance. The functions, methods, and routines of instructions will be explained in more detail below.

Processor 120 may retrieve, store, or modify data 134 according to instructions 132. For example, although the claimed subject matter is not limited to any particular data structure, the data may be stored in a computing device register in a relational database as a table with multiple different fields and records, an XML document, or a flat file. The data may also be formatted in any computing device readable format.

One or more processors 120 may be any conventional processor, such as a commercially available CPU or GPU. Alternatively, one or more processors may be a dedicated device, such as an ASIC or other hardware-based processor. Although the processor, memory, and other elements of the computing device 110 are functionally shown in the same box in FIG. 1 , a person of ordinary skill in the art will understand that the processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be stored in the same physical housing. For example, the memory may be a hard drive or other storage medium located in a housing different from the housing of the computing device 110. Therefore, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

The computing device 110 may also be connected to one or more speakers 112 and one or more user inputs 114. The speakers may enable the computing device to provide audible messages and information to occupants of the vehicle, including the driver. In some examples, the computing device may be connected to one or more vibration devices that are configured to vibrate based on signals from the computing device to provide tactile feedback to the driver and/or any other occupants of the vehicle. As examples, the vibration device may consist of a vibration motor or one or more linear resonant actuators that are placed under or behind one or more occupants of the vehicle, such as embedded in one or more seats of the vehicle.

The user input may include buttons, touch screens, or other devices that enable an occupant of the vehicle, such as a driver, to provide input to the computing device 110 as described herein. As an example, a button or option on a touch screen may be specifically designed to cause a transition from an autonomous driving mode to a manual driving mode or a semi-autonomous driving mode.

In one aspect, computing device 110 can be part of an autonomous control system that can communicate with various components of a vehicle to control the vehicle in an autonomous driving mode. For example, returning to FIG. 1 , computing device 110 can communicate with various systems of vehicle 100, such as deceleration system 160, acceleration system 162, steering system 164, route selection system 166, planning system 168, positioning system 170, and perception system 172, so as to control the movement, speed, etc. of vehicle 100 according to instructions 132 of memory 130 in an autonomous driving mode. In this regard, each of these systems can be one or more processors, memories, data, and instructions. Such processors, memories, instructions, and data can be configured similarly to one or more processors 120, memories 130, instructions 132, and data 134 of computing device 110.

As an example, computing device 110 may interact with deceleration system 160 and acceleration system 162 to control the speed of the vehicle. Similarly, computing device 110 may use steering system 164 to control the direction of vehicle 100. For example, if vehicle 100 is configured for on-road use, such as a car or truck, the steering system may include components that control the angle of the wheels to steer the vehicle.

The computing device 110 may use the planning system 168 to determine and travel to a location along a route generated by the routing system 166. For example, the routing system 166 may use the map information to determine a route from the vehicle's current location to a drop-off location. The planning system 168 may periodically generate a trajectory or short-term plan for controlling the vehicle over a future period of time to reach a destination along a route (the vehicle's current route). In this regard, the planning system 168, the routing system 166, and/or the data 134 may store detailed map information, such as a highly detailed map identifying the shape and elevation of roads, lane lines, intersections, crosswalks, speed limits, traffic signals, buildings, signs, real-time traffic information, vegetation, or other such objects and information. In addition, the map information may identify area types, such as built-up areas, school areas, residential areas, parking lots, etc.

The map information may include one or more road graphs or a graphical network of information such as roads, lanes, intersections, and connections between these features, which may be represented by road segments. Each feature may be stored as graph data and may be associated with information such as its geographic location and whether it is linked to other related features, e.g., a stop sign may be linked to roads and intersections, etc. In some examples, the associated data may include a grid-based index of the road graph to allow efficient lookup of certain road graph features.

2 is an example of map information 200 for a road segment including intersections 202 and 204. The map information 200 may be a local version of the map information stored in the memory 130 of the computing device 110. Other versions of the map information may also be stored in the storage system 450 discussed further below. In this example, the map information 200 includes information identifying the shape, location, and other characteristics of lane lines 210, 212, 214, traffic lights 220, 222, stop lines 224, crosswalks 230, sidewalks 240, stop signs 250, 252, and yield signs 260 that define lanes 216, 218. In this regard, the map information includes the three-dimensional (3D) location of the traffic lights 220, 222 and information identifying the lanes controlled by these traffic lights.

In some examples, the map information may identify additional information about the traffic light. This information may include, for example, the expected state and duration (e.g., how long a green, yellow, or red light should last) and information identifying which lane the traffic light controls. In this regard, the map information may store the expected state and duration of the traffic lights 220, 222 and information indicating that the traffic lights control lanes 216, 218, respectively.

However, rather than simply pre-storing a default yellow light duration for a traffic light, multiple default yellow light durations may be stored based on different possible transitions of the traffic light. For example, while some traffic lights may only have transitions from green to yellow to red to green, other types of traffic lights may have different types of transitions, such as from flashing yellow to solid yellow.

Regarding the first example, the traffic light 220 may include four lights or four light states. These lights or light states may include a permanent left turn red arrow (SLRA), a permanent left turn yellow arrow (SLYA), a flashing left turn yellow arrow (FLYA), and a permanent left turn green arrow (SLGA). The yellow light duration of the permanent left turn yellow arrow (SLYA) of the traffic light 220 may be different depending on the transition. For example, when the traffic light transitions from FLYA to SLYA, the duration of SLYA may be different from the duration of SLYA when the traffic light transitions from SLGA to SLYA.

As a second example, the traffic light 222 may include four lights or four light states. These lights or light states may include a permanent red circle (SRC), a permanent green arrow (SGA), a permanent green circle (SGC), and a permanent yellow circle (SYC). The yellow light duration of the permanent yellow circle (SYC) of the traffic light 222 may be different depending on the transition. For example, when the traffic light transitions from SGC to SYC, the duration of SYC may be different from the duration of SYC when the traffic light transitions from SGA to SYC.

Other traffic light configurations may also undergo different transitions where there are multiple green elements (such as circles and arrows) and only one yellow light.

Thus, for each of the different possible transitions of a traffic light, a different default yellow light duration may be stored. For example, these durations may be stored in a table for each traffic light that is attached to the map information used to control the vehicle. In this regard, all or some of the traffic lights in the map information may be associated with a single table identifying the durations. Some traffic lights (such as those with only three lights) may be associated with only a single default yellow light duration, while other traffic lights may be associated with multiple default yellow light durations (one for each possible transition).

For example, Table 1 below is an example table of traffic lights 220. In this example, duration 1 is different from duration 2. The table may be attached to the map information 200 and thus pre-stored.

Table 1-Traffic Light 220

Regarding another example, Table 2 below is an example table of traffic lights 222. In this example, duration 3 is different from duration 4. Again, this table may be attached to the map information 200 and thus pre-stored.

Table 2-Traffic Light 222

Observed light status Next yellow light Next yellow light duration SLGA SYC Duration 3 SGC SYC Duration 4

Of course, if there are not enough observations and data for a particular traffic light, there may not be a table for that particular traffic light. If there is no table for a particular traffic light, a default yellow light duration may be used based on the speed of the traffic lanes at the intersection, the geometry of the traffic lanes, and/or other formulas used by the city or other agency to set yellow light durations.

The yellow light duration can be determined in a variety of ways. For example, a human labeler can be asked to label the duration of a yellow light from a video. For example, for different frames (or images) of a traffic light, the labeler can identify the first frame in which a yellow light is observed and the last frame in which a yellow light is observed, and can extrapolate the amount of time between these frames. However, because human labelers are not perfect and their labels will vary slightly, for a particular yellow light duration, an average of multiple different observations of the same traffic light can be used.

As another approach, the durations may be determined automatically by the autonomous vehicle's onboard traffic light detection system software module. This data may be noisier because lights may be obscured, misdetected, and other heuristic algorithms may change the yellow light duration settings. Additionally, it may be difficult to determine which is a reliable segment of sensor data from which these yellow light durations were extracted. However, by determining the yellow light durations in real time, this may help detect changes to the input pre-stored durations and can be used to indicate which traffic lights need to be updated.

Although the map information can be an image-based map, the map information need not be entirely image-based (e.g., raster-based). For example, the map information can include one or more road graphics or a graphical network of information such as roads, lanes, intersections represented by nodes, and connections between these features, which can be represented by road segments. Each feature can be stored as graph data and can be associated with information such as a geographic location and whether it is linked to other related features, for example, a stop sign can be linked to roads and intersections, etc. In some examples, the associated data can include a grid-based index of the road graphics to allow efficient lookup of certain road graphics features.

The computing device 110 may use a positioning system 170 to determine the relative or absolute position of the vehicle on a map or the earth. The positioning system 170 may also include a GPS receiver to determine the latitude, longitude, and/or altitude position of the device relative to the earth. Other positioning systems, such as a laser-based positioning system, an inertial assisted GPS, or a camera-based positioning system may also be used to identify the location of the vehicle. The location of the vehicle may include an absolute geographic location, such as latitude, longitude, and altitude, and relative position information, such as relative to the location of other cars in its immediate vicinity, which can often be determined with less noise than the absolute geographic location.

Positioning system 170 may also include other devices in communication with computing device 110, such as an accelerometer, a gyroscope, or another direction/speed detection device to determine the direction and speed of the vehicle or changes thereto. By way of example only, an acceleration device may determine its pitch, yaw, or roll angle (or changes thereto) relative to the direction of gravity or a plane perpendicular thereto. The device may also track increases or decreases in speed and the direction of such changes. Position and orientation data provided by the devices as described herein may be automatically provided to computing device 110, other computing devices, and combinations of the foregoing.

The perception system 172 also includes one or more components for detecting objects external to the vehicle, such as other vehicles, obstacles in the road, traffic signals, signs, trees, etc. For example, the perception system 172 may include lasers, sonar, radar, cameras, and/or any other detection devices that record data that can be processed by the computing device 110. In the case where the vehicle is a passenger vehicle, such as a minivan, the minivan may include a laser or other sensor mounted on the roof or other convenient location.

For example, FIG. 3 is an example exterior view of a vehicle 100. In this example, a roof shell 310 and roof shells 312, 314 may include a LIDAR sensor and various cameras and radar units. In addition, a housing 320 located at the front end of the vehicle 100 and housings 330, 332 on the driver and passenger sides of the vehicle may each store a LIDAR sensor. For example, housing 330 is located in front of doors 360, 362 that also include windows 364, 366. The vehicle 100 also includes housings 340, 342 for radar units and/or cameras that are also located on the roof of the vehicle 100. Other radar units and cameras (not shown) may be located at the front and rear ends of the vehicle 100 and/or at other locations along the roof or roof shell 310.

The computing device 110 is capable of communicating with various components of the vehicle to control the movement of the vehicle 100 according to the primary vehicle control code of the memory of the computing device 110. For example, returning to FIG. 1 , the computing device 110 may include various computing devices that communicate with various systems of the vehicle 100, such as the deceleration system 160, the acceleration system 162, the routing system 166, the planning system 168, the positioning system 170, the perception system 172, and the power system 174 (i.e., the engine or motor of the vehicle) to control the movement, speed, etc. of the vehicle 100 according to the instructions 132 of the memory 130.

Various systems of the vehicle may operate using the autonomous vehicle control software to determine how and where to control the vehicle. As an example, the perception system software module of the perception system 172 may use sensor data generated by one or more sensors of the autonomous vehicle (such as cameras, LIDAR sensors, radar units, sonar units, etc.) to detect and identify objects and their features. These features may include location, type, orientation, position, speed, acceleration, change in acceleration, size, shape, etc. In some examples, the features may be input into a behavior prediction system software module that uses various behavior models based on the object type to output a predicted future behavior for the detected object.

In other examples, the features may be input into one or more detection system software modules, such as a traffic light detection system software module configured to detect a known traffic signal state, a school bus detection system software module configured to detect a school bus, a construction area detection system software module configured to detect a construction area, a detection system software module configured to detect one or more persons (e.g., pedestrians) directing traffic, a traffic accident detection system software module configured to detect a traffic accident, an emergency vehicle detection system configured to detect an emergency vehicle, etc. These detection system software modules may be incorporated into the perception system 172 or the computing device 110. Each of these detection system software modules may input sensor data generated by the perception system 172 and/or one or more sensors (and in some examples, map information of the area surrounding the vehicle) into various models, and the various models may output the likelihood of a certain traffic light state, the likelihood that the object is a school bus, the area of the construction area, the likelihood that the object is a person directing traffic, the area of the traffic accident, the likelihood that the object is an emergency vehicle, etc.

Detected objects, predicted future behaviors, various possibilities from the detection system software module, map information identifying the vehicle's environment, location information from the positioning system 170 identifying the vehicle's position and orientation, the vehicle's destination, and feedback from various other systems of the vehicle may be input into the planning system software module of the planning system 168. The planning system may use this input to generate a trajectory for the vehicle to travel along in the future short period of time based on the vehicle's current route generated by the route selection module of the route selection system 166. The control system software module of the computing device 110 may be configured to control the movement of the vehicle, such as by controlling the braking, acceleration, and steering of the vehicle, in order to travel along the trajectory.

The computing device 110 may also include one or more wireless network connections 150 to facilitate communication with other computing devices, such as client computing devices and server computing devices described in detail below. The wireless network connections may include short-range communication protocols such as Bluetooth, Bluetooth Low Energy (LE), cellular connections, and various configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using a communication protocol proprietary to one or more companies, Ethernet, WiFi, and HTTP, as well as various combinations of the above.

The computing device 110 can control the vehicle in the autonomous driving mode by controlling various components. For example, as an example, the computing device 110 can use data from the detailed map information and planning system 168 to fully autonomously navigate the vehicle to the destination location. The computing device 110 can use the positioning system 170 to determine the location of the vehicle, and use the perception system 172 to detect objects and respond to objects when necessary to safely reach the location. Furthermore, in order to do so, the computing device 110 can generate trajectories and drive the vehicle along these trajectories, for example, accelerating the vehicle (e.g., the acceleration system 162 supplies fuel or other energy to the engine or power system 174), decelerating (e.g., reducing the fuel supplied to the engine or power system 174, shifting and/or deceleration system 160 applying brakes), changing direction (e.g., the steering system 164 steers the front wheels or rear wheels of the vehicle 100), and sending signals of such changes (e.g., using a turn signal). Therefore, the acceleration system 162 and the deceleration system 160 can be part of a powertrain system including various components between the vehicle engine and the vehicle wheels. Furthermore, by controlling these systems, the computing device 110 may also control the vehicle's powertrain to autonomously maneuver the vehicle.

The computing device 110 of the vehicle 100 may also receive information from or transmit information to other computing devices, such as those computing devices that are part of a transportation service and other computing devices. FIGS. 3 and 4 are a visual diagram and a functional diagram, respectively, of an example system 400 including a plurality of computing devices 410, 420, 430, 440 and a storage system 450 connected via a network 460. The system 400 also includes the vehicle 100 and vehicles 100A, 100B that may be configured the same or similarly to the vehicle 100. Although only a few vehicles and computing devices are depicted for simplicity, a typical system may include significantly more vehicles and computing devices.

4 , each of computing devices 410 , 420 , 430 , 440 may include one or more processors, memories, data, and instructions. Such processors, memories, data, and instructions may be configured similarly to one or more processors 120 , memories 130 , data 132 , and instructions 134 of computing device 110 .

The network 460 and intermediate nodes may include a variety of configurations and protocols, including short-range communication protocols such as Bluetooth, Bluetooth low energy, the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using a communication protocol proprietary to one or more companies, Ethernet, WiFi, and HTTP, and various combinations of the above. Such communication may be facilitated by any device capable of sending data to or from other computing devices, such as modems and wireless interfaces.

In one example, the one or more computing devices 410 may include one or more server computing devices having multiple computing devices, such as a load balancing server cluster, that exchange information with different nodes of a network in order to receive, process, and send data to or from other computing devices. For example, the one or more computing devices 410 may include one or more server computing devices that are capable of communicating with the computing device 110 of the vehicle 100 or a similar computing device of the vehicle 100A and computing devices 420, 430, 440 via the network 460. For example, the vehicles 100, 100A may be part of a fleet that may be dispatched to various locations by the server computing devices. In this regard, the server computing device 410 may act as a verification computing system that may be used to verify that vehicles (such as the vehicles 100 and the vehicles 100A) may be used for autonomous control software operating in an autonomous driving mode. Additionally, server computing device 410 may use network 460 to transmit and present information to users, such as users 422, 432, 442 on displays, such as displays 424, 434, 444 of computing devices 420, 430, 440. In this regard, computing devices 420, 430, 440 may be considered client computing devices.

As shown in Figure 4, each client computing device 420, 430, 440 can be a personal computing device intended for use by one or more users 422, 432, 442, and has all the components commonly used in conjunction with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory (e.g., RAM and internal hard drive) to store data and instructions, a display such as a display 424, 434, 444 (e.g., a monitor with a screen, touch screen, projector, television, or other device operable to display information), and a user input device 426, 436, 446 (e.g., a mouse, keyboard, touch screen, or microphone). The client computing device may also include a camera for recording a video stream, a speaker, a microphone, a network interface device, and all the components for connecting these elements to each other.

Although client computing devices 420, 430, and 440 may each include a full-size personal computing device, they may alternatively include client computing devices capable of wirelessly exchanging data with a server over a network, such as the Internet. By way of example only, client computing device 420 may be a mobile phone or a device such as a wireless-enabled PDA, a tablet PC, a wearable computing device or system, or a netbook capable of acquiring information via the Internet or other networks. In another example, client computing device 430 may be a wearable computing system, such as a smartwatch as shown in FIG. 4. As examples, a user may input information using a keypad, a keyboard, a microphone, a camera using visual signals, or a touch screen.

In some examples, client computing device 420 may be a mobile phone used by a vehicle passenger. In other words, user 422 may represent a passenger. Additionally, client communication device 430 may represent a smartwatch of a vehicle passenger. In other words, user 432 may represent a passenger. Client communication device 430 may represent an operator's workstation, such as a remote assistance operator or someone who may provide remote assistance to a vehicle and/or passenger. In other words, user 442 may represent a remote assistance operator. Although only a few passengers and operators are shown in FIGS. 4 and 5 , any number of such passengers and operators (and their corresponding client computing devices) may be included in a typical system.

As with memory 130, storage system 450 may be any type of computerized memory capable of storing information accessible by server computing device 410, such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory. In addition, storage system 450 may include a distributed storage system in which data is stored on multiple different storage devices that may be physically located in the same or different geographic locations. Storage system 450 may be connected to computing devices via network 460 as shown in FIGS. 4 and 5 and/or may be directly connected to or incorporated into any of computing devices 110, 410, 420, 430, 440, etc.

Storage system 450 may store various types of information that may be retrieved or accessed by server computing devices, such as one or more server computing devices 410 , in order to perform various actions.

Example Method

In addition to the operations described above and shown in the figures, various operations will now be described. It should be understood that the following operations do not have to be performed in the exact order described below. More precisely, various steps can be processed in different orders or simultaneously, and steps can also be added or omitted.

8 includes an example flow chart 800 for some of the examples of controlling a vehicle with an autonomous driving mode, which may be executed by one or more processors, such as processor 120 of computing device 110, to determine when a traffic light will next turn red. For example, at block 810, a current state of a traffic light is determined.

When an autonomous vehicle, such as vehicle 100, is traveling in an autonomous driving mode, the vehicle's computing device 110 and/or perception system 172 may detect and identify the location and current state of traffic lights. As described above, a traffic light detection system software module may be incorporated into the perception system 172 or computing device 110 and may access map information to determine where the traffic light detection system software model should expect to perceive traffic lights.

FIG. 6 depicts a vehicle 100 being maneuvered on a section of road 600 including intersections 602 and 604. In this example, intersections 602 and 604 correspond to intersections 202 and 204, respectively, of map information 200. In this example, lane lines 610, 612, 614 and lanes 616, 618 correspond to the shapes, positions, and other characteristics of lane lines 210, 212, 214 and lanes 216, 218, respectively. Similarly, crosswalk 630 corresponds to the shape, position, and other characteristics of crosswalk 230, respectively; sidewalk 640 corresponds to sidewalk 240; traffic lights 620, 622 correspond to traffic lights 220, 222, respectively; stop signs 650, 652 correspond to stop signs 250, 252, respectively; and yield sign 660 corresponds to yield sign 260. In this example, vehicle 100 is approaching intersection 602 controlled by traffic light 620, as shown in the map information.

The computing device 110 and/or the perception system 172 may attempt to identify the status and location of traffic lights along the route. In the example of FIG. 6 , the computing device 110 and/or the perception system 172 of the vehicle may use the traffic light detection system software module to attempt to locate traffic light 220 (corresponding to traffic light 620 ) and thereby determine the status of traffic light 620 .

As part of this, perception system 172 may use a camera to capture images of the environment of vehicle 100. FIG7 is an example camera image 700 captured by a camera of perception system 172 of vehicle 100 as the vehicle approaches intersection 602 from lane 616, such as the example of FIG6. In this example, camera image 700 includes traffic light 620.

These images, as well as other information as described above, can be input into the traffic light detection system software module to determine the state of the traffic light 620 and other information, such as its three-dimensional position. The state can be any number of different values, including, for example, "off", "not detected", "steady red circle", "steady yellow circle", "steady green circle", "flashing red circle", "flashing yellow circle", "steady yellow arrow", "steady red arrow", "steady green arrow", "flashing yellow arrow", "flashing green arrow", etc. In this regard, "steady" means that the light does not flash, or more precisely, it appears to be always on to an observer. In addition, each arrow (steady or flashing) can also be further delineated into its direction, such as "steady left turn yellow arrow", "steady left turn red arrow", "steady left turn green arrow", "flashing left turn yellow arrow", "flashing left turn green arrow", "steady right turn yellow arrow", "steady right turn red arrow", "steady right turn green arrow", "flashing right turn yellow arrow", "flashing right turn green arrow", etc. A state, such as off and not detected, may correspond to a traffic light being obscured (e.g., by another vehicle, a building or other structure, vegetation, weather conditions, etc.) or too far from the vehicle to be practically determined. This information may be published by the traffic light detection system software module and may be made available to other computing devices and/or systems of the vehicle 100.

For example, the traffic light detection system can process the camera image 700 and identify the area 720 of the image that includes the traffic light 620. The traffic light detection system can identify which of the four lights A, B, C, D of the traffic light 620 is currently on. The currently on light will correspond to the state of the traffic light 620.

Returning to FIG. 8 at block 820, one of the multiple yellow light durations is selected based on the current state of the traffic light. For example, based on the current state of the traffic light, the computing device 110 may use the current state of the traffic light to find or select the duration of the next or current yellow light. For example, if the current state of the traffic light is any type of red (e.g., solid red circle, flashing red circle, solid red arrow, flashing red arrow), the computing device may not need to determine the duration of the next or current yellow light. In other words, for a red light, it may not be necessary to determine the duration of the next or current yellow light, and therefore may not be performed by the computing device 110.

If the current state of the traffic light is any type of green (e.g., a solid circle, a flashing circle, a solid arrow, a flashing arrow) or a flashing yellow (e.g., a flashing yellow circle or a flashing yellow arrow), the computing device may use the current state of the traffic light to determine the duration of the next yellow light. In this regard, the computing device of the vehicle may access a traffic light duration table for the traffic light and select the yellow light duration based solely on the current state of the traffic light. Thus, when the current state of the traffic light is any type of green or flashing yellow, the determination of the next yellow light duration is based solely on the most currently detected state of the traffic light and not any previous state. This may reduce the likelihood of an erroneous determination if the traffic light is partially obscured at different points in time (which may result in errors in the transitions actually observed by the traffic light state detection system).

As described above, if there are multiple possible transitions of the traffic light, the table may store multiple possible yellow light durations. Therefore, the computing device 110 may select one of the multiple yellow light durations from the table. For example, returning to the example of the traffic light 620, if the current state of the traffic light is determined to be a flashing left turn yellow arrow (FLYA), then with reference to Table 1, the next yellow light duration will be duration 1. Therefore, duration 1 may be selected. However, when the current state of the traffic light is determined to be a green arrow (GA), then with reference to Table 1, the next yellow light duration will be duration 2. Therefore, duration 2 may be selected.

As another example, for example, returning to the example of traffic light 622, if the current state of the traffic light is determined to be a green arrow (GA), then referring to Table 2, the next yellow light duration will be duration 3. Therefore, duration 3 may be selected. However, when the current state of the traffic light is determined to be a steady green circle (SGC), then referring to Table 2, the next yellow light duration will be duration 4. Therefore, duration 4 may be selected.

Returning to FIG. 8 , at box 830 , the selected yellow light duration is used to predict when the traffic light will next turn red, and then at box 840 , the prediction is used to control the vehicle in the autonomous driving mode. For example, the computing device 110 may then use the selected yellow light duration to predict when the traffic light will turn red, and in turn determine whether to stop for the traffic light. However, in order to provide an accurate prediction of when the traffic light may turn red, the computing device of the vehicle may continue to detect the current state of the traffic light in order to accurately observe when the solid or flashing green light (arrow or circle) or the flashing yellow light (arrow or circle) turns into a solid yellow light (arrow or circle). At this point, the computing device of the vehicle may simply add the selected yellow light duration to the time point of the last observed solid or flashing green light or flashing yellow light (arrow or circle) to predict when the traffic light will turn red. In this regard, if the last observed solid or flashing green light or flashing yellow light (arrow or circle) is not detected, the computing device of the vehicle may assume that the solid yellow light may turn red at any time and will plan accordingly.

When a traffic light is predicted to turn red, it can be used as an input to the planning system 168 to decide whether to stop for the traffic light. Alternatively, the planning system can use the selected yellow light duration and the last observed solid or flashing green light (arrow or circle) or flashing yellow light (arrow or circle) as input to predict when the light is likely to turn red, which in turn will inform the vehicle's decision whether to stop for the traffic light. In other words, when the traffic light is a solid yellow arrow or circle, the planning system can accurately determine whether the vehicle has enough time to safely pass through the intersection. The planning system can accordingly generate a trajectory for the vehicle to travel along in the future.

As described above, if there is no table for a particular traffic light, for example because there have not been sufficient labels or observations for the yellow light duration of the traffic light, the computing device 110 may use a default yellow light duration based on the speed of the traffic lanes at the intersection and the geometry of the traffic lanes.

If the current state of the traffic light is a solid yellow (e.g., a solid yellow circle or a solid yellow arrow) or more specifically the traffic light is a yellow light and not flashing, the computing device 110 may not need to determine the duration of the next yellow light. However, to determine the duration of the current yellow light, the computing device 110 may use the state of the last solid or flashing green or yellow light. In this regard, a flashing yellow light is considered a solid or flashing green light. Therefore, the computing device of the vehicle may access the traffic light duration table of the traffic light and select a yellow light duration for the current yellow light based on the previous state of the traffic light and the current state of the traffic light.

Of course, over time, the state of the traffic light will change. The sensors of the perception system 172 will continue to capture sensor data, and the computing device 110 and/or the perception system 172 can input the sensor data into the traffic light detection system software module. Therefore, the traffic light detection system software module can determine the updated state of the traffic light. Each time the state of the traffic light changes, as long as the updated state is not red, the computing device 110 can use the change or update state to determine the duration of the next or current yellow light. For example, the computing device 110 can select the second of multiple yellow light durations for a given traffic light based on the updated state of the traffic light. In other words, each time the state of the traffic light changes, the computing device 110 can re-evaluate the duration of the next or current yellow light and determine the duration accordingly as described above.

Additionally, each time the vehicle's traffic light detection system software module observes a yellow light at a traffic light, the duration of the traffic light may be determined by, for example, the traffic light detection system software module. In simple terms, the duration may be the length of time during which the traffic light detection system software module observes a yellow light (such as a solid yellow circle or a solid yellow arrow). In some examples, if the confidence level of detecting a yellow light and/or its duration is relatively high, the duration may be stored for later use by the computing device 110, for example, by appending the duration to a table for the traffic light.

This information may also be shared with a remote computing device, such as server computing device 410. In this regard, computing device 110 may send the duration of the traffic light to server computing device 410, for example, via network 460.

The server computing device 410 can use this information in various ways. For example, when there is a sufficient difference between the duration in the table and the additional duration, the server computing device 410 can use it as a signal to mark the traffic light for additional analysis. For example, a statistical model (e.g., mean and standard deviation) can be used to determine whether the yellow light duration of a traffic light deviates sufficiently from the duration of the traffic light in the table, and the traffic light can be marked for inspection. Such a model can also take into account other factors, such as occlusion, false detection, or sun angle, and the pre-stored yellow light duration is generally trusted unless there is a high confidence that the pre-stored duration is no longer reliable, good, usable, etc. In addition or alternatively, the server computing device can broadcast this information to other nearby vehicles or all vehicles in the fleet.

For example, the server computing device 410 may also use the differences to determine if there are any bimodal distributions for a particular traffic light. In other words, the yellow light duration may vary at different times of the day and/or different days of the week (e.g., longer during rush hour or less busy times). In this regard, in some examples, a table for a particular traffic light may include different yellow light durations not only for different states of the traffic light but also for different times of the day and/or different days of the week. Thus, the vehicle's computing device may also use the current time of the day and day of the week to select a default yellow light value. However, in some examples, the boundaries of the yellow light switching duration may be challenging, so some way may be needed to allow for uncertainty about which is correct or to allow for approaching time boundaries. Additionally or alternatively, given enough observations of different durations, a statistical model can be built about when lights change duration.

The features described in this article can enable autonomous vehicles to recognize and respond to different yellow light durations at a single traffic light. This can reduce the likelihood that such vehicles will run red lights or brake suddenly at inappropriate times.

Unless otherwise stated, the above alternative examples are not mutually exclusive, but may be implemented in different combinations to achieve unique advantages. Since these and other variations and combinations of the above features can be used without departing from the subject matter defined in the claims, the above description of the embodiments should be made by way of illustration rather than by way of limitation of the subject matter defined in the claims. In addition, the provisions of the examples described herein and clauses expressed as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only one of many possible embodiments. In addition, the same figure numerals in different figures may identify the same or similar elements.

Claims (18)

1. A method of controlling a vehicle having an autonomous driving mode, the method comprising:

Receiving, by one or more processors of one or more computing devices of the vehicle, a current state of a traffic light at an intersection, the current state of the traffic light indicating whether a currently-lit light of the traffic light is green, yellow, or red;

when the current status of the traffic light indicates that the currently-lit light of the traffic light is green or blinking yellow, selecting, by the one or more processors, a first yellow light duration from a plurality of yellow light durations stored in a pre-stored table associated with the traffic light, the first yellow light duration corresponding to the current status of the traffic light, wherein each yellow light duration stored in the pre-stored table corresponds to a different status of the traffic light;

Predicting, by the one or more processors, when the traffic light will turn red based on the selected first yellow light duration; and

The vehicle in autonomous driving mode is controlled by the one or more processors to stop waiting for traffic lights or continue through the intersection based on when the traffic lights are predicted to turn red.

2. The method of claim 1, further comprising:

receiving, by the one or more processors, an updated status of the traffic lamp; and

Selecting, by the one or more processors, a second yellow light duration from a plurality of yellow light durations stored in a pre-stored table based on the updated status of the traffic light, wherein the first yellow light duration and the second yellow light duration are different.

3. The method of claim 2, wherein selecting a second yellow light duration is independent of any state of the traffic light prior to an updated state of the traffic light.

4. The method of claim 1, further comprising determining a time when the traffic light was last observed in the particular state, and wherein predicting when the traffic light will turn red is further based on the determination of the time when the traffic light was last observed in the particular state.

5. The method of claim 4, wherein the particular state is a flashing yellow light.

6. The method of claim 4, wherein the particular state is a flashing green light.

7. The method of claim 2, wherein a first yellow light duration is used for a normally-on green circle state of the traffic light and a second yellow light duration is used for a green arrow state of the traffic light.

8. The method of claim 2, wherein a first yellow light duration is used for a normally-on green arrow state of the traffic light and a second yellow light duration is used for a blinking yellow arrow state of the traffic light.

9. The method of claim 1, further comprising:

Determining, by the one or more processors, a duration of a yellow light of the traffic light; and

The determined duration is stored for later use.

10. The method of claim 9, further comprising transmitting the determined duration to a remote computing device.

11. A system for controlling a vehicle having an autonomous driving mode, the system comprising:

One or more processors configured to:

receiving a current state of a traffic light at an intersection, the current state of the traffic light indicating whether a currently lit light of the traffic light is green, yellow or red;

when the current status of the traffic light indicates that the currently lit light of the traffic light is green or blinking yellow, selecting a first yellow light duration from a plurality of yellow light durations stored in a pre-stored table associated with the traffic light, the first yellow light duration corresponding to the current status of the traffic light, wherein each yellow light duration stored in the pre-stored table corresponds to a different status of the traffic light;

Predicting when the traffic light will turn red based on the selected first yellow light duration; and

The vehicle in autonomous driving mode is controlled to stop waiting for traffic lights or continue through the intersection based on when the traffic lights are predicted to turn red.

12. The system of claim 11, further comprising the vehicle.

13. The system of claim 11, wherein the one or more processors are further configured to:

Receiving an updated state of the traffic light; and

A second yellow light duration is selected from a plurality of yellow light durations stored in a pre-stored table based on the updated status of the traffic light, wherein the first yellow light duration and the second yellow light duration are different.

14. The system of claim 13, wherein the one or more processors are further configured to select a second yellow light duration independent of any state of the traffic light prior to the updated state of the traffic light.

15. The system of claim 13, wherein a first yellow light duration is used for a normally-on green circle state of the traffic light and a second yellow light duration is used for a green arrow state of the traffic light.

16. The system of claim 13, wherein a first yellow light duration is used for a green arrow state of the traffic light and a second yellow light duration is used for a blinking yellow arrow state of the traffic light.

17. The system of claim 11, wherein the one or more processors are further configured to:

determining a duration of a yellow light of the traffic light; and

The determined duration is stored for later use.

18. The system of claim 17, wherein the one or more processors are further configured to transmit the determined duration to a remote computing device.