DE102022106248A1 - System for controlling a camera to support human verification of sensor information - Google Patents

Abstract

A camera control system is provided for controlling the operation of a camera (126, 148) mounted on a vehicle (100). The system includes a processor (110) and a memory (115) communicably coupled to the processor. The memory stores a camera control module (141) configured to compare a detection confidence level associated with a feature to a predetermined threshold and in response to the detection confidence level being below the predetermined threshold and using location information associated with the associated with the feature controls operation of the camera (126, 148) to capture at least one image of the feature during movement of the vehicle (100).

Description

The present disclosure relates to interpreting vehicle sensor data to detect a sensed object, and more particularly to human review of sensor data to correlate the data to a detected object for the purpose of updating object detection capabilities of vehicle sensor systems.

Vehicles typically include lidar and radar sensors to detect features in a vehicle's surroundings. The vehicle may also include sensor data processing routines and hardware configured to interpret the sensor data to attempt to recognize and/or categorize sensed features, for example by comparing to stored information describing known and previously encountered features. However, some detected features may not be recognizable by the sensor data processing routines, for example because the feature has never been encountered before, because of the state of the detected feature, or because an angle at which the feature is "viewed" by the sensor makes it difficult to accurately compare the processed data to known characteristics. Software also exists for generating two-dimensional and three-dimensional representations of sensor data (e.g., point clouds) for human review. However, due to limitations of such software, the precise identity of a detected object may be unclear even to a human examiner examining the generated representation of the feature.

In one aspect of the embodiments described herein, a camera control system for controlling operation of a camera mounted on a vehicle is provided. The system includes a processor and a memory communicably coupled to the processor. The memory stores a camera control module that includes computer-readable instructions that, when executed by the processor, cause the processor to compare a detection confidence level associated with a feature in an area surrounding the vehicle to a predetermined threshold and, in response, the detection confidence level is below the predetermined threshold and, by using location information associated with the feature, controls operation of the camera to capture at least one image of the feature during movement of the vehicle.

In another aspect of the embodiments described herein, a method for controlling operation of a camera mounted on a vehicle is provided. The method includes the steps of: comparing a recognition confidence level associated with a feature to a predetermined threshold and, in response to the recognition confidence level being below the predetermined threshold, and using location information associated with the feature, controlling operation the camera to capture at least one image of the feature during movement of the vehicle.

In another aspect of the embodiments described herein, a non-transitory computer-readable medium for controlling operation of a camera mounted on a vehicle is provided. The medium has instructions stored thereon which, when executed by a computer system, cause the computer system to perform functions including: comparing a detection confidence level associated with a feature to a predetermined threshold and, in response to the detection confidence level is below the predetermined threshold, and by using location information associated with the feature, controlling operation of the camera to capture at least one image of the feature during movement of the vehicle.

• 1 ist ein schematisches Blockdiagramm eines Fahrzeugs, das ein Kamerasteuersystem gemäß hierin beschriebenen Ausführungsbeispielen enthält.
• 2 ist ein Ablaufdiagramm, das den Betrieb eines Kamerasteuersystems gemäß einem hierin beschriebenen Ausführungsbeispiel darstellt.
• 3 ist eine schematische Draufsicht eines Fahrzeugs, das sich während des Betriebs des Kamerasteuersystems gemäß einem hierin beschriebenen Ausführungsbeispiel entlang einer Straße bewegt.

The accompanying drawings, which are incorporated in and form a part of the specification, illustrate various systems, methods, and other embodiments of the disclosure. It should be understood that the illustrated entity boundaries (eg, boxes, groups of boxes, or other shapes) in the figures represent an embodiment of the boundaries. In some embodiments, an element can be designed as multiple elements or multiple elements can be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component and vice versa. Additionally, elements may not be drawn to scale.

• 1 1 is a schematic block diagram of a vehicle including a camera control system according to embodiments described herein.
• 2 FIG. 12 is a flow chart illustrating the operation of a camera control system according to an embodiment described herein.
• 3 12 is a schematic plan view of a vehicle that moves during operation of the camera control system according to a herein described embodiment moves along a road.

Embodiments described herein relate to a camera control system for controlling operation of a camera mounted on a vehicle. The system includes a processor and a memory communicably coupled to the processor. The memory stores a camera control module configured to compare a recognition confidence level associated with a feature to a predetermined threshold. In response to the detection confidence level being below the predetermined threshold and using location information associated with the feature, operation of the camera is controlled to capture at least one image of the feature as the vehicle moves. The captured image can be temporally correlated with lidar and/or radar data regarding the detected feature and obtained simultaneously with the camera image. If the confidence level is below the predetermined threshold, the time-correlated camera images and processed lidar/radar information can be forwarded to a human examiner. The human inspector can compare and correlate the lidar/radar information with camera images of the actual feature represented in the lidar/radar information. When the lidar/radar information is categorized or associated with an actual known feature, the newly obtained lidar/radar information related to the feature can be added to a catalog of known lidar/radar information associated with the feature, thereby expanding the resource database , which is used for comparison with newly acquired lidar/radar information. This may facilitate future detection of that feature or a similar feature in other vehicles using the same database as a source of known lidar/radar information for comparison.

1 10 shows a schematic block diagram of a vehicle 100 including a camera control system, according to embodiments described herein. The camera control system may be configured to control the operation of an inspection support camera 148 mounted on the vehicle 100 . As used herein, a "vehicle" is any form of motorized transportation. In one or more implementations, the vehicle 100 is a conventional passenger vehicle, such as a sedan, pickup truck, or SUV, although the vehicle 100 may be any form of motorized transportation that includes a lidar sensor and/or a radar sensor as part of the vehicle's sensors contains.

The vehicle 100 also includes various elements. It is understood that in various exemplary embodiments it may not be necessary for the vehicle 100 to have all of the 1 has elements shown. The vehicle 100 can be any combination of the various in 1 have elements shown. Furthermore, the vehicle 100 can have additional elements to those in 1 have shown. In some arrangements, the vehicle 100 may be without one or more of the 1 elements shown can be implemented. While the various elements in 1 are shown as being located within the vehicle 100, it is understood that one or more of these elements may be located outside of the vehicle 100.

Some of the possible elements of the vehicle 100 are in 1 shown and described in relation thereto. In addition, it should be understood that for simplicity and clarity of presentation, where appropriate, reference numbers may have been repeated between the different figures to indicate corresponding or analogous elements. In addition, the discussion outlines numerous specific details in order to provide a thorough understanding of the embodiments described herein. However, those skilled in the art will understand that the embodiments described herein can be practiced using various combinations of these elements.

In some cases, the vehicle 100 may be configured to selectively switch between an autonomous mode, one or more semi-autonomous operating modes, and/or a manual mode. Such switching can be implemented in any suitable manner now known or later developed. "Manual Mode" means that all or a majority of the navigation and/or maneuvering of the vehicle is performed in accordance with inputs received from a user (e.g., a human driver). In one or more arrangements, the vehicle 100 may be a conventional vehicle configured to operate in a manual mode only.

In one or more embodiments, vehicle 100 is an autonomous vehicle. As used herein, “autonomous vehicle” refers to a vehicle that can be operated in an autonomous mode. “Autonomous mode” refers to navigating and/or maneuvering the vehicle 100 along a route through the use of one or more computers systems to control the vehicle 100 with minimal or no input from a human driver. In one or more embodiments, the vehicle 100 is highly automated or fully automated. In one or more arrangements, the vehicle 100 is configured with one or more semi-autonomous modes of operation in which one or more computing systems perform a portion of navigating and/or maneuvering the vehicle along a travel route, and a vehicle operator (ie, driver) provides inputs to the vehicle ready to perform a portion of navigating and/or maneuvering the vehicle 100 along the route.

Embodiments of the camera control module 141 described herein can autonomously control the operation of the inspection support camera 148 for the purposes described herein, both when the vehicle is operating in a manual mode and in an autonomous mode. The camera control module 141 may interface with the autonomous driving module 160 to facilitate autonomous control of the camera 148 as needed.

The vehicle 100 may include one or more processors 110 . In one or more arrangements, the processor(s) 110 may be a main processor(s) of the vehicle 100 . For example, the processor(s) 110 may be an electronic control unit (ECU). The vehicle 100 may include one or more data stores 115 for storing one or more types of data. Data store(s) 115 may include volatile and/or non-volatile memory. Examples of suitable data storage 115 include RAM (Random Access Memory), flash memory, ROM (Read Only Memory), PROM (Programmable Read Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory ), registers, magnetic disks, optical disks, hard disks or any other suitable storage medium or any combination thereof. The data store(s) 115 may be a component of the processor(s) 110, or the data store(s) 115 may be operatively connected to the processor(s) 110 for use thereby. The term "operatively connected" as used in this specification may include direct or indirect connections, including connections without direct physical contact.

The one or more data stores 115 may contain sensor data 119 . In this context, “sensor data” means any information about the sensors that the vehicle 100 is equipped with, including capabilities and other information about such sensors. As discussed below, the vehicle 100 may include the sensor system 120 . The sensor data 119 can relate to one or more sensors of the sensor system 120 . As an example, sensor data 119 may include operational information and specifications related to radar sensor(s) 109, lidar sensor(s) 125, and camera(s) 126 of sensor system 120 in one or more arrangements. The data store(s) 115 may be operatively connected to the vehicle wireless communication interface 169 to transmit information to a cloud or other entity outside the vehicle.

In one or more arrangements, the one or more data stores 115 may include map data 189 . Map data 189 may include maps of one or more geographic areas. In some cases, the map data 189 may include information or data regarding roads, traffic control devices, road markings, structures, points of interest, and/or landmarks in the one or more geographic areas. Map data 189 may be in any suitable form. In some cases, the map data 189 may include aerial photos of an area. In some cases, the map data 189 may include ground views of an area, including 360 degree ground views. Map data 189 may include measurements, dimensions, distances, and/or information for one or more items included in map data 189 and/or relating to other items included in map data 189 . Map data 189 may include a digital map with information about road geometry, road grades and gradients, and other road information, including information about the road on which vehicle 100 is currently traveling. The map data 189 can be of high quality and/or very detailed. In one or more arrangements, the map data 189 may be updated continuously (i.e., as information regarding revisions becomes available) or periodically from a cloud source or other off-vehicle source. Information from the map data 189 can be used to determine location information related to features of the surrounding area captured by lidar and radar scans.

Camera image information 165 may include images (such as digital images) of features of the environment captured by a camera (such as inspection support camera 148) and/or generated from data captured by the camera be obtained. The camera image information 165 can include camera image data from which camera images can be generated that can be viewed by a human reviewer. The camera control module 141 may include computer-readable instructions that, when executed by the processor, cause the processor to temporally correlate camera image information 165 with simultaneously acquired lidar scan information and/or radar scan information (e.g., camera image data may be temporally correlated with lidar data and/or radar data are linked, which are obtained simultaneously with the lidar and/or radar data). Thus, for example, a camera image taken by the verification support camera 148 at 2:05 p.m. during a daytime drive with the vehicle can be time-correlated with a lidar scan performed by a lidar sensor 125 at 2:05 p.m. during the daytime drive .

The lidar scan information 118 may include scan data captured by a lidar sensor 125 during a lidar scan, location information describing locations of features recognized by lidar scans, computer generated identifications and/or classifications of the features, confidence levels in relation to the feature identifications, information describing point clouds generated from associated lidar scans, dates and times of day when the associated lidar scans are performed, information describing known or identified point cloud configurations for the purpose of comparison with newly obtained scan data and associated point clouds, and other information related to lidar scans performed by a lidar sensor 125 . As previously indicated, the lidar scan information may be temporally correlated with simultaneously obtained camera image information and/or radar scan information (i.e., lidar data may be temporally linked to camera images and/or radar data that are obtained simultaneously with camera images and/or radar data ). Using the data collected from lidar and/or radar scans, the processor(s) 110 may generate point clouds representative of features detected in the vehicle's surroundings.

The radar scan information 123 may include scan data captured by a radar sensor 109 during a radar scan, location information describing locations of features captured by radar scans, computer generated identifications and/or classifications of the features, confidence levels in relation to the feature identifications, information describing point clouds generated from associated radar scans, dates and times of day when associated radar scans are performed, information describing known or identified point cloud configurations for the purpose of comparison with newly acquired scan data and describe associated point clouds and other information related to radar scans performed by a radar sensor 109 . As previously indicated, the radar scan information may be temporally correlated with simultaneously obtained camera image information and/or radar scan information (i.e., lidar data may be temporally linked to camera images and/or radar data that are obtained simultaneously with camera images and/or radar data .

Using the data obtained from lidar and/or radar scans, the processor(s) 110 can generate point clouds representative of features detected in the vehicle environment in a known manner.

As previously noted, the vehicle 100 may include the sensor system 120 . Sensor system 120 may include one or more sensors. “Sensor” means any device, component and/or system that can sense and/or measure something. The one or more sensors may be configured to sense and/or measure in real time. As used herein, the term "real-time" means a degree or measure of processing responsiveness that a user or system perceives as sufficiently immediate for a particular process or determination to be performed, or that allows the processor to interact with an external process Keep up. In arrangements where the sensor system 120 includes a plurality of sensors, the sensors can operate independently of one another. Alternatively, two or more of the sensors can work in combination. In this case, the two or more sensors can form a sensor network. The sensor system 120 and/or the one or more sensors may be operatively connected to the processor(s) 110, the data storage(s) 115, and/or other element(s) of the vehicle 100 (including any of the elements in 1 are shown).

Sensor system 120 may include any suitable type of sensor. Various examples of different types of sensors are described herein. However, it goes without saying that the exemplary embodiments are not limited to the specific sensors described. Various examples of sensors of the sensor system 120 are described herein. The example sensors may be part of the one or more environmental sensors 122 and/or the one or the multiple vehicle sensors 121 . However, it goes without saying that the exemplary embodiments are not limited to the specific sensors described. Sensor system 120 may include any sensors suitable and/or required to perform any of the data acquisition and/or vehicle control operations contemplated herein.

Sensors of the sensor system 120 may be communicably coupled to the various systems and components of the vehicle 100 . The sensors may be operatively connected to the vehicle wireless communication interface 169 to transmit information to a cloud or other storage device or for vehicle-to-vehicle (V2V) or vehicle-to-everything (V2X) communications. The sensors may also be operatively connected to other vehicle systems and components, such as data storage 115 and processor(s) 110 to facilitate storage and processing of vehicle and environmental sensor data. The presence of the various conditions described herein may be calculated or otherwise determined using sensor data.

Sensors of sensor system 120 may include (or be operatively associated with) one or more timers or clocks (not shown) configured to obtain, track, store, generate, and/or process time-correlated sensor data and other information as described herein enable. This time-correlated information can be provided to a human inspector to assist the inspector in associating features detected by lidar and radar scans with physical features shown in associated camera images.

Sensor system 120 may include one or more vehicle sensors 121 . The vehicle sensor(s) 121 may sense, determine, and/or measure information about the vehicle 100 itself and/or occupants within the vehicle. Vehicle sensor(s) 121 may include sensors configured to sense conditions and/or events within the vehicle interior or cabin. In one or more arrangements, the vehicle sensor(s) 121 may be configured to detect and/or measure changes in position and orientation of the vehicle 100, such as, for example, based on inertial acceleration. In one or more arrangements, the vehicle sensor(s) 121 may include one or more accelerometers, one or more gyroscopes, an inertial measurement unit (IMU), a dead reckoning system, a global navigation satellite system (GNSS), a global positioning system (GPS), a Navigation system 147 and/or other suitable sensors included. The vehicle sensor(s) 121 may be configured to sense and/or measure one or more characteristics of the vehicle 100, such as the current geographic location of the vehicle. In one or more arrangements, the vehicle sensor(s) 121 may include a tachometer to determine a current speed and acceleration/deceleration of the vehicle 100 . Vehicle sensor(s) 121 may include vehicle direction sensors (not shown) configured to determine a current direction of travel of the vehicle or a direction that the vehicle is pointing. Vehicle sensor(s) 121 may include sensors configured to sense aspects of the vehicle's mechanical and electrical components and/or systems to assist in determining a mechanical condition of the vehicle and existing and/or potential issues to help with the vehicle.

The sensor system 120 may include one or more environmental sensors 122 configured to obtain data from at least a portion of the vehicle's 100 external environment (e.g., nearby objects). The environmental sensors 122 may collect data or information about the external environment in which the vehicle is located, or one or more portions thereof. For example, the environmental sensors 122 may be configured to sense, quantify, and/or measure obstacles in at least a portion of the vehicle's 100 exterior environment and/or information/data about such obstacles. Such obstacles can be stationary objects and/or dynamic objects. The one or more environmental sensors 122 may be configured to sense, determine, quantify, and/or measure other things in the external environment of the vehicle 100, such as lane markers, signs, traffic lights, traffic signs, lane lines, crosswalks, curbs near vehicle 100, off-road objects, weather conditions, other vehicles, etc. As an example, environmental sensors 122 may include, in one or more arrangements, one or more radar sensors 109, one or more lidar sensors 125, one or more cameras 126 (including inspection support camera 148 ) and/or other types of sensors 127 included.

Surrounding sensors such as lidar sensor(s) 125 and radar sensor(s) 109 may be operable to attempt to sense and recognize features in the vehicle's surroundings. Radar 109, lidar 125, camera(s) 126, and/or other sensors may also be usable to monitor the speed, acceleration, position, and other properties of features in the vicinity of vehicle 100, such as signage, foliage , other vehicles, pedestrians, etc. The environmental sensors 122 may be configured to sense aspects of road geometry, road quality/grade, traffic conditions, movements and characteristics of other vehicles, and other external events and conditions.

In one or more arrangements, the environmental sensors 122 may be configured to determine or obtain (either alone or in cooperation with the navigation system 147 and the sensor control module 142) location information related to features of the vehicle's surroundings sensed by the sensors. This location information may be associated with sensor data to enable determination of the locations of scanned features relative to the location of the vehicle 100 when the sensor data is obtained.

“Location Information” may be information describing the location of a feature, point cloud, or lidar/radar data point generated from lidar data sufficient to enable camera control module 141 (described below) to point verification support camera 148 to the site or the feature to direct and focus. The location information may be in the form of spatial coordinates or in any other form suitable for the purposes described herein. “Spatial Coordinates” refers to a numeric or other designation representing a location of any feature of the vehicle's surroundings (including a feature of the vehicle's surroundings with a low detection confidence level, as described herein). A system of spatial coordinates may be used that provides a location with a low-confidence-of-detection feature that the inspection support camera may attempt to focus the camera on for image capture purposes.

In the case of a low-confidence feature of detection that is static with respect to the surface of the earth, the spatial coordinates may represent a location with respect to a frame of reference (e.g., on the surface of the earth) that may be considered fixed with respect to the moving vehicle . In the case of a low-confidence feature of detection that is moving relative to the surface of the earth, the spatial coordinates may represent a current location of the moving object relative to the vehicle itself. The spatial coordinates may be Cartesian, polar, geographic, and/or any other type or types of coordinates suitable for specifying a location of the vehicle's environmental feature. The spatial coordinates may be associated with a detected feature by the processor(s) 110 operating in conjunction with the vehicle navigation system 147 and/or using map data 189 after a location of the feature with respect to the relevant coordinate system has been determined.

A verification support camera 148 may be a camera configured to capture images of or photograph features of the environment. "Capturing an image" of a feature may include obtaining data from which a digital image can be constructed. "Taking photographs" by the verification support camera may include taking a photograph using a camera (e.g., a digital camera) or obtaining data from which a photograph (such as a digital photograph) can be created.

In one or more arrangements, verification support camera 148 may be a specialized camera dedicated solely to photographing or capturing images of features with a low recognition confidence level, as described herein, for purposes of assisting human verification of information related to the features low detection confidence level. Alternatively, another environmental sensor camera may be reassigned to perform verification support functions as needed and as described herein. In this case, the dedicated verification support camera can be omitted and control of the other camera can be transferred from the sensor control module 142 to the camera control module 141 to capture images that can be used to support human verification of the lidar/radar scan information.

Inspection support camera 148 may include various elements that are configured to be controllable by camera control module 141 (described in more detail below). The verification support camera 148 can be a camera, any camera mounts, motor(s), gears, mechanisms and/or other elements (not separately shown) needed to enable the orientation, field of view, and/or focus of the camera 148 to be adjusted for the purposes described herein. The camera 148 may also include other elements (not shown) as needed.

The field of view of the camera 148 may be a volume of space (relative to a current position of the camera) at which the camera lens may be aimed and focused to enable capturing an image of a feature located within the volume of space. Then, when the camera 148 is controlled to include a feature in the camera's field of view, an image captured by the camera will include an image of the feature. The camera 148 can also be oriented and focused so that its field of view includes a set of spatial coordinates that contain a feature such that a captured image of the coordinates also shows the feature.

Referring again to 1 For example, a sensor fusion algorithm 167 may be an algorithm (or a computing device storing an algorithm) that is configured to accept data from the sensor system 120 as an input. For example, the data may include data representing information measured at sensors of sensor system 120 . For example, the sensor fusion algorithm 167 may include or be configured to be performed using a Kalman filter, Bayesian network, or other algorithm. The sensor fusion algorithm 167 can provide various scores based on the data from the sensor system 120 . Depending on the exemplary embodiment, the assessments can include evaluations of individual objects and/or features in the environment of the vehicle 100, evaluations of specific situations, and/or evaluations of possible effects based on the specific situations. Other ratings are possible. Thus, for example, in one or more arrangements, sensor fusion can be used to attempt to identify a feature from a recently scanned point cloud.

The vehicle wireless communication interface 169 may be configured to enable communication between the vehicle's components and systems and entities (such as cloud facilities, cellular and other mobile communication devices, other vehicles, remote servers, pedestrians, etc.) outside the vehicle and/or or to facilitate. The wireless communications interface 169 may be configured to facilitate, establish, maintain, and terminate wireless V2V and V2X communications with any entity outside the vehicle, for example, other connectably configured vehicles and connected vehicles, pedestrians, servers, and entities that located in the cloud, edge servers and other information sources and entities. Information such as sensor data, camera image information, lidar and radar scan information, and other types of information may be transmitted and received over communication interface 169 . The wireless communication interface 169 may include or communicate with any network interfaces needed to communicate with any entities and/or networks outside of the vehicle.

The vehicle 100 may include an input system 130 . An "input system" includes any device, component, system, element, or assembly, or groups thereof, that enable information/data to be input into a machine. For example, the input system 130 may include a keypad, touch screen or other interactive display, a voice recognition system, and/or any other device or system that facilitates communication between a user and the vehicle. The input system 130 may receive input from a vehicle occupant (e.g., a driver or passenger) or a user remote from the vehicle 100 . The vehicle 100 may also include an output system 135 . An "output system" includes any device, component, or arrangement, or groups thereof, that enable information/data to be presented to a vehicle occupant (e.g., a driver, a vehicle occupant, etc.) or a remote user.

Vehicle 100 may include one or more vehicle systems, collectively referred to as 140 . The vehicle systems 140 may include a navigation system 147 . The navigation system 147 may include one or more devices, applications, and/or combinations thereof, now known or later developed, that are configured to determine the geographic location of the vehicle 100 and/or determine a route for the vehicle 100 to travel . The navigation system 147 may include one or more mapping applications to determine a route for the vehicle 100 to travel. The navigation system 147 may include a global positioning system, a local positioning system, or a geolocation system. The navigation system 147 may be configured to track a vehicle's path along a trip route to follow. The navigation system 147 may be configured to work in conjunction with the autonomous driving module 160 to guide the vehicle along a user selected driving route. The navigation system 147 may be configured to work in conjunction with the camera control module 141 to determine and update location information regarding a feature with a low level of detection confidence, and in guiding and maintaining the direction and focus of the camera 148 toward a feature low detection confidence level during vehicle 100 movement.

The vehicle systems 140 may also include other systems (collectively referred to as 145). For example, the vehicle systems 140 may include a powertrain system, a braking system, a steering system, a throttle system, a suspension system, a transmission system (none of which are explicitly described in 1 is shown). Each of these vehicle systems may include one or more devices, components, and/or a combination thereof that are now known or later developed.

The vehicle 100 may include one or more actuators 150 . The actuators 150 may be any element or combination of elements operable to modify, adjust and/or change one or more of the vehicle systems 140 or components thereof to respond to receiving signals or other inputs from the/ the processor(s) 110, any modules stored in the memory 112, and/or any other vehicle components or systems. Any suitable actuator can be used. For example, the one or more actuators 150 may include motors, pneumatic actuators, hydraulic pistons, relays, solenoids, and/or piezoelectric actuators, to name a few possibilities.

In embodiments described herein, a memory 112 may store an autonomous driving module 160 , a sensor control module 142 , a feature recognition module 143 , and a camera control module 141 . Memory 112 may be random access memory (RAM), read only memory (ROM), hard disk drive, flash memory, or other suitable memory for storing modules 160, 141, 142, and 143. Modules 160, 141, 142, and 143 are, for example, computer-readable instructions that, when executed by processor(s) 110, cause processor(s) 110 to perform the various functions disclosed herein. Additional modules (not shown) may also be stored in memory 112 .

The vehicle 100 may include one or more modules, at least some of which are described herein. The modules may be implemented as computer-readable program code that, when executed by the processor(s) 110, implements one or more of the various processes described herein. One or more of the modules may be a component of processor(s) 110, or one or more of the modules may be executed and/or distributed on other processing systems to which processor(s) 110 is operatively connected . The modules may contain instructions (e.g., program logic) executable by processor(s) 110 . Alternatively or additionally, one or more data stores 115 or another portion of the vehicle 100 may contain such instructions.

In general, as used herein, a module contains routines, programs, objects, components, data structures, and so on that perform specific tasks or implement specific data types. In further aspects, a memory generally stores the mentioned modules. The memory associated with a module may be a processor embedded buffer or cache, RAM, ROM, flash memory, or other suitable electronic storage medium. In further aspects, a module as contemplated by the present disclosure is embodied as an application specific integrated circuit (ASIC), a hardware component of a system on a chip (SoC), a programmable logic array (PLA), or other suitable one Implements a hardware component embedded with a defined set of configurations (e.g. instructions) for performing the disclosed functions.

In one or more arrangements, one or more of the modules described herein may include elements of artificial or computational intelligence, e.g., neural networks, fuzzy logic, or other machine learning algorithms. Furthermore, in one or more arrangements, one or more of the modules may be distributed among a plurality of the modules described herein. In one or more arrangements, two or more of the modules described herein can be combined into a single module.

The vehicle 100 may include one or more autonomous driving modules 160 . The autonomous driving module(s) 160 may be configured to receive data from the sensor system 120 and/or any other type of system capable of providing information regarding the vehicle 100 and/or the external environment of the vehicle 100 record. The Autono mes driving module(s) 160 may determine position and speed of the vehicle 100 . The autonomous driving module(s) 160 can determine the location of obstacles, obstacles, or other environmental features including traffic signs, trees, shrubs, neighboring vehicles, pedestrians, and so forth. The autonomous driving module(s) 160 may be configured to obtain location information for obstacles within the external environment of the vehicle 100 for use by the processor(s) 110, and/or one or more of those described herein Modules for estimating the position and orientation of the vehicle 100, the vehicle's position in global coordinates based on signals from a plurality of satellites, or any other data and/or signals that could be used to determine the current state of the vehicle 100 or the determine, receive and/or determine position of the vehicle 100 in relation to its surroundings for use in either creating a map or determining the position of the vehicle 100 in relation to map data.

The autonomous driving module(s) 160 may be configured to determine driving path(s), current autonomous driving maneuvers for the vehicle 100, future autonomous driving maneuvers, and/or modifications to current autonomous driving maneuvers based on data received from sensor system 120 and/or information received from a navigation system such as navigation system 147 . “Driving Manoeuvre” means one or more actions that affect the movement of a vehicle. Examples of driving maneuvers include: accelerating, decelerating, braking, turning, moving in a lateral direction of the vehicle 100, changing lanes, merging into a lane, and/or reversing, to name a few possibilities. The autonomous driving module(s) 160 may be configured to implement particular driving maneuvers. The autonomous driving module(s) may directly or indirectly cause such autonomous driving maneuvers to be implemented. As used herein, "causing" means that an event or action is brought about, compelled, caused, directed, commanded, instructed and/or enabled, or at least is in a state in which such event or action will occur or be caused. can occur, either directly or indirectly.

The autonomous driving module(s) 160 may be configured to perform various vehicle functions and/or transmit data to/from the vehicle 100 or one or more systems thereof (e.g., one or more vehicle systems 140). , receive, interact and/or control. The autonomous driving module(s) 160 may be configured to autonomously control the user's vehicle to drive the vehicle along a driving route from an initial or starting location to a destination.

The processor(s) 110, autonomous driving module 160, camera control module 141, and/or other modules described herein may be operatively linked to communicate with each other and with the other elements of the vehicle, including various vehicle systems 140 and/or or individual components thereof. For example, regarding 1 the processor(s) 110 and the autonomous driving module(s) 160 are in communication to send and/or receive information from the various vehicle systems 140 to determine movement, speed, maneuvering, to control the course, direction, etc. of the vehicle 100 . The processor(s) 110, the autonomous driving module(s) 160, and/or other elements of the vehicle may control some or all of the vehicle systems 140 and thus may be partially or fully autonomous.

The processor(s) 110, the autonomous driving module(s) 160, and the navigation system 147 may be operable to provide navigation and/or maneuvering of the vehicle 100 by controlling one or more of the To control vehicle systems 140 and / or its components. For example, when operating in an autonomous mode, the processor(s) 110 and/or autonomous driving module(s) 160 may control the direction and/or speed of the vehicle 100 . The processor(s) and/or autonomous driving module(s) 160 may cause the vehicle 100 to accelerate (e.g., by increasing the fueling provided to the engine), decelerate (e.g., by decreasing the fuel supply provided to the engine and/or by applying the brakes) and/or changes direction (e.g. by turning or steering the two front wheels).

The vehicle 100 may include a sensor control module 142 . The sensor control module 142 may include computer-readable instructions that, when executed by the processor(s) 110, cause the processor(s) to control operation of the vehicle sensors 121 and/or the environmental sensors 122. For example, the sensor control module 142 may be configured to control operations of one or more sensors to vary sensor scan rates, vary scan or viewing angles, and control other sensor operating parameters according to existing conditions and detection requirements. Alternatively, one or more of the vehicle sensors 121 and/or the environmental sensors 122 may be configured for automatic or internal control by incorporating independent processors and/or memory therein, thereby avoiding the need for separate control for the relevant sensor(s). becomes.

In one or more particular arrangements, the sensor control module 142 may be configured to control operations of one or more of the environmental sensors 122 other than the verification support camera 148 , operations of which may be controlled by the camera control module 141 . In other particular arrangements, the control functions of the camera control module 141 as described herein may be incorporated into (and performed by) the sensor control module 142.

The vehicle 100 may include a feature recognition module 143 . Feature detection module 143 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to extract lidar and radar scan data and/or other information obtained from the lidar and/or or generated, and/or process radar scans of the environment to attempt to detect or identify features of the vehicle environment represented by the data and/or other information.

The resulting feature identifications can then be communicated to a human driver to support the driver's decision making. The resulting feature identifications can also be used or further processed by other modules and/or other vehicle systems for autonomous decision-making.

The vehicle environment may include everything outside of the vehicle interiors and may be separate from the vehicle 100 . A feature of the vehicle's surroundings can be a discrete object or a portion of a discrete object that is located in the vehicle's surroundings. The feature may or may not be identified or detected through processing of lidar and/or radar scan data by the feature detection module 143 . Upon receiving data from the lidar scan, the feature detection module 143 (or other module), in cooperation with the processor(s) 110, may process the lidar and/or radar data to generate one or more point clouds representing representations of scanned features included. The feature recognition module 143, in cooperation with the processor(s) 110, may then execute feature recognition routines aimed at identifying or recognizing the feature represented by the point clouds. The resulting feature identifications can then be communicated to a human driver to support the driver's decision making. The resulting feature identifications can also be used or further processed by other modules and/or other vehicle systems for autonomous decision-making. For example, the feature detection module 143 may be configured to compare point clouds generated from lidar and/or radar scan data to point clouds and/or other information representing known features that may be present in a vehicle environment. Information regarding the known features may be stored in memory (such as lidar scan information 118, radar scan information 123, and/or other memory configured to be accessible to feature detection module 143).

The feature detection module 143 may include computer-readable instructions that, when executed by the processor(s), cause the processor(s) to cooperate with the environmental sensors 122 and the navigation system 147 to obtain location information regarding the features of the identified To determine the environment through the feature recognition module. The feature locations may be determined using location information stored in lidar scan information 118 and/or radar scan information 123 obtained during lidar and/or radar scans. Sensor scan rates can be sufficiently high so that once a particular feature is identified, it can be reliably tracked and monitored for changes in its location, detection confidence level, and other associated properties, and so that feature information stored in memory can be updated.

The feature recognition module 143 may include computer-readable instructions that, when executed by the processor(s) 110, cause the processor(s) to calculate recognition confidence levels for feature identifications generated by the feature recognition module 143. The recognition confidence levels may be stored in memory in association with the feature identification.

Some of the features captured by the lidar scans may not be readily identifiable by the feature detection module 143 due to the angle relative to the vehicle 100 at which the feature was scanned, due to the point cloud representing the feature not being sufficiently similar to the point clouds stored in memory for comparison, due to the scan being partially obstructed and/or due to other reasons. Thus, the feature identifications generated by the feature recognition module 143 may have different or varying degrees of certainty associated with them. Accordingly, the feature recognition module 143 may be configured to calculate a confidence level (CL) to be associated with the identification of each feature. The confidence level may be a measure of confidence or certainty that the feature identification module 143's identification of the feature is correct. For example, a review of a point cloud generated from sourced data when a feature is at an unusual viewing angle (due to strong winds or damage to the scanned feature) may result in a relatively lower calculated confidence level based on a computer-generated recognition of the feature than checking the same feature under ideal sensor scanning conditions.

A “detection confidence level” or “confidence level” is a numerical value or other indicator of a level of confidence in a computer-generated recognition of a feature of the vehicle's surroundings. A recognition confidence level indicator may be a numeric indicator determined by a processor and adapted for comparison to a numerical predetermined feature recognition confidence level threshold. The feature detection module may be configured to assign a detection confidence level (CL) to a computer-generated identification of a feature detected by a lidar or radar scan. A "low detection confidence level" refers to a detection confidence level that is below the predetermined threshold. A "feature of low detection confidence level" is a feature of the environment that has a detection confidence level that is below the predetermined threshold.

For the purposes described herein, the detection confidence levels can be based on any number of criteria. For example, in one or more arrangements, the confidence levels may be calculated based at least in part on a comparison between information derived from a series of consecutive lidar/radar scans taken during vehicle movement and one or more reference point clouds when a distance from the vehicle to the scanned feature decreases and/or as an angle at which a portion of the sensor scan wave contacts the feature changes during vehicle movement. This “scan angle” can be linked to how the feature appears visually when viewed from the perspective of the moving vehicle. If the comparison shows that the scanned feature appears increasingly similar to the feature represented by the reference point cloud as the vehicle moves, a relatively higher level of confidence can be calculated and assigned to the feature identification. Any of a plurality of additional or alternative criteria may be used in an algorithm configured to calculate a confidence level to be associated with a feature identification.

The feature recognition module 143 may include computer-readable instructions that, when executed by the processor(s) 110, cause the processor(s) to compare a confidence level of the feature identification to a predetermined threshold. If the level of confidence in a given feature identifier is above the threshold, the feature identifier determined by the feature recognition module may be used by the vehicle systems in performing further calculations and functions. However, if the confidence level in a given feature identification is at or below the threshold, low confidence images of the feature may be captured as described herein for review by a human reviewer.

Feature recognition module 143 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to obtain location information related to identified features, associated confidence levels, information related to feature classifications (e.g., "foliage" or . foliage”) and other information related to the scans of captured environmental features. For example, such information may be stored in memory, such as lidar scan information 118 (for information related to lidar scans).

Feature detection module 143 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to update feature information (in memory) based on recent lidar/radar scans. Such updated Feature information may include revised recognition confidence levels, updated feature location information, updated point cloud information representing the feature, and any other information related to the feature. In one or more arrangements, such information may be updated in memory (e.g., lidar scan information 118) where the information is stored.

The vehicle 100 may include a camera control module 141 . Camera control module 141 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to compare a recognition confidence level associated with a feature to a predetermined threshold, and in response thereto that the detection confidence level is below the predetermined threshold and using location information associated with the feature, control operation of the camera to capture at least one image of the feature while the vehicle is moving.

Camera control module 141 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to control operation of inspection support camera 148 to capture images of features surrounding the vehicle concurrently with operation of the lidar sensor 125 record. The camera control module may also include computer-readable instructions that, when executed by the processor, cause the processor to control operation of the vehicle camera to capture images of the feature concurrently with operation of a radar sensor 109 . These simultaneous operations of the inspection support camera 148 and the radar and lidar sensors allow the information obtained from these sensors to be correlated and associated in time.

The camera control module 141 may contain computer-readable instructions that, when executed by the processor(s) 110, cause the processor(s) to control all inspection support camera operations, including movement, pointing, focusing, and image capture in response to vehicle movement direction and speed , location information related to low-confidence features of the environment to be imaged, and other relevant information.

Camera control module 141 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to generate lidar data and/or radar data (i.e., data obtained during scans from the lidar and radar sensors temporally correlated with at least one simultaneous image of the feature captured by camera 148 (i.e., an image captured at the same time as the lidar/radar scan data was acquired by the lidar scan and/or radar -Scan) and vice versa. This temporal correlation can be used to associate lidar and radar scans with simultaneously captured images for human review. For example, a camera image captured by the verification support camera 148 at 2:05 p.m. during a vehicle drive may be combined with a lidar scan performed by a lidar sensor 125 at 2:05 p.m Easier to review lidar data and associated camera images.

The camera control module 141 may include computer-readable instructions that, when executed by the processor(s) 110, cause the processor(s) to temporally correlate lidar data with at least one simultaneous image of the feature captured by the vehicle camera.

The camera control module may also include computer-readable instructions that, when executed by the processor(s) 110, cause the processor(s) to temporally correlate radar data with at least one simultaneous image of the feature captured by the camera.

The camera images may be temporally correlated with simultaneously acquired lidar scan data and/or radar scan data (e.g., camera images may be temporally linked with lidar and/or radar data acquired at the same time as the lidar and/or radar data).

Camera control module 141 may contain computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to cooperate with navigation system 147 to locate a feature or geographic or spatial location during movement of the To “track” or stay focused on vehicle 100 . The ability to operate the camera 148 to maintain focus on a feature or area may depend on the degree to which the camera 148 can be panned, translated, raised, and depressed and/or the degree to which the lens may be directed and the degree to which it may be refocused, etc. It is desirable to maintain the feature of the vehicle's surroundings in the current field of view of the camera 148 while the vehicle 100 moved to enable high frequency acquisition of images of features while the vehicle is moving.

In one or more arrangements, for the purpose of accessing feature information needed to control the operation of the inspection support camera 148, the camera control module 141 may include computer-readable instructions that, when executed by the processor(s) 110, the Processor(s) access memory (e.g., lidar scan information 118 or radar scan information 123) storing feature information for feature location information, confidence levels, and other information. Accessing location information related to a feature with low confidence may allow the camera control module 141 to control the camera 148 to capture images of the feature by focusing the camera on the feature or on a location where the feature is located .

Camera control module 141 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to update feature information either periodically or in response to updates in lidar/radar scan information access a feature that was previously determined to be a low-confidence feature and that is currently selected for imaging by the camera 148 . This can be done to determine if the level of confidence in the feature has changed between sensor scans.

Camera control module 141 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to determine the current detection confidence level (i.e., the confidence level obtained using data from the last sensor scan is determined) compare with a feature to a predetermined threshold value. This comparison may allow the camera control module 141 to determine whether the confidence level has risen above the predetermined threshold based on an interpretation of the latest radar/lidar scan data (for example, due to a change in a sensor's scan angle caused by movement of the Vehicle is caused) by comparing the current detection confidence level with the predetermined threshold. Based on this comparison, the camera control module 141 can control the camera 148 to either stop attempting to capture images of the feature (when the detection confidence level is at or above the threshold) or to continue attempting to capture images of the feature (when the camera is currently attempts to capture images of the feature, the current detection confidence level associated with the feature is low, and the last determined detection confidence level was also low).

Camera control module 141 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s), in response to the detection confidence level being below the predetermined threshold, and using the location information associated with the feature control operation of the camera 148 to capture at least one image of the feature as the vehicle 100 moves. Thus, when the confidence level associated with a feature is first determined to be below the threshold, the camera control module 141 can control the verification support camera 148 to begin capturing images of the feature. Additionally, as long as the confidence level associated with the feature remains below the threshold, the camera control module 141 may control the verification support camera 148 to continue attempting to capture images of the feature.

The camera control module 141 may include computer-readable instructions that, when executed by the processor(s) 110 (the processor(s)), in response to the detection confidence level being at or above the predetermined threshold, cause the operation of the camera 148 control to stop attempting to capture images of the feature. Thus, when the feature detection module 143 has identified a feature with a sufficient level of confidence, there may no longer be a need for human inspection of the feature, and the inspection support camera 148 can be controlled to stop capturing images of the feature.

Camera control module 141 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s) to determine whether a feature is currently in a camera lockout location after camera 148 takes at least one image of the feature, and in response to the feature being in a camera lockout location, control operation of the camera 148 to discontinue attempting to capture images of the feature.

A low-confidence feature may be determined to be in a "camera cut-off location" if the feature with a low detection confidence level is no longer in a position relative to the vehicle 100 such that the camera 148 can be controlled to include the feature in a field of view of the camera (ie, the camera 148 can no longer be panned, elevated , lowered, moved, or otherwise controlled to bring the feature into their field of view).

Camera control module 141 may include computer-readable instructions that, when executed by processor(s) 110, cause the processor(s), in response to a feature being in a camera exclusion zone, after vehicle camera 148 at least captured a photograph of the feature, control operation of the vehicle 100 to transmit feature information to an entity external to the vehicle for human review. In one or more arrangements, the off-vehicle unit may include a human verification facility 196 ( 1 ) be. The human verification facility 196 may include a human verification manager 195 configured to allocate or allocate feature information received from various vehicles for human verification by human verifiers HR1, HR2, HR3, and so on. In certain arrangements, the feature information may be relayed to the human reviewer 196 in real time once the attempted image capture by the camera 148 is discontinued. This allows for quick review of the feature information and classification of the feature with low recognition confidence.

In certain arrangements, the feature information may include point cloud data representing the feature with a low level of confidence and one or more time-correlated images of the feature captured by camera 148 . Human reviewer 196 may include hardware and software needed to process point cloud data to generate two-dimensional or three-dimensional graphical representations of the feature represented by the point cloud for comparison by a human reviewer to one or more associated camera images.

In one or more arrangements, after the feature has been classified by a human examiner with a low recognition confidence level, the identification, the point cloud(s) associated with the feature and other relevant information can be forwarded to a server facility or other entity 197 where the new feature information may be incorporated into a feature identification system (not shown) configured to facilitate analysis and identification of point cloud information. The feature identification system may include known reference point clouds for comparison to lidar and radar scan data obtained from other vehicles 198, 199 using the feature identification system as a source of known reference point clouds. The other vehicles 198, 199 can then have access to the new feature information for comparison to point clouds generated from lidar and radar scans performed by those vehicles. The new feature information can be stored in the server facility 197 for remote access by the other vehicles 198, 199 as needed, or the new feature information can be propagated to the vehicles 198, 199 for local use as needed.

2 FIG. 12 is a flow chart illustrating operation of a camera control system in accordance with an embodiment described herein. 3 12 is a schematic plan view of a vehicle 100 moving along a road 153 in direction S1 during operation of the camera control system. 3 shows the movement of vehicle 100 as it moves along road 153 ( 3 ) moved from position "A" to position "D". Although the operation of the verification assist camera control system is described in relation to augmenting data obtained from a lidar sensor 125, it should be understood that the following description may also apply to use with data obtained from a radar sensor or any other sensor that obtains point cloud data or other data that can be processed to identify or recognize features in the vehicle environment.

While moving along Road 153 ( 3 ), the sensor control module 142 may control operation of the environmental sensors 122 to survey the vehicle's surroundings for possible threat detection, to determine a position of the vehicle relative to features in the surroundings, and for other purposes. The vehicle environment may contain various features that are detectable by a lidar scan, such as road signs or traffic lights, foliage, pedestrians, other vehicles, etc. Along the path of the vehicle 100, an unrecognized feature 99 or a feature with low recognition confidence level set back from an edge 153a of the road 153. The feature can through a portion 125a of the lidar scan can be captured when the vehicle 100 is in position "A".

In relation to 2 and 3 the sensor control module 142 in position "A" (in block 302, 2 ) control the operation of a lidar sensor 125 to scan the vehicle's surroundings. The feature 99 can be detected by a portion 125a of the lidar scan. Lidar scans of the vehicle's surroundings can last at a specified scan rate.

Upon receiving data from the lidar scan, the feature detection module 143 (or other module), in cooperation with the processor(s) 110, may process the lidar data to generate one or more point clouds containing representations of scanned features. The feature recognition module 143 can then (in block 304) execute routines directed to identifying detected features, including feature 99. The feature recognition module 143 can then calculate recognition confidence levels for the feature identifications, including a confidence level for the identification of feature 99. The feature recognition module 143 may then (in block 306) store feature information regarding all detected features (including feature 99) in memory (such as lidar scan information 118).

While the vehicle 100 is still in position "A" (or in a position between the moving vehicle's position "A" and position "B"), the camera control module 141 may access (in block 307) the stored feature information in memory to determine whether a detection confidence level of any detected feature is below the predetermined threshold. in the in 2 In the example shown, feature 99 may have a confidence level below the threshold. If none of the stored confidence levels are below the predetermined threshold, control may return to block 302 where the lidar sensor 125 may continue to scan the vehicle environment. However, if any of the stored confidence levels is below the predetermined threshold, the camera control module 141 may access (in block 308) from memory location information describing the location of the low confidence feature. Thus, in the example, the camera control module 141 can access location information describing the location of the feature 99 . The camera control module 141 may then determine (in block 310) whether the low confidence feature 99 is in a camera lockout location.

At this time, the vehicle 100 has moved to the position "B" ( 3 ) emotional. If the feature 99 is in a camera cut-off location with low confidence, the camera 148 will be unable to capture images of the feature. In this case, the camera control module 141 (in block 312 ) cannot control the camera 148 to attempt to capture images of the feature 99 . However, if the feature 99 is not in a camera lockout location with low confidence, the camera control module 141 may (in block 314 ) control operations of the verification support camera 148 to begin capturing images of the feature 99 . A line of sight from camera 148 to feature 99 is in 3 shown as 148a. After the camera 148 focuses on the feature 99, the camera control module 141 may also (in block 314) temporally correlate each captured image of the feature 99 with a simultaneous lidar scan that occurs after the camera 148 begins capturing images. At the same time, the processes of lidar scanning (block 316) and feature identification/confidence level determination (block 318) continue while images of feature 99 are acquired. Feature information regarding feature 99 is also updated in lidar scan information 118 (block 320) as a new scan is performed and new scan data is processed.

In response to each feature information update, the camera control module 141 may access (in block 322 ) the confidence level associated with the feature 99 . The camera control module may then determine (in block 324 ) whether the low confidence feature 99 is currently in a camera lockout location relative to the vehicle 100 . If the low confidence feature 99 is not in a camera lockout location, the camera control module 141 may (in block 326) compare the updated confidence level to the predetermined threshold to determine whether the updated confidence level is still below the threshold. The vehicle can now be moved to position "C". 2 be. If the updated confidence level is still below the threshold, control may return to block 314 where the camera control module continues to control the camera 148 to capture images of the feature 99 .

As just described, the camera control module 141 may continue to control the camera 148 to capture images of the feature 99 as long as the confidence level associated with the feature 99 remains below the threshold and as long as the feature 99 is within a field of view of the camera 148 (ie, as long as the feature is not in a camera lockout location).

Returning to block 326 ( 2 ), if the updated confidence level is still not below the threshold, the confidence level has increased to a level at which feature 99 can be considered "recognized" or "identified". Control may then proceed to block 328 where the camera control module 141 may control the camera 148 to stop attempting to capture images of the feature 99 .

Returning to block 324, if feature 99 is in a camera cut-off location with low confidence, camera 148 is no longer able to capture images of the feature. In this case, the camera control module 141 can control (in block 330 ) the camera 148 to stop attempting to capture images of the feature 99 . 2 12 shows the vehicle 100 in a position "D" in which the feature 99 is in an exclusion location from which the camera 148 can no longer focus on the feature 99. FIG. However, the lidar sensor 125 can still detect feature 99 (as well as other vehicle surrounding features) from location "D".

Additionally, since at block 330 the confidence level associated with feature 99 has never reached a level at or above the threshold during image acquisition and feature information update, feature 99 remains classified as "unidentified." Thus, after block 330, the camera control module 141 may perform a final check (at block 332) to determine if any images of the feature 99 have been captured up to this point. If no images were captured, there are no camera images to pass to a human reviewer and the image capture routine may end at block 334 . However, if images of the feature 99 were captured with a low level of confidence, the camera control module 141 may control (in block 336) operation of the vehicle 100 to transmit feature information related to the feature 99 for human review.

In the embodiments described herein, the human examiner may compare and correlate (in real time) the lidar/radar information with associated camera images of the actual feature represented in the lidar/radar information. The provision of camera images of a feature whose identification is being verified by a human can allow faster and more accurate identification of the feature and consequently more accurate correlations between point clouds and the physical objects they represent. Additionally, if the lidar/radar information is categorized or associated with an actual known feature, the newly obtained lidar/radar information related to the feature may be added to a catalog of known lidar/radar information associated with the feature, thereby increasing the resource database used by a computer system to compare with newly acquired lidar/radar information. This may facilitate future detection of that feature or a similar feature in other vehicles using the same database as a source of known lidar/radar information for comparison.

Detailed example embodiments are disclosed herein. However, it should be understood that the disclosed embodiments are intended as examples only. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the aspects herein in virtually any reasonably detailed structure. Furthermore, the terms and expressions used herein are not intended to be limiting, but rather to provide an understandable description of possible implementations. In the 1-4B Various example embodiments are shown, but the example embodiments are not limited to the illustrated structure or application.

The flowcharts and block diagrams in the figures represent the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various exemplary embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code that comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown sequentially may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order depending on the functionality involved.

The systems, components and/or processes described above may be implemented in hardware or a combination of hardware and software and may be implemented in a centralized manner in a processing system or in a distributed manner in which various elements are distributed across multiple interconnected processing systems. Any type of processing system or other device suitable for performing the methods described herein is suitable. A typical combination of hardware and software may be a processing system having computer-usable program code that, when loaded and executed, controls the processing system to perform the methods described herein. The systems, components and/or processes may also be embedded in computer readable storage, such as a computer program product or other data program storage device readable by a machine and tangibly embodying a program of instructions executable by the machine to implement methods and carry out the processes described here. These elements can also be embedded in an application product that includes all the features that enable the implementation of the methods described herein and that, when loaded into a processing system, is capable of executing those methods.

Additionally, arrangements described herein may take the form of a computer program product embodied in one or more computer readable media having computer readable program code embodied thereon, e.g. Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The term "computer-readable storage medium" means a non-transitory storage medium. A computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media would be the following: a portable computer floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a read-only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a compact disc portable read only memory (CD-ROM), a digital versatile disc (DVD), an optical storage device, a magnetic storage device, or any suitable combination of the previously described. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, fiber optic, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for performing operations for aspects of the present arrangements may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java™, Smalltalk, C++ or the like and a conventional procedural programming language such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server . In the latter scenario, the remote computer can be connected to the user's computer over any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be to an external computer (for example, via the internet via an internet service provider).

The terms "a" and "an" as used herein are defined as one or more than one. The term "plurality" as used herein is defined as two or more than two. The term "other" as used herein is defined as at least one second or more. The terms "include/including" and/or "have/comprise" as used herein are defined as including (i.e. open-ended language meaning). The phrase "at least one of ... and ....", as used herein, refers to and includes all possible combinations of one or more of the associated listed items. As an example, the phrase "at least one of A, B, and C" includes only A, only B, only C, or any combination thereof (e.g., AB, AC, BC, or ABC).

Aspects herein may be embodied in other forms without departing from the spirit or essential attributes thereof. Accordingly, reference should be made to the following claims, rather than the foregoing description, for an indication of their scope.

Claims (20)

A camera control system for controlling the operation of a camera (126, 148) mounted on a vehicle (100), comprising: a processor (110); and a memory (115) communicably coupled to the processor (110) and storing a camera control module (141) containing computer-readable instructions that, when executed by the processor (110), cause the processor (110) to: compares a detection confidence level associated with a feature in an area surrounding the vehicle (100) to a predetermined threshold; and in response to the detection confidence level being below the predetermined threshold and by using location information associated with the feature, controls operation of the camera (126, 148) to capture at least one image of the feature during movement of the vehicle (100) record. camera control system claim 1 wherein the camera control module (141) includes computer-readable instructions that, when executed by the processor (110), cause the processor (110), in response to the detection confidence level being at or above the predetermined threshold, to cease operation of the camera (126 , 148) to stop attempting to capture images of the feature. camera control system claim 1 or 2 , wherein the camera control module (141) includes computer-readable instructions that, when executed by the processor (110), cause the processor (110) to: determine whether the feature is currently in a camera lockout location after the camera (126, 148) has at least one took a picture of the feature; and in response to the feature being at a camera lockout location, controlling operation of the camera (126, 148) to discontinue attempting to capture images of the feature. camera control system claim 3 wherein the camera control module (141) includes computer-readable instructions that, when executed by the processor (110), cause the processor (110), in response to the feature being at a camera lockout location, after the camera (126, 148) at least captured an image of the feature, controls operation of the vehicle (100) to transmit feature information regarding the feature to an entity external to the vehicle for human review. camera control system claim 4 , wherein the feature information includes point cloud data representing the feature and one or more images of the feature captured by the camera (126, 148) and temporally correlated with the point cloud data. Camera control system according to one of Claims 1 until 5 wherein the vehicle (100) includes a lidar sensor (125) operable to attempt to capture the feature in the vehicle environment, and the camera control module (141) includes computer-readable instructions that, when executed by the processor (110 ) cause the processor (110) to control operation of the camera (126, 148) to capture one or more images of the feature concurrently with operation of the lidar sensor (125). camera control system claim 6 wherein the camera control module (141) includes computer-readable instructions that, when executed by the processor (110), cause the processor (110) to temporally correlate lidar data with at least one simultaneous image of the feature captured by the camera (126, 148). . Camera control system according to one of Claims 1 until 7 wherein the vehicle (100) includes a radar sensor (109) operable to attempt to detect the feature in the vehicle environment, and the camera control module (141) includes computer-readable instructions that when executed by the processor (110) cause and the processor (110) controlling operation of the camera (126, 148) to capture images of the feature concurrently with operation of the radar sensor (109). camera control system claim 8 wherein the camera control module (141) includes computer-readable instructions that, when executed by the processor (110), cause the processor (110) to temporally correlate radar data with at least one simultaneous image of the feature captured by the camera (126, 148). . A computer-implemented method of controlling the operation of a camera (126,148) mounted on a vehicle (100), the method comprising the steps of: comparing a recognition confidence level associated with a feature to a predetermined threshold; and in response to the recognition confidence level being below the predetermined threshold and by using location information associated with the feature, controlling operation of the camera (126, 148) to at least one capture an image of the feature during movement of the vehicle (100). procedure after claim 10 , further comprising the step of, in response to the detection confidence level being at or above the predetermined threshold, controlling operation of the camera (126, 148) to discontinue attempting to capture images of the feature. procedure after claim 10 or 11 , further comprising the steps of: after the camera (126, 148) captures at least one image of the feature, determining whether the feature is currently at a camera lockout location; and in response to the feature being at a camera lockout location, controlling operation of the camera (126, 148) to discontinue attempting to capture images of the feature. procedure after claim 12 , further comprising the step of, in response to the feature being at a camera lockout location after the camera (126, 148) captures at least one image of the feature, controlling operation of the vehicle (100) to display feature information regarding the feature transmit an entity outside the vehicle for human verification. procedure after Claim 13 wherein the feature information includes point cloud data representing the feature and one or more images of the feature captured by the camera (126, 148) and correlated in time with the point cloud data. Procedure according to one of Claims 10 until 14 wherein the vehicle (100) includes a lidar sensor (125) operable to attempt to detect the feature in the vehicle environment, and the method further includes the step of controlling operation of the camera (126, 148), to capture one or more images of the feature concurrently with operation of the lidar sensor (125). procedure after claim 15 , further comprising time correlating lidar data with at least one simultaneous image of the feature captured by the camera (126, 148). Procedure according to one of Claims 10 until 16 wherein the vehicle (100) includes a radar sensor (109) operable to attempt to detect the feature in the vehicle environment, and the method further comprises the step of capturing images of the feature concurrently with operation of the radar sensor ( 109) contains. procedure after Claim 17 , further comprising the step of time-correlating radar data with at least one simultaneous image of the feature captured by the camera (126, 148). A non-transitory computer-readable medium for controlling the operation of a camera (126, 148) mounted on a vehicle (100), the medium having instructions stored thereon which, when executed by a computer system, cause the computer system to perform functions , which contain: comparing a recognition confidence level associated with a feature to a predetermined threshold; and in response to the detection confidence level being below the predetermined threshold and by using location information associated with the feature, controlling operation of the camera (126, 148) to capture at least one image of the feature during movement of the vehicle (100) record. non-transitory computer-readable medium claim 19 , further comprising the steps of: after the camera (126, 148) captures at least one image of the feature, determining whether the feature is currently at a camera lockout location; and in response to the feature being at a camera lockout location, controlling operation of the camera (126, 148) to discontinue attempting to capture images of the feature.

Images