CN110073243A - Fast-scanning lidar with dynamic voxel detection - Google Patents

Abstract

The laser radar system includes: a scanner; a receiver; and one or more processor devices to perform acts comprising: scanning the continuous beam over the field of view in a first scanning channel; detecting photons of the continuous beam reflected from the one or more objects; determining a coarse range to the one or more objects based on a departure time of a photon of the continuous beam and an arrival time of the photon at the receiver; scanning the light pulses over the field of view in a second scanning channel; detecting photons from the light pulses reflected from the one or more objects; a fine range to the one or more objects is determined based on a departure time of a photon of the light pulse and an arrival time of the photon at the receiver.

Description

Fast-scanning lidar with dynamic voxel detection

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is an application for a patent for invention based on previously filed U.S. Provisional Patent Application Serial No. 62/496,888, filed October 31, 2016, claiming the benefit of the filing date of that application under 35 U.S.C. §119(e), And the entire content of this application is also incorporated herein by reference.

technical field

The present invention relates generally to LIDAR systems and methods of making and using LIDAR systems. The present invention also relates to a multi-channel LIDAR system utilizing synchronous time-selectively triggered dynamic voxel detection with multi-channel granular resolution refinement, detail image contrast enhancement, ambient light suppression, and hyperspectral color options, and Methods involved in making and using LIDAR systems.

Background technique

Range determination systems may be used to determine the range, distance, location, and/or trajectory of remote objects (eg, aircraft, missiles, drones, artillery shells, baseballs, vehicles, etc.). The system can track remote objects based on detection of photons or other signals emitted and/or reflected by the remote objects. Range determination systems can utilize electromagnetic waves or light beams emitted by the system to illuminate remote objects. The system can detect a portion of the beam that is reflected or scattered by a remote object. The system may suffer from one or more of undesired speed, undesired accuracy, or undesired susceptibility to noise.

Description of drawings

Figure 1 illustrates an embodiment of an exemplary environment in which various embodiments of the invention may be implemented;

Figure 2 shows an embodiment of an exemplary mobile computer that may be included in a system such as that shown in Figure 1;

Figure 3 shows an embodiment of an exemplary network computer that may be included in a system such as that shown in Figure 1;

Figure 4 illustrates an embodiment of a two-dimensional view of an exemplary lidar system;

5 illustrates an embodiment of a logic flow diagram for an exemplary method of range or distance determination using a multi-scan process;

Figure 6A shows an embodiment of a two-dimensional view of an exemplary scan using a continuous beam for coarse range or distance determination;

Figure 6B shows an embodiment of a two-dimensional view of an exemplary scan using a pulsed beam for fine range or distance determination;

7 illustrates an embodiment of a logic flow diagram for an exemplary method of range or distance determination using a multi-scan process with color or color contrast determination;

Figure 8 shows an embodiment of a two-dimensional view of an exemplary receiver configuration with rows of pixels for color or color contrast determination;

Figure 9 shows an embodiment of a three-dimensional perspective view of an exemplary scanner configuration with a fast scanner and a slow scanner;

Figure 10A shows another embodiment of a two-dimensional view of an exemplary receiver configuration with spaced apart rows of pixels;

Figure 10B shows another embodiment of a two-dimensional view of an exemplary receiver configuration with oblique, spaced apart rows of pixels;

Figure 11 shows an embodiment of a two-dimensional view of a graph illustrating a two-dimensional focus (foveation) scan pattern;

Figure 12 shows an embodiment of a two-dimensional view of an exemplary scanner with optics for widening the field of view;

Figure 13 shows an embodiment of a two-dimensional view of an exemplary receiver with optics for widening the received light to provide more pixels to the receiver;

Figure 14 shows another embodiment of a two-dimensional view of an exemplary receiver configuration in which rows of pixels have different pixel densities;

Figure 15 illustrates an embodiment of a two-dimensional view of an exemplary scanner operating over a limited field of view;

Figure 16A shows an embodiment of a two-dimensional view of a portion of an exemplary lidar system, and where the effect of fog or drizzle on the light and receiver is shown; and

Figure 16B shows another embodiment of a two-dimensional view of a portion of an exemplary lidar system, and where the effect of fog or drizzle on the light and receiver is shown.

Detailed ways

Various embodiments will now be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show by way of illustration specific embodiments in which the invention may be practiced. Embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the embodiments to those skilled in the art. range. Among them, various embodiments may be methods, systems, media or devices. Accordingly, various embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Therefore, the following detailed description should not be viewed in a limiting sense.

Throughout the specification and claims, unless the context clearly dictates otherwise, the following terms take on the meanings explicitly associated herein. The phrase "in one embodiment" as used herein does not necessarily refer to the same embodiment, although it may. Furthermore, the phrase "in another embodiment" as used herein does not necessarily refer to a different embodiment, although it may. Accordingly, various embodiments of the present invention may be readily combined as described below without departing from the scope or spirit of the present invention.

Additionally, the term "or" as used herein is an inclusive "or" operator and is equivalent to the term "and/or" unless the context clearly dictates otherwise. Unless the context clearly dictates otherwise, the term "based on" is not exclusive and allows for other factors not described. In addition, throughout the specification, the meanings of "a", "an" and "the" include plural references. The meaning of "in" includes "in" and "on".

As used herein, the terms "photon beam", "beam", "electromagnetic beam", "image beam" or "beam" refer to a beam of photons that is localized (in time and space) or within the electromagnetic (EM) spectrum EM waves with various frequencies or wavelengths. The exit beam is the beam emitted by various of the various embodiments disclosed herein. The incident light beam is the light beam detected by various ones of the various embodiments disclosed herein.

As used herein, the term "light source", "photon source" or "source" refers to a variety of equipment. A light source or photon source may emit one or more outgoing beams. The photon source may be a laser, light emitting diode (LED), organic light emitting diode (OLED), light bulb, or the like. Photon sources can generate photons by stimulated emission of atoms or molecules, incandescent processes, or various other mechanisms that generate EM waves or one or more photons. The photon source may provide a continuous or pulsed outgoing beam of predetermined frequency or frequency range. The outgoing beam may be a coherent beam. Photons radiated by a light source can be of various wavelengths or frequencies.

As used herein, the terms "receiver", "photon receiver", "photon detector", "photodetector", "detector", "photon sensor", "photosensor" or "sensor" refer to are various devices that are sensitive to the presence of one or more photons at one or more wavelengths or frequencies of the EM spectrum. The photon detector may comprise a photon detector array, such as an arrangement of multiple photon detection or sensing pixels. One or more pixels may be photosensors sensitive to the absorption of one or more photons. A photon detector can generate a signal in response to the absorption of one or more photons. The photon detector may include a one-dimensional (1D) array of pixels. However, in other embodiments, the photon detector may include at least a two-dimensional (2D) array of pixels. Pixels can include various photon-sensitive technologies such as active pixel sensors (APS), charge-coupled devices (CCD), single-photon avalanche detectors (SPAD) (operating in avalanche or Geiger mode), complementary metal-oxide-semiconductor One or more of (CMOS) devices, silicon photomultipliers (SiPMs), photovoltaic cells, phototransistors, twitchy pixels, and the like. A photon detector can detect one or more incident light beams.

As used herein, the term "target" is one or more various 2D or 3D volumes that reflect or scatter at least a portion of incident light, EM waves or photons. Targets can also be referred to as "objects". For example, targets or objects may scatter or reflect outgoing beams emitted by various embodiments disclosed herein. In various embodiments described herein, one or more light sources may move relative to one or more receivers and/or one or more targets or objects. Similarly, one or more receivers may move relative to one or more light sources and/or one or more targets or objects. One or more targets or objects may move relative to one or more light sources and/or one or more receivers.

Embodiments of the invention are briefly described below in order to provide a basic understanding of some aspects of the invention. This brief description is not an extensive overview. It is not intended to identify key or critical elements, or to delineate or narrow the scope. Its purpose is merely to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

Briefly, various embodiments relate to measuring distance or range to a target or other object using light radiated from a light source and a receiver that receives the reflection. The system can scan the field of view with a fast scanner to perform a first scan across the field of view with a continuous beam from the light source for a rough range, then use a second slower scanner to sweep across the field of view using short pulses from the light source Do a second scan to refine the range. Additional scans can be performed to further refine the range or determine the color of a target or other object. A second, slower scanner can be added to rotate about a different axis than the first scanner to scan a two dimensional area.

Illustrative Operating Environment

Figure 1 illustrates exemplary components of one embodiment of an exemplary environment in which various exemplary embodiments of the present invention may be practiced. Not all components are required to practice the invention, and the arrangement and type of components may vary without departing from the spirit or scope of the invention. As shown, system 100 of FIG. 1 includes network 102 , light source 104 , scanner 105 , receiver 106 , one or more objects or targets 108 , and system computer equipment 110 . In some embodiments, system 100 may include one or more other computers, such as, but not limited to, laptop computer 112 and/or a mobile computer, such as but not limited to, smartphone or tablet 114 . In some embodiments, light source 104 and/or receiver 106 may include one or more components included in a computer, such as, but not limited to, various ones of computers 110 , 112 , or 114 . Light source 104 , scanner 105 and receiver 106 may be coupled directly to computer 110 , 112 or 114 by any wireless or wired technology, or may be coupled to computer 110 , 112 or 104 through network 102 .

System 100, as well as other systems discussed herein, may be ordered pixel photon projection systems. In one or more embodiments, system 100 is an ordered pixel laser projection system that includes a source of visible and/or invisible photons. Described in detail at least in U.S. Patent No. 8,282,222, U.S. Patent No. 8,430,512, U.S. Patent No. 8,696,141, U.S. Patent No. 8.711,370, U.S. Patent Publication No. 2013/0300,637, and U.S. Patent Publication No. 2016/0041266 Various embodiments of such a system are described. It is noted that each of the US patents and US patent publications listed above are hereby incorporated by reference in their entirety.

Object 108 may be a three-dimensional object. Object 108 is not an idealized black body, ie it reflects or scatters at least some of the incident photons. Light source 104 may include one or more light sources for emitting light or photon beams. Examples of suitable light sources include lasers, laser diodes, light emitting diodes, organic light emitting diodes, and the like. For example, light source 104 may include one or more visible and/or invisible laser sources. In at least some embodiments, light source 104 includes one or more of red (R), green (G), or blue (B) laser sources. In at least some embodiments, the light source includes one or more non-visible laser sources, such as near infrared (NIR) or infrared (IR) lasers. The light source may provide a continuous or pulsed beam of light having a predetermined frequency or frequency range. The provided light beam may be a coherent light beam. Light source 104 may include various features, components or functionality of a computer device, including but not limited to mobile computer 200 of FIG. 2 and/or network computer 300 of FIG. 3 .

The light source 104 may also include an optical system including optical components for directing or focusing the transmitted or exiting light beams. Optical systems can target and shape the spatial and temporal beam profile of the outgoing beam. The optical system can collimate, fan-out, or otherwise manipulate the exiting beam. At least a portion of the outgoing beam is aimed at a scanner 105 which aims the beam at an object 108 .

Scanner 105 receives light from a light source and then rotates or otherwise moves to scan the light across a field of view. Scanner 105 may be any suitable scanning device, including but not limited to MEMS scanning mirrors, acousto-optic, electro-optic scanners, or fast phased arrays, such as one-dimensional strip MEMS arrays or optical phased arrays (OPA). Scanner 105 may also include an optical system including optical components for directing or focusing incoming or outgoing light beams. Optical systems can target and shape the spatial and temporal beam profile of an incoming or outgoing beam. Optical systems can collimate, fan out, or otherwise manipulate incoming or outgoing beams. Scanner 105 may include various features, components or functionality of a computer device, including but not limited to mobile computer 200 of FIG. 2 and/or network computer 300 of FIG. 3 .

Receiver 106 will be described in more detail below. In brief, however, receiver 106 may include one or more arrays of photon-sensitive or photon-detecting sensor pixels. The sensor pixel array detects the continuous or pulsed beam of light reflected from the target 108 . The pixel array can be a one-dimensional array or a two-dimensional array. Pixels may include SPAD pixels or other photosensitive elements that avalanche when illuminated with one or a few incident photons. Pixels can have ultra-fast response times when detecting single or few photons on the order of nanoseconds. A pixel may be sensitive to frequencies radiated or emitted by light source 104 and relatively insensitive to other frequencies. Receiver 106 also includes an optical system that includes optical components for directing and focusing the received beam of light over the pixel array. Receiver 106 may include various features, components or functionality of a computer device, including but not limited to mobile computer 200 of FIG. 2 and/or network computer 300 of FIG. 3 .

Various embodiments of computer device 110 are described in more detail below in conjunction with FIGS. 2-3 (eg, computer device 110 may be an embodiment of mobile computer 200 of FIG. 2 and/or network computer 300 of FIG. 3 ). However, in brief, computer device 110 actually includes various computer devices capable of performing the various range or distance determination processes and/or methods discussed herein based on the analysis of data from one or Detection of photons reflected from a plurality of surfaces, including but not limited to surfaces of an object or target 108 . Based on the detected photons or beams, computing device 110 may change or otherwise modify one or more configurations of light source 104 and receiver 106 . It should be understood that the functions of the computer device 110 may be performed by the light source 104, the scanner 105, the receiver 106, or a combination thereof without communicating with a separate device.

In some embodiments, at least some range or distance determining functions may be performed by other computers, including but not limited to laptop computer 112 and/or mobile computers such as but not limited to smartphone or tablet 114 . Various embodiments of such computers are described in more detail below in conjunction with mobile computer 200 of FIG. 2 and/or network computer 300 of FIG. 3 .

Network 102 may be configured to couple network computers with other computing devices, including light source 104 , photon receiver 106 , tracking computer device 110 , laptop 112 , or smartphone/tablet 114 . Network 102 may include various wired and/or wireless technologies for communicating with remote devices, such as, but not limited to, USB cables, Bluetooth, Wi-Fi, and the like. In some embodiments, network 102 may be a network configured to couple network computers with other computing devices. In various embodiments, information passed between devices may include a variety of information including, but not limited to, processor readable instructions, remote requests, server responses, program modules, applications, raw data, control data, system information (e.g. log files), video data, voice data, image data, text data, structured/unstructured data, etc. In some embodiments, this information may be communicated between devices using one or more technologies and/or network protocols.

In some embodiments, such networks may include various wired networks, wireless networks, or various combinations thereof. In various embodiments, network 102 may be capable of using various forms of communication technologies, topologies, computer-readable media, etc., for communicating information from one electronic device to another. For example, in addition to the Internet, network 102 may include LANs, WANs, personal area networks (PANs), campus area networks, metropolitan area networks (MANs), direct communication connections (e.g., through Universal Serial Bus (USB) ports), etc. or various combinations thereof.

In various embodiments, communication links within and/or between networks may include, but are not limited to, twisted pair, fiber optics, open-air lasers, coaxial cable, plain old telephone service (POTS), waveguide, acoustic, full or Parts of dedicated digital lines (such as T1, T2, T3 or T4), electronic carriers, integrated services digital network (ISDN), digital subscriber lines (DSL), radio links (including satellite links) or other links and/or this Vector mechanisms known to those skilled in the art. Additionally, the communication link may employ any of a variety of digital signaling techniques, including but not limited to, for example, DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC -12, OC-48, etc. In some embodiments, a router (or other intermediate network device) may serve as a link between various networks (including those based on different architectures and/or protocols) to enable the transfer of information from one network to another. In other embodiments, a remote computer and/or other related electronic device may be connected to the network via a modem and a temporary telephone link. In essence, network 102 may include various communication technologies by which information may travel between computing devices.

In some embodiments, network 102 may include various wireless networks that may be configured to couple various portable network devices, remote computers, wired networks, other wireless networks, and the like. The wireless network may include various sub-networks that may further overlay separate ad-hoc networks, etc., for at least client computers (e.g., laptop 112 or smartphone or tablet 114) (or other mobile devices) to provide infrastructure-oriented connectivity. Such sub-networks may include mesh networks, wireless LAN (WLAN) networks, cellular networks, and the like. In one or more of the various embodiments, the system may include more than one wireless network.

Network 102 may employ a variety of wired and/or wireless communication protocols and/or techniques. Examples of communication protocols and/or technologies of various generations (e.g., third generation (3G), fourth generation (4G), or fifth generation (SG)) that may be used by the network may include, but are not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access 2000 (CDMA2000), High Speed Downlink Packet Access (HSDPA), Long Term Evolution (LTE), Universal Mobile Telecommunications System (UMTS), Evolution Data Optimized (Ev-DO), Worldwide Interoperability for Microwave Access (WiMax), Time Division Multiple Access (TDMA) , Orthogonal Frequency Division Multiplexing (OFDM), Ultra Wideband (UWB), Wireless Application Protocol (WAP), User Datagram Protocol (UDP), Transmission Control Protocol/Internet Protocol (TCP IP), Open Systems Interconnection (OSI) Portions of the model protocol, Session Initiated Protocol/Real-time Transport Protocol (SIP/RTP), Short Message Service (SMS), Multimedia Message Service (MMS), or various other communication protocols and/or technologies. Essentially, a network may include communication technology by which information may travel between light source 104, photon receiver 106, and tracking computing device 110, as well as other computing devices not shown.

In various embodiments, at least a portion of the network 102 may be arranged as nodes, links, paths, terminals, gateways, routers, switches, firewalls, load balancers, repeaters, repeaters, optical-to-electrical converters, etc. (they Autonomous systems that can be connected by various communication links). These autonomous systems can be configured to self-organize based on current operating conditions and/or rule-based policies, allowing the network topology of the network to be modified.

Illustrative mobile computer

FIG. 2 illustrates one embodiment of an exemplary mobile computer 200, which may include more or fewer components than those illustrated. Mobile computer 200 may represent, for example, one or more embodiments of laptop 112 , smartphone/tablet 114 , and/or computer 110 of system 100 of FIG. 1 . Thus, mobile computer 200 may include mobile devices (eg, smartphones or tablets), stationary/desktop computers, and the like.

Client computer 200 may include processor 202 in communication with memory 204 via bus 206 . Client computer 200 may also include a power supply 208, a network interface 210, a processor-readable fixed storage device 212, a processor-readable removable storage device 214, an input/output interface 216, a camera(s) 218, a video interface 220, touch interface 222, hardware security module (HSM) 224, projector 226, display 228, keypad 230, illuminator 232, audio interface 234, global positioning system (GPS) transceiver 236, open air gesture interface 238, temperature interface 240 . Haptic interface 242 and pointing device interface 244 . Client computer 200 may optionally communicate with a base station (not shown), or directly with another computer. And in one embodiment, although not shown, a gyroscope may be employed within the client computer 200 to measure and/or maintain the orientation of the client computer 200 .

Power supply 208 may provide power to client computer 200 . Rechargeable or non-rechargeable batteries can be used to provide power. Power may also be provided by an external power source, such as an AC adapter or a motorized docking stand that supplements and/or recharges the batteries.

Network interface 210 includes circuitry for coupling client computer 200 to one or more networks and is configured for use with one or more communication protocols and technologies, including but not limited to Implement OSI mobile communication (GSM) model, CDMA, time division multiple access (TDMA), UDP, TCP IP, SMS, MMS, GPRS, WAP, UWB, WiMax, SIP/RTP, GPRS, EDGE, WCDMA, LTE, UMTS, OFDM , CDMA2000, EV-DO, HSDPA, or the protocols and technologies of various parts of various other wireless communication protocols. Network interface 210 is sometimes referred to as a transceiver, transceiving device, or network interface card (NIC).

The audio interface 234 may be arranged to generate and receive audio signals, such as the sound of a human voice. For example, audio interface 234 may be coupled to a speaker and microphone (not shown) to enable communication with other persons, or to generate audio confirmation for certain actions. The microphone in audio interface 234 may also be used for input to or control of client computer 200, eg, using speech recognition, detecting touch based on sound, and the like.

Display 228 may be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LED), organic LED (OLED), or various other types of light reflective or light transmissive displays that may be used with a computer. The display 228 may also include a touch interface 222 arranged to receive input from an object such as a finger of a human hand or a stylus and may use resistive, capacitive, surface acoustic wave (SAW), infrared, radar, or other techniques to sense touch and/or gestures.

Projector 226 may be a remote handheld projector or an integrated projector capable of projecting images on a remote wall or various other reflective objects such as a remote screen.

Video interface 220 may be arranged to capture video images, such as still photographs, video clips, infrared video, and the like. For example, video interface 220 may be coupled to a digital video camera, web camera, or the like. Video interface 220 may include a lens, an image sensor, and other electronic devices. Image sensors may include complementary metal oxide semiconductor (CMOS) integrated circuits, charge coupled devices (CCDs), or various other integrated circuits for sensing light.

Keypad 230 may include various input devices arranged to receive input from a user. For example, keypad 230 may include a button number dial or keypad. Keypad 230 may also include command buttons associated with selecting and sending images.

Illuminators 232 may provide status indications and/or provide light. Illuminators 232 may remain active for a certain period of time or in response to event messages. For example, if illuminator 232 is active, it can backlight the buttons on keypad 230 and remain on while the client computer is powered on. Also, the illuminator 232 may backlight these buttons in various patterns if certain actions are performed, such as dialing another client computer. The illuminator 232 may also cause a light source located within a transparent or translucent housing of the client computer to illuminate in response to an action.

Additionally, client computer 200 may also include HSM 224 for providing additional protection against generating, storing, and/or using secure/encrypted information (eg, keys, digital certificates, passwords, passphrases, two-factor authentication information, etc.). Tamper protection measures. In some embodiments, hardware security modules may be employed to support one or more standard public key infrastructures (PKIs), and may be employed to generate, manage, and/or store key pairs, and the like. In some embodiments, HSM 224 may be a stand-alone computer, in other cases, HSM 224 may be arranged as a hardware card that may be added to a client computer.

The client computer 200 may also include an input/output interface 216 for communicating with external peripherals or other computers such as other client computers and network computers. Peripherals may include audio headsets, virtual reality headsets, display glasses, remote speaker systems, remote speaker and microphone systems, and more. Input/output interface 216 may use one or more technologies such as Universal Serial Bus (USB), infrared, Wi-Fi ^™ , WiMax, Bluetooth ^™ , and the like.

The input/output interface 216 may also include one or more sensors for determining geolocation information (e.g., GPS), monitoring power conditions (e.g., voltage sensors, current sensors, frequency sensors, etc.), monitoring weather (e.g., thermostat , barometer, anemometer, humidity detector, precipitation gauge, etc.). A sensor may be one or more hardware sensors that collect and/or measure data external to client computer 200 .

Haptic interface 242 may be arranged to provide tactile feedback to a user of the client computer. For example, haptic interface 242 may be employed to vibrate client computer 200 in a particular manner if another user of the computer is calling. Temperature interface 240 may be used to provide temperature measurement input and/or temperature change output to a user of client computer 200 . The open-air gesture interface 238 may sense the body gestures of the user of the client computer 200, for example, through the use of single or stereo video cameras, radar, gyroscopic sensors within the computer held or worn by the user, or the like. Camera 218 may be used to track the physical eye movements of a user of client computer 200 .

GPS transceiver 236 can determine the physical coordinates of client computer 200 on the surface of the earth, which typically outputs the location as latitude and longitude values. GPS transceiver 236 may also employ other geolocation mechanisms including, but not limited to, triangulation, Assisted GPS (AGPS), Enhanced Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Augmented GPS Timing Advance (ETA), Base Station Subsystem (BSS), etc. to further determine the physical location of the client computer 200 on the surface of the earth. It should be understood that GPS transceiver 236 may determine the physical location of client computer 200 under various conditions. However, in one or more embodiments, client computer 200 may, through other components, provide other information that may be used to determine the client computer's physical location, including, for example, a Media Access Control (MAC) address, IP address, and the like.

A human interface component may be a peripheral device physically separate from client computer 200 that allows remote input and/or output to client computer 200 . For example, information routed through a human interface component such as display 228 or keypad 230 as described herein may instead be routed through network interface 210 to an appropriate human interface component located remotely. Examples of human interface peripheral components that may be remote include, but are not limited to, audio devices, pointing devices, keyboards, displays, cameras, projectors, and the like. These peripheral components may communicate over a piconet (such as Bluetooth ^™ , Zigbee ^™ , etc.). A non-limiting example of a client computer with such peripheral human interface components is a wearable computer, which may include a remote pico-projector and one or more cameras in remote communication with a separately located client computer to sense the user's A gesture of an image portion projected by a pico projector onto a reflective surface such as a wall or the user's hand.

Memory 204 may include RAM, ROM, and/or other types of memory. Memory 204 illustrates an example of a computer-readable storage medium (device) for storing information such as computer-readable instructions, data structures, program modules, or other data. Memory 204 may store BIOS 246 for controlling low-level operations of client computer 200 . The memory may also store an operating system 248 for controlling the operation of the client computer 200 . It should be understood that this component may include a general-purpose operating system (such as UNIX or LINUX ^™ versions), or a dedicated client computer communication operating system (such as Windows Phone ^™ , or Symbian operating system). The operating system may include or interface with a Java virtual machine module capable of controlling hardware components and/or operating system operations through Java applications.

Memory 204 may also include one or more data storage devices 250 that client computer 200 may utilize to store applications 252 and/or other data, among other things. For example, data storage 250 may also be used to store information describing various capabilities of client computer 200 . In one or more of the various embodiments, data storage 250 may store range or distance information 251 . Information 251 may then be provided to another device or computer based on various methods, including sent as part of a header during communication, sent on request, and the like. Data storage 250 may also be used to store social networking information, including address books, buddy lists, aliases, user profile information, and the like. The data storage device 250 may also include program codes, data, algorithms, etc. for the processor (eg, the processor 202) to implement and perform actions. In one embodiment, at least a portion of data storage 250 may also be stored on another component of client computer 200, including but not limited to non-transitory processor readable fixed storage 212, processor readable removable storage device 214, or even external to the client computer.

Applications 252 may include computer-executable instructions that, if executed by client computer 200 , send, receive and/or otherwise process instructions and data. Applications 252 may include, for example, a range/distance determination client engine 254, other client engines 256, a web browser 258, and the like. The client computer may be arranged to exchange communications, such as queries, searches, messages, notification messages, event messages, alerts, performance metrics, log data, API calls, etc., with the application server, network file system application, and/or storage management application, and combinations thereof.

Web browser engine 226 may be configured to receive and send web pages, web-based messages, graphics, text, multimedia, and the like. The browser engine 226 of the client computer can use virtual various programming languages, including wireless application protocol message (WAP) and so on. In one or more embodiments, browser engine 258 is capable of using Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SGML), Hypertext Markup Language (HTML), Extensible Markup Language (XML), HTML5, etc.

Other examples of applications include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs , security applications, spreadsheet programs, games, search programs, etc.

In addition, in one or more embodiments (not shown), the client computer 200 may include embedded logic hardware devices instead of a CPU, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), Programmable Array Logic (PAL), etc., or a combination thereof. Embedded Logic Hardware devices can directly execute their embedded logic to perform actions. Also, in one or more embodiments (not shown), client computer 200 may include a hardware microcontroller instead of a CPU. In one or more embodiments, the microcontroller can directly execute its own embedded logic to perform actions and access its own internal memory and its own external input and output interfaces (e.g., hardware pins and/or wireless Transceiver) to perform actions, such as system-on-chip (SOC), etc.

illustrative network computer

FIG. 3 illustrates one embodiment of an example network computer 300 that may be included in an example system implementing one or more of the various embodiments. Network computer 300 may include more or fewer components than those shown in FIG. 3 . However, the components shown are sufficient to disclose an illustrative embodiment for practicing the innovations. Network computers 300 may include desktop computers, laptop computers, server computers, client computers, and the like. Network computer 300 may represent, for example, one embodiment of one or more of laptop 112 , smartphone/tablet 114 , and/or computer 110 of system 100 of FIG. 1 .

As shown in FIG. 3 , network computer 300 includes processor 302 that can communicate with memory 304 via bus 306 . In some embodiments, processor 302 may include one or more hardware processors or one or more processor cores. In some cases, one or more of the one or more processors may be specialized processors designed to perform one or more specialized actions, such as those described herein. Network computer 300 also includes power supply 308, network interface 310, processor readable fixed storage device 312, processor readable removable storage device 314, input/output interface 316, GPS transceiver 318, display 320, keyboard 322, audio interface 324 , directional device interface 326 and HSM 328 . Power supply 308 provides power to network computer 300 .

Network interface 310 includes circuitry for coupling network computer 300 to one or more networks and is configured for use with one or more communication protocols and technologies including, but not Limited to the implementation of the Open Systems Interconnection Model (OSI model), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), User Datagram Protocol (UDP), Transmission Control Protocol/Internet Protocol ( TCP IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), General Packet Radio Service (GPRS), WAP, Ultra Wideband (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMax), Session Initiation Protocol /Real-time Transport Protocol (SIP/RTP), or various other protocols and technologies for various parts of wired and wireless communication protocols. Network interface 310 is sometimes referred to as a transceiver, transceiving device, or network interface card (NIC). Network computer 300 may optionally communicate with a base station (not shown), or directly with another computer.

The audio interface 324 is arranged to generate and receive audio signals, eg the sound of a human voice. For example, audio interface 324 may be coupled to a speaker and microphone (not shown) to enable communication with others and/or to generate audio confirmation of an action. The microphone in audio interface 324 may also be used for input to or control of network computer 300, for example, using speech recognition.

Display 320 may be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LED), organic LED (OLED), or various other types of light reflective or light transmissive displays that may be used with a computer. Display 320 may be a handheld or pico projector capable of projecting images on a wall or other object.

The network computer 300 may also include an input/output interface 316 for communicating with external devices or computers not shown in FIG. 3 . The input/output interface 316 may utilize one or more wired or wireless communication technologies such as USB ^™ , Firewire ^™ , Wi-Fi ^™ , WiMax, Thunderbolt ^™ , Infrared, Bluetooth ^™ , Zigbee ^™ , serial port, parallel port, etc. .

Additionally, input/output interface 316 may include one or more sensors for determining geolocation information (e.g., GPS), monitoring power conditions (e.g., voltage sensors, current sensors, frequency sensors, etc.), monitoring weather (e.g., thermostats, barometers, anemometers, humidity detectors, precipitation gauges, etc.) etc. A sensor may be one or more hardware sensors that collect and/or measure data external to network computer 300 . Human interface components may be physically separate from network computer 300 , which allows remote input and/or output to network computer 300 . For example, information routed through a human interface component such as display 320 or keyboard 322 as described herein may instead be routed through network interface 310 to an appropriate human interface component located elsewhere on the network. Human interface components include various components that allow a computer to obtain input from or send output to a human user of the computer. Accordingly, a pointing device such as a mouse, stylus, trackball, etc. may communicate through pointing device interface 326 to receive user input.

GPS transceiver 318 can determine the physical coordinates of network computer 300 on the surface of the earth, which typically output the location as latitude and longitude values. GPS transceiver 318 may also employ other geolocation mechanisms including, but not limited to, triangulation, Assisted GPS (AGPS), Enhanced Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Augmented GPS Timing Advance (ETA), Base Station Subsystem (BSS), etc. to further determine the physical location of the network computer 300 on the surface of the earth. It should be understood that GPS transceiver 318 may determine the physical location of network computer 300 under various conditions. However, in one or more embodiments, network computer 300 may, through other components, provide other information that may be used to determine the physical location of the client computer, including, for example, Media Access Control (MAC) addresses, IP addresses, and the like.

Memory 304 may include random access memory (RAM), read only memory (ROM), and/or other types of memory. Memory 304 illustrates an example of a computer-readable storage medium (device) for storing information such as computer-readable instructions, data structures, program modules, or other data. The memory 304 stores a basic input/output system (BIOS) 330 for controlling low-level operations of the network computer 300 . The memory also stores an operating system 332 for controlling the operation of the network computer 300 . It should be understood that this component may include a general-purpose operating system (such as UNIX or LINUX ^™ versions), or a dedicated operating system (such as Microsoft Corporation's operating system, or Apple's operating system). The operating system may include a Java virtual machine module, or be interfaced with the Java virtual machine module, and the Java virtual machine module may control hardware components and/or operating system operations through Java application programs. Likewise, other runtime environments can be included.

Memory 304 may also include one or more data storage devices 334 that network computer 300 may utilize to store applications 336 and/or other data, among other things. For example, data storage device 334 may also be employed to store information describing various capabilities of network computer 300 . In one or more of the various embodiments, data storage 334 may store range or distance information 335 . Range or distance information 335 may then be provided to another device or computer based on various methods, including sent as part of a header during communication, sent on request, and the like. Data storage 334 may also be used to store social networking information, including address books, buddy lists, aliases, user profile information, and the like. Data storage 334 may also include program code, data, algorithms, etc., for use by one or more processors (eg, processor 302 ) in implementing and performing actions such as those described below. In one embodiment, at least part of data storage 334 may also be stored on another component of network computer 300, including but not limited to non-transitory processor readable fixed storage 312, processor readable removable storage 314 , or non-transitory media in various other computer-readable storage devices within network computer 300 or even external to network computer 300 .

Applications 336 may include computer-executable instructions that, if executed by network computer 300, send, receive, and/or otherwise process messages (e.g., SMS, Multimedia Messaging Service (MMS), Instant Messaging (IM), email, and and/or other messages), audio, video, and enable telecommunications with another user of another mobile computer. Other examples of applications include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, etc. Applications 336 may include a range or distance determination engine 346 that performs actions described further below. In one or more of the various embodiments, one or more applications may be implemented as a module and/or component of another application. Additionally, in one or more of the various embodiments, applications may be implemented as operating system extensions, modules, plug-ins, and the like.

Additionally, in one or more of the various embodiments, range or distance determination engine 346 may operate in a cloud-based computing environment. In one or more of the various embodiments, these and other applications can execute within virtual machines and/or virtual servers that can be managed in a cloud-based computing environment. In one or more of the various embodiments, in this context, applications may flow from one physical network computer to another within a cloud-based environment, depending on the performance and Scaling considerations. Likewise, in one or more of the various embodiments, virtual machines and/or virtual servers dedicated to range or distance determination engine 346 may be automatically configured and deactivated.

Furthermore, in one or more of the various embodiments, the range or distance determination engine 346, etc. may reside in a virtual server running in a cloud-based computing environment rather than being bound to one or more specific physical networks computer.

Additionally, network computer 300 may include HSM 328 for providing additional tamper-resistant protection for generating, storing, and/or using secure/encrypted information (e.g., keys, digital certificates, passwords, passphrases, two-factor authentication information, etc.) security measures. In some embodiments, hardware security modules may be employed to support one or more standard public key infrastructures (PKIs), and may be employed to generate, manage, and/or store key pairs, and the like. In some embodiments, HSM 328 may be a stand-alone network computer, in other cases HSM 328 may be arranged as a hardware card that may be installed in a network computer.

Additionally, in one or more embodiments (not shown), the network computer may include one or more embedded logic hardware devices instead of one or more CPUs, such as application-specific integrated circuits (ASICs), field programmable Gate Array (FPGA), Programmable Array Logic (PAL), etc., or a combination thereof. Embedded logic hardware devices can directly execute embedded logic to perform actions. Also, in one or more embodiments (not shown), the network computer may include one or more hardware microcontrollers instead of a CPU. In one or more embodiments, one or more microcontrollers can directly execute their own embedded logic to perform actions and access their own internal memory and their own external input and output interfaces (e.g., hardware pins and/or wireless transceiver) to perform actions, such as a system on chip (SOC), etc.

descriptive system

One embodiment of a lidar system 400 is shown in FIG. 4 . In at least some embodiments, lidar system 400 is a fast scanning system that continuously (e.g., smoothly, rapidly) without stopping) to move the scanning beam from the light source 404 through a number of positions of the one or more objects 108 (see FIG. 1 ). Light reflected by one or more objects in field of view (FoV) 403 passes through aperture 407 and is received and detected by receiver 406 . In some embodiments, scanner 405 utilizes ultrafast resonant rotation of a MEMS scanning mirror (or other suitable scanning mirror or device), which moves rapidly over a range of angles to scan field of view 403 . As described in more detail below, other slower scanners 405 may also be used in techniques employing two or more scan channels.

When using a fast scanner 405, the direction of the beam from the scanner changes so rapidly that each fraction of the angular direction can be paired in time with an ultra-short time interval of at least nanosecond duration. This establishes angular position as a function of time (time => angle), the function can later be reversed to create an inverse 1-1 function (angle => time), e.g. in a lookup table, to generate The pixel observed reflections for each incident direction are precisely bounded by the possible departure time ranges of reflected photons. In at least some embodiments, the coarse departure time can be derived from the angular direction at which the reflected light is observed, which can be determined by the position of the pixel of the receiver 406 that detected the light.

In at least some embodiments of the lidar system 400, the receiver 406 is co-located with or near the scanner 405 and detects reflections from one or more objects in the field of view when photons return to the receiver 406. of photons. These photons return at the same angle - but now travel in the opposite "return to sender" direction. In at least some embodiments, receiver 406 is a one-dimensional or two-dimensional receiver.

Any suitable photon receptor 406 may be used, including any suitable pixelated photon receptor 406 . Examples of pixelated photon receptors include, but are not limited to, pixels arranged as a Spatio-temporal Sorting Array (SSA), eg, an array of fast asynchronous SPAD (Single Photon Avalanche Diode) pixels that record direction and time of arrival. Examples of SSA arrays can be found in US Patent Nos. 8,282,222, 8,430,512 and 8,696,141. All of these US patents are hereby incorporated by reference in their entirety. A spatiotemporal sorting array can be compared to a camera with a detector array located in the focal plane of the imaging system that spatially quantizes the incident ray direction to match the incident direction of the beamlets to individual pixels. In fact, the SSA can be a camera with a 2D pixel array, or alternatively, a camera as in U.S. Patent Nos. 8,282,222, Any of the asynchronous sensing arrays described in No. 2013/0300637 and No. 2016/0041266, all of these US patents and patent application publications are hereby incorporated by reference in their entirety. Other suitable arrays for receiver 406 include, but are not limited to, the use of CMOS (Complementary Metal Oxide Semiconductor), CCD (Charge Coupled Device), APD (Avalanche Photodiode), SPADS, SiPM (Silicon Photomultiplier), etc. Any combination as 1D and 2D imaging arrays of pixels.

In at least some embodiments, the scanning speed of the scanner 405 of the lidar system 400 and the spatial resolution of the array of receivers 406 are preferably relatively high for single pass scanning techniques. For example, in a fast scanning system, a full scan across the FoV (field of view) may take only 1 microsecond or less. As reflections of the scanning beam return into the bore, the incident directions are sorted into eg 100, 500, 1000, 2000, 5000 or 10,000 or more bins. For example, using an array with 1000 SPAD pixels in a row aligned with the scan direction, the departure time ( _Td ) of each reflection can be resolved to 1 by the scanner position (beam direction) recorded in a 1 microsecond scan Nanoseconds (1 microsecond/1000 bins). The time of arrival (T _a ) is also resolved in time to nanosecond instants (or less for SPAD arrays). Using the time-of-departure and time-of-arrival, the round-trip time-of-flight (ToF) of the arriving photon can be determined. The distance to the object reflecting the detected photon is 2 times the ToF divided by the speed of the photon (i.e. the speed of light c). Examples of this system may achieve a ranging resolution of 1/2 foot (about 0.35 meters) or less.

The resolution of the lidar system 400 may depend on having enough pixels, since the more spatially temporally ordered bins (ie, pixels) in the array, the better. For example, 10,000 tiny 1-micron CMOS "twitch pixels" could provide high resolution, provided the instantaneously reflected photon intensity is high enough to trigger tiny pixels within nanoseconds. US Patent No. 9,753,125, the entire contents of which are incorporated herein by reference, describes "twitching pixels" as sensor array pixels that provide a nearly instantaneous signal output once the photocurrent exceeds a minimum level. For example, in at least some embodiments, a "twitching pixel" may be a photodiode connected to a source follower or other circuit that immediately amplifies the photodiode's current. The amplified signal is in turn connected to a sense line. The sensing line can be an entire column or an entire row such a shared function between "twitching pixels". The basic "twitch" function of a pixel is binary; its primary function is to report when and/or where a signal photon has reached the receiver. In lidar system 400 , “twitching pixel” and SPAD may be used interchangeably in receiver 406 .

In at least some embodiments of single-pass technology, the lidar system 400 uses a very fast scanner that can scan the entire width of the FoV in a few microseconds (e.g., 5, 3, 2, or 1 microsecond or less) (or height). Very fast scanners 405 may include, but are not limited to, acousto-optic, electro-optic scanners, or fast phased arrays, such as ID ribbon MEMS arrays or optical phased arrays (OPA). Such scanners may have a limited deflection angle, and an additional optical stage may be used to amplify the scan angle to overcome the limited deflection angle. Furthermore, in at least some embodiments, these scanners can only operate with monochromatic beams in a very limited portion of the spectrum. As a result, such ultrafast scanners can be expensive, fragile or bulky, and can be difficult to use, especially for small mobile applications.

In some embodiments of the lidar system 400, a slower scanner 405, such as a resonant MEMS scanning mirror, may be used. In some embodiments, the scanner may scan no faster than 100, 75, 60 or 50 kHz or less. Scanning techniques utilizing two or more scanning channels can be used to produce robust and accurate lidar systems.

Figure 5 shows the steps in the dual scan technique. In step 502 , a continuous light beam from light source 404 is scanned over field of view (FoV) 403 using scanner 405 . For example, a continuous beam may scan the entire FoV over eg 5, 10 or 20 microseconds or more, but slower or faster scan times may be used.

In step 504, photons reflected from one or more objects in the FoV are detected by receiver 406 and, as described above, the detected photons may be used to provide an initial coarse range to the one or more objects. FIG. 6A shows one embodiment of this first scan, where scanner 405 ( FIG. 4 ) is scanning in direction 609 . Light 611 is reflected from object 608 and is then received at _i -th pixel pi 606i comprising n pixels of receiver 406 (FIG. 4). The departure time (T _d ) of a photon detected by pixel p _i can be roughly resolved with resolution 617 as a function of ΔT _d , which is the maximum departure time of a photon detected by pixel p _{i (} T _dmax ) and the minimum departure time (Τ _dmin ). As an example, a 1000 pixel 1D receiver can be used to detect photons from a 10 microsecond scan (which gives a ΔTd of _10ns per pixel) (eg using a 50kHz bi-directional 1D resonant MEMS scanning mirror as a scanner). Using simple ToF ranging calculations, the time-of-arrival (T _a ) has a temporal resolution of 1 ns, e.g., for each reflection observed by the receiver, the initial coarse range resolution 617 can be resolved to e.g. 5 feet (about 1.5 meters ). Thus, an estimated range to an object can be roughly resolved, and in some embodiments, the system can record those pixels that detect photons and those that do not.

In step 506, the same FoV is scanned, but instead of using a continuous beam, short pulses 611' (eg, sharp "needling" pulses) emitted by light source 404 are used, as shown in Figure 6B. In some embodiments, this second scan pass (or "refinement" scan) may be performed by retracing the same scan in the opposite direction on the return stroke of the scanner. In other embodiments, the scanner returns to its initial position and then scans in the same direction. The short pulses have a pulse width ΔT _dpi shorter than AT _d of the first scan, and each pulse is synchronized to correspond to one pixel. Preferably, the pulse width AT _dpi is not greater than 30%, 25%, 10%, 5% or less of _ATd of the first scan. In at least some embodiments, the pulse width of the light pulse is less than the scan time of the second scan channel divided by the number of pixels in a single row of the receiver. In at least some embodiments, the pulse width is no greater than 1 nanosecond or 500 or 100 picoseconds or less.

Optionally, a pulse is only fired if a reflection from the corresponding object position was observed in a previous coarse scan. Individual pixels in the array can be actively enabled. An initial continuous thick-line scan can inform the system which specific pixels to selectively activate, and when exactly to activate each pixel during a second "fine-line" scan. In cases where only a small subset of the FoV has reflective objects of interest within the lidar range, only a small fraction of pixels may be activated.

In step 508, the photon receiver 406 receives the reflected pulse and determines the arrival time T _a of the reflected pulse. For arrival times known from pixels, the distance or range to one or more objects 408 can be determined, just as with the initial coarse range resolution in step 504, but with greater accuracy. The departure time of each short pulse from the light source is also known, so reflected light pulses can be associated with discrete departure times (T _d ). These departure times are known to a high precision, eg for a 100ps pulse the departure times are known to a precision of 100ps. The reflected pulses are confined to known intervals (eg, 100 picoseconds (ps)) and uniquely matched to a single pixel in the array. Continuing with the example given above, the short pulses may be 100 ps pulses, where each pulse is synchronized in time to receive reflections through individual pixel locations of the receiver 406 (eg, the center of each of the 1000 pixel locations). When the SPAD array clocks the incoming arrival time (T _a ) and matches it to the corresponding departure time (T _d ) with a resolution of eg 100 ps, distance observation can be improved by 1/100. For example, for an initial coarse range resolution of 5 feet in the example provided above, the refined range resolution may be 0.05 feet or approximately 1.5 centimeters.

In some embodiments, an initial coarse range determined from the first scan informs the control system when to activate individual pixels so that the system can narrowly limit pixels to time them to be active for only a few nanoseconds. Thus, by using this anticipatory activation method, not only can the beam pulses be timed to orientationally match the receiver's exact pixel location, but each individual pixel can be activated only for the expected arrival time T _a , for example only for 10 nanoseconds (where 10 nanoseconds is the time uncertainty determined for the reflection in the gaze direction of this pixel during the previous coarse scan (ToF Range Uncertainty)).

In some embodiments using anticipatory activation techniques, the system can reduce interference from ambient or stray light. For example, using the expected activation technique on a second scan with a 10 ns window for each pixel, ambient light will have at most 10 ns (compared to 10 ms for a full FoV scan) to interfere with the reflected light received by the pixel. Thus, only one millionth of sunlight (at most 1/10 lux, even in strongly blinding environments (one millionth of 100K lux = full direct sunlight)) is received by the pixel .

In some embodiments, SPAD pixels can be activated in Geiger mode (characterized by a highly variable high voltage, reverse biased by the photodiode), and are therefore particularly sensitive, producing a strong, transient low Jitter pulse.

It should be noted that during the second "refined" scan, the scan pulses may be very sparse, limited to obtaining more accurate data at only a few selected detected objects (e.g., a small object in the quadcopter's planned flight path). Good fine grained appearance. With the nanosecond anticipatory activation of SPADS, ambient light can be suppressed to such an extent that each pulse requires very little energy and the total energy emitted can be kept below safe levels. In optional steps 510 and 512, the process of steps 506 and 508 is repeated one or more times, except that the short pulses in successive scans are shifted in time by small increments (e.g., fractions of nanoseconds) (ie, steps 510 and 512 may be repeated multiple times). This has the effect of accessing locations directly adjacent to the locations identified on the surface of the object in steps 506 and 508 . On successive surfaces, the reflections of these later short pulses should arrive predictably within 100 ps of reflections obtained from previously scanned short pulses. Given that picosecond-accurate surface observations can be part of voxel-wise motion datasets of objects fed to downstream vision processing systems, surface models (e.g., of cars, drones, vehicles, etc.) can help clarify images computationally .

Optionally, the system can also enable a range selection feature, for which the _coarse range determination of the first scan channel indicates that an object may be present in all within the selected range. For example, in at least some embodiments, a range selection of 50 feet reduces SPAD activation to short periods of only 100 nanoseconds, e.g., enabling short SPAD pixel turn-on times in Geiger mode, which may increase system sensitivity .

In at least some embodiments, the system may be filterless, since narrow bandpass filters may no longer be required. In at least some embodiments, multispectral illumination may be enabled during a second scan pass or during a subsequent scan pass.

Figure 7 shows the method for color lidar. Steps 702-708 in FIG. 7 are the same as steps 502-508 in FIG. 5 .

In step 710, similar to the scan channels performed in steps 508 and 708 using short pulses of light, the visible primaries (red, green, blue (or other colors) )) executes one or more scan channels. In some embodiments, a single scan pass may be performed using three (or more) different colored light beams or using a single white light source. In other embodiments, a single, differently colored beam may be used to continuously scan channels during each scan. In at least some embodiments, the one or more scan passes retrace the same or similar trajectory on the surface of the object as the second scan pass in step 708 .

In step 712, reflected photons of a particular color are detected by the color-sensitive pixels of the receiver and used to determine the color or color contrast of the surface of the object. FIG. 8 shows an embodiment of a receiver 806 that includes a row of pixels for detecting light from the first and second scan channels and a row of pixels 820r for detecting red, green, and blue light, respectively. , 820g, 820b. Color-sensitive pixels may be specifically designed to be activated by the associated color, or may contain color filters to remove light of other colors, or any other arrangement for color-sensing a pixel. Because each color pulse is deterministically matched to a specific sensor pixel, and because there are still as many as 1000 or more pixels in the array, the pulse rate and range of the system can be 1000's of conventional pulsed lidars using a single APD detector (or more) times.

The method produces a three (or more) channel system for super-resolution color lidar. 1) Using an initial coarse pass of successively on beams, it finds the reflection of the surface and establishes the approximate extent and position of each surface point (ie bold voxel). 2) A second refinement channel utilizing picosecond-accurate light pulses for centimeter-accurate range resolution. 3) A specific effective range gated final channel (or set of channels) with effectively nanosecond accurate pixels, virtually eliminating all remaining ambient light and enabling accurate color reflectance measurements using eg selected spectral primary illuminants.

It is also possible to use the fast scanner 405 and the slow scanner 922 to make a two-dimensional (2D) scanning lidar system, as shown in FIG. 9 . As an example, a MEMS scanning mirror or any other suitable fast 1D scanner can be used as scanner 405 . In at least some embodiments, scanner 405 will scan at a rate of 25 kHz or 50 kHz or higher. The slow scanner 922 provides a second scanning dimension by creating a bi-directional scanning path. For example, the hexagonal scanner 922 (or octagonal scanner or any other suitable scanner) may be slowly rotated about an axis perpendicular to the scanner surface to move along the first dimension as the fast scanner 405 repeatedly scans along the first dimension. Two-dimensional slow scan. A slow polygonal surface deflects outgoing rays or pulses and incoming reflections equally during its rotation, for example at a FoV of 90 degrees (or larger or smaller). Another example of a slow scanner 922 is a slow two-dimensional quasi-static MEMS mirror, which can operate at 1 kHz to 4 kHz.

Reflected photons may only be directed to a one-dimensional (or two-dimensional) receiver 406, just like the single scanner embodiment described above. For example, incident photons can be detected by an array of 1000 pixels with a coarse resolution of 5-10 feet (coarse time 10 to 20 nanoseconds).

The fast scan period of the fast scanner 405 is orders of magnitude shorter (no more than a few microseconds) than the slow scan period (milliseconds or longer) required by the slow scanner 922 . For example, in one embodiment, each fast scan takes no more than 10 microseconds, during which time the slow scanner 922 moves only a small distance. For example, an octagonal scanner rotating at 10 Hz (producing 80 full frame detections per second with a field of view up to 90 degrees) has a slow axis rotation speed of about 7200 degrees/second. Therefore, in 10 microseconds, the scan line shifts only 0.072 degrees.

Another example of a slow scanner 922 is a slow two-dimensional mirror, such as a two-axis MEMS mirror that can operate at 1 to 4 kHz. The relatively slow scanning speed of the slow scanner 922 can be used to generate a two-dimensional scanning pattern 1150 similar to the focusing motion of the eye, as shown in FIG. 11 . Movement along scan direction 1152 of fast scanner 405 is faster than movement along scan direction 1154 of slow scanner 922 . In at least some embodiments, after detecting and/or classifying objects, the system may use focus motion to lock onto an object of interest (eg, a child crossing a street or a nearby vehicle).

Although a one-dimensional receiver 406 can be used with a two-scanner system, in some embodiments a receiver 1006 with two or more rows of pixels 1020, 1020a (as shown in Figures 10A and 10B) can be used to consider Slow rotation of the slow scanner 922. In the embodiment shown in Figure 10A, two or more rows of pixels 1020, 1020a may be provided such that a first row 1020 detects photons reflected during a first scan and a second row 1020a detects photons reflected during a second scan. photon. The separation distance between the first row and the second row may reflect the amount of rotation of the slow scanner 922 between the first scan and the second scan. Also, in some embodiments, a first scan is performed in one direction along the first row of pixels 1020, and then a second scan is made in the opposite direction along the second row of pixels 1020a when the scanner 405 returns to its initial position. conduct. In other embodiments, a first scan is performed in one direction along the first row of pixels 1020, then the scanner returns to its initial position, and then a second scan is performed in the same direction along the second row of pixels 1020a. In the latter case, the separation between rows may be greater due to the extra time for the scanner to return to its initial position.

In the embodiment shown in FIG. 10B, two or more rows of pixels 1020, 1020a are angled (exaggerated in FIG. 10B) to account for slight rotation of the slow scanner 922 during the first or second scan, respectively. . In the embodiment shown in Figure 10B, a first scan is made in one direction along the first row of pixels 1020, the scanner is then returned to its initial position, and then a second scan is made in the same direction along the second row of pixels 1020a. conduct. Alternatively, a first scan may be performed in one direction along the first row of pixels 1020, and then a second scan may be performed in the opposite direction along the second row of pixels 1020a when the scanner 405 returns to its original position; In this case, the second row of pixels 1020a will be tilted in the opposite direction to the first row of pixels 1020 .

In some embodiments, optics may be used to enhance the system. For example, in FIG. 12, lens 1260 may be positioned to receive light from scanner 405 to spread the light over a wider field of view than is accessible from the scanner. In FIG. 13, telescope optics 1362 can be used to widen the range of reflected photons, which can provide a larger pixel array (eg, more pixels) in the receiver.

Figure 14 shows another embodiment of a receiver 1406 that may be used, for example, to provide a system that reduces potential harm to a viewer. In this system, the first scan is performed using a near-infrared or infrared light source (eg, a 1550nm NIR laser), which typically does not damage the viewer's retina. The first set of pixels 1420a of the receiver 1406 is designed to detect corresponding photons. A second scan can be performed using a visible laser, such as a blue diode laser, but this scan emits only short pulses of light. The second set of pixels 1420b of the receiver 1460 is designed to detect these photons. Alternatively, a second scan is performed using a near-infrared or infrared light source, followed by a third scan using a visible laser. Infrared or near-infrared sources can be higher in energy and be continuous or intensely pulsed over long ranges to achieve concentrated bursts.

In at least some embodiments, when the system finds objects within the field of view using the first scan (and optionally the second scan), the system may decide to refine the range using pulses from the visible laser. These pulses may utilize the anticipatory activation technique described above, where a pulse is only emitted when the first scan indicates that the object is within the range of interest. Pulses of visible light can thus be very sparse, but they are easily resolved by an array of tiny pixels. These pixels 1420b may even be smaller than the pixels in the first set of pixels 1420a, as shown in FIG. 14 . As an example, a receiver may have a 10mm line, where a row of 1000 10 micron SPADs is designed to detect 1500nm photons and a second row has, for example, 10,000 1 micron blue sensitive pixels (or alternatively, a second photoreceiver with multiple The main scanner is co-located with a less resolution 1550nm sensitive array, eg InGaAs). These two individual receivers or two rows of pixels will be positioned with the optical center aligned with the axis of the scanner.

In at least some embodiments, scanner 405 may operate on a reduced field of view to provide faster scanning and more pixels per degree of field of view. This can yield higher relative angular resolution and more accurate temporal resolution. Such an arrangement is shown in Figure 15, where graph 1570 corresponds to the angular deflection of scanner 405 over time. A solid line extending from scanner 405 represents the full field of view. However, if the field of view is limited to the dashed lines in FIG. 15 , the scanner 405 operates in the region between the dashed lines on graph 1570 . Receiver 406 is configured to receive light only from the reduced field of view.

Using the techniques described above, including the anticipatory activation method, the system can reliably detect objects even in fog or drizzle. A probabilistic prediction model (eg Bayesian model) is provided that looks at photons arriving at a pixel within a very short time interval. Arriving first are those photons that take the shortest path and return exactly from the direction they were sent in the pixel-sequential scan. With this in mind, gated pixels (such as those gated using the prospective activation method described above) then expect light to arrive at short predictable intervals. Using this method of anticipatory activation not only filters ambient light, but also distinguishes light from other directions, for example, light that ends up traveling through indirect (i.e., longer) paths, such as those scattered or deflected by fog or raindrops.

In a conventional camera, even with strong headlights (especially with strong headlights), the pixels in the array can see "bundle bins" in their respective light directions (required for human vision to distinguish The ratio ("AKA 20/20 Vision") of the system is 1/60 degree by 1/60 degree) for all light in the end. Coarser systems, such as traditional lidar APDs and SPADS, can typically resolve only one degree of squareness, which is 3600 times larger than the beams of light seen by the CMOS camera pixels in a mobile phone. Therefore, in these traditional coarse-scan lidar systems, more (a higher proportion) of stray and partially scattered light ends up in each bin.

When the light from the headlights is scattered by fog or drizzle, they disappear from the straight path they are supposed to travel. This has two effects: 1) Any scatter paths they follow are defined as longer paths than the straight light path from the source to the object surface, and the straight return path from that surface back to the detector. 2) When a ray deviates from a straight path, it will most likely end up illuminating the surface at a different location and will end up in another pixel in the SSA even without further scattering. And if the reflected light is scattered further on the way back, it's even less likely that it will end up in the aperture of the detector and anywhere near the line pixel directly.

Thus, in the described system, the activation of the pixels is controlled using pulsed emission in conjunction with the expected activation of the pixels, the reception of the pixels will be highly selective and filter out the vast majority of all scattered light. Each pixel only sees light that traveled the shortest path and arrived at exactly the expected time. The signal is reduced (or filtered) to only photons captured by selectively activated pixels in the receiver (where each pixel is activated for a specific nanosecond). The system can select (or focus on) only the rays that are directly emitted and directly reflected, the first arriving photons that actually touch the surface of the object in the fog are those that traveled the shortest path there (the object's surface) and returned. This is shown in Figure 16A, where unscattered light 1611 reflected by object 1608 is received and detected by activated pixel 1606i, but scattered light 1611' is directed to other inactive pixels of the receiver and is therefore not detected. Similarly, as shown in FIG. 16B , in a triangular lidar system (where light from scanner 1605 is reflected at an angle from object 1608 toward receiver 1606), light 1611' scattered by fog or drizzle is generally not detected by Activation of pixel 1606i is detected.

Conversely, reflected or scattered light arriving from that direction (either too early in time or too late) in the direct path may be filtered by the system. The shorter the activation period (for example, from 1 nanosecond to 10 nanoseconds for a coarse scan, or for example 100 picoseconds to 500 picoseconds for a fine scan), the greater the selectivity, which favors unscattered photons.

It will be understood that each block of the flowchart illustration, and combinations of blocks in the flowchart illustration (or acts explained above with reference to one or more systems or combinations of systems), can be implemented by computer program instructions. These program instructions may be provided to a processor to produce a machine, such that the instructions, which execute on the processor, create means for implementing the actions specified in one or more flowchart blocks. Computer program instructions may be executed by a processor such that a series of operational steps are performed by the processor to produce a computer-implemented process such that the instructions executed on the processor provide for implementing the actions specified in one or more flowchart blocks A step of. The computer program instructions may also cause at least some of the operational steps shown in the blocks of the flowchart to be performed in parallel. In addition, some of the steps may also be performed across multiple processors, such as might occur in a multi-processor computer system. Furthermore, one or more blocks or combinations of blocks in the flowcharts may be executed concurrently with other blocks or combinations of blocks, or even in an order different from that shown without departing from the scope or spirit of the invention. Execute sequentially.

In addition, one or more steps or blocks may be implemented using embedded logic hardware, such as Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), Programmable Array Logic (PALs), etc., or other combination without the use of computer programs. Embedded logic hardware may directly execute embedded logic to perform some or all of the actions in one or more steps or blocks. Furthermore, in one or more embodiments (not shown in the figures), some or all of the actions in one or more steps or blocks may be performed by a hardware microcontroller instead of a CPU. In one or more embodiments, the microcontroller can directly execute its own embedded logic to perform actions and access its own internal memory and its own external input and output interfaces (e.g., hardware pins and/or wireless Transceiver) to perform actions, such as system-on-chip (SOC), etc.

The above specification, examples and data provide a complete description of the manufacture and use of the compositions of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended.

Claims (30)

CLAIMS 1. A method for measuring the extent to a surface of one or more objects, the method comprising: a) scanning the continuous light beam on the field of view in the first scanning channel by the first scanner; b) detecting, by a receiver, photons of said continuous light beam reflected from one or more portions of a surface of said one or more objects, wherein said receiver comprises a plurality of pixels; c) determining, by one or more processor devices, the time of departure of the photons of the continuous beam from the scanner and the arrival time of the photons at the receiver, to the one or more objects the rough extent of said one or more portions of the surface; d) scanning a plurality of light pulses across the field of view in a second scanning channel by the first scanner; e) detecting, by the receiver, photons from the plurality of light pulses reflected from the one or more portions of the surface of the one or more objects; and f) determining, by the one or more processor devices, to the one or more The fine extent of the one or more portions of the surface of the object. 2. The method of claim 1, wherein determining the coarse range comprises determining a time of departure of the photon based on a location of a pixel at which the receiver detected the photon. 3. The method of claim 1, wherein the pulse width of the light pulse is any one of: a) not greater than 1 nanosecond, or b) less than the scan time of the second scan channel Divide by the number of pixels in a single row of the receiver. 4. The method of claim 1 , further comprising: repeating steps d) through f) one or more times to further refine the fine range, wherein, in each repetition, the light pulses for the repetition Shift in time the light pulses from each previously scanned channel. 5. The method of claim 1, further comprising: repeating steps d) through f) one or more times, wherein, in each repetition, the light pulse for the repetition has the same of light pulses of different colors. 6. The method of claim 5, wherein the scanner comprises rows of pixels, wherein for each different color of light pulses, one or more rows of pixels are configured to detect that color of light. 7. The method of claim 6, wherein the continuous light beam is a near-infrared light beam and the receiver includes one or more rows of pixels configured to detect near-infrared light. 8. The method of claim 1, wherein scanning the continuous beam comprises scanning the said continuous beam over a field of view, wherein said second scanner scans along a different axis than said first scanner; and The plurality of light pulses across the field of view are scanned in the second scanning channel by an ordered combination of the first scanner and the second scanner. 9. The method of claim 8, further comprising repeating steps a) through f) to scan the two-dimensional field of view. 10. The method of claim 8, further comprising one or more of the following: a) the combination of the first scanner and the second scanner is configured to scan the two-dimensional field of view in a focused mode; or b) the scan period of the second scanner does not exceed 1% of the scan period of the first scanner; or c) said receiver comprises rows of pixels spaced apart to account for movement of said second scanner relative to said first scanner. 11. A system for measuring extent to a surface of one or more objects, comprising: a first scanner configured to scan the received light across the field of view; a receiver comprising a plurality of pixels, wherein each pixel is configured to detect photons received by the pixel; one or more memory devices that store instructions; and One or more processor devices that execute the stored instructions to perform actions, including: a) scanning the continuous light beam on the field of view in the first scanning channel by the first scanner; b) detecting, by said receiver, photons of said continuous beam of light reflected from one or more portions of a surface of said one or more objects; c) determining, by the one or more processor devices, to the one or more a rough extent of the one or more portions of the surface of the object; d) scanning a plurality of light pulses across the field of view in a second scanning channel by the first scanner; e) detecting, by the receiver, photons from the plurality of light pulses reflected from the one or more portions of the surface of the one or more objects; and f) determining, by the one or more processor devices, to the one or more The fine extent of the one or more portions of the surface of the object. 12. The system of claim 11, wherein determining the coarse range includes determining a time of departure of the photon based on a location of a pixel at the receiver that detected the photon. 13. The system of claim 11 , wherein the instructions are configured such that the pulse width of the light pulse is any of: a) no greater than 1 nanosecond, or b) less than the The scan time of the second scan channel is divided by the number of pixels in a single row of the receiver. 14. The system of claim 11 , wherein the actions further comprise: repeating actions d) to f) one or more times to further refine the fine range, wherein, in each repetition, for The repeated light pulses are offset in time from the light pulses from each previously scanned channel. 15. The system of claim 11 , wherein the acts further comprise: repeating acts d) through f) one or more times, wherein in each repetition the light pulse for the repetition has the same A different color for the light pulse of each previously scanned channel. 16. The system of claim 15, wherein the scanner comprises rows of pixels, wherein for each different color of light pulses, one or more rows of pixels are configured to detect that color of light. 17. The system of claim 15, wherein the light source is configured to emit the continuous light beam as a near-infrared beam, and the receiver includes one or more rows of pixels configured to detect near-infrared light. 18. The system of claim 11, wherein scanning the continuous beam comprises scanning the said continuous beam over a field of view, wherein said second scanner scans along a different axis than said first scanner; and The plurality of light pulses across the field of view are scanned in the second scanning channel by an ordered combination of the first scanner and the second scanner. 19. The system of claim 18, further comprising one or more of the following: a) the combination of the first scanner and the second scanner is configured to scan a two-dimensional field of view in a focused mode; or b) The scanning period of the second scanner does not exceed 1% of the scanning period of the first scanner. 20. The system of claim 18, wherein the receiver includes rows of pixels spaced apart to account for movement of the second scanner relative to the first scanner. 21. A non-transitory processor-readable storage medium comprising instructions for measuring a range to a surface of one or more objects, wherein execution of said instructions by one or more processor devices causes said one or a plurality of processor devices to perform actions, the actions comprising: a) scanning the continuous light beam on the field of view in the first scanning channel by the first scanner; b) detection of photons of said continuous light beam reflected from one or more parts of the surface of said one or more objects by a receiver, wherein said receiver comprises a plurality of pixel; c) determining, by one or more processor devices, the time of departure of the photons of the continuous beam from the scanner and the arrival time of the photons at the receiver, to the one or more objects the rough extent of said one or more portions of the surface; d) scanning a plurality of light pulses across the field of view in a second scanning channel by the first scanner; e) detecting, by the receiver, photons from the plurality of light pulses reflected from the one or more portions of the surface of the one or more objects; and f) determining, by the one or more processor devices, to the one or more The fine extent of the one or more portions of the surface of the object. 22. The non-transitory processor readable storage medium of claim 21, wherein determining the coarse range comprises determining a time of departure of the photon based on a location of a pixel at which the receiver detected the photon. 23. The non-transitory processor readable storage medium of claim 21 , wherein the pulse width of the light pulse is any one of: a) not greater than 1 nanosecond, or b) less than the The scan time of the second scan channel is divided by the number of pixels in a single row of the receiver. 24. The non-transitory processor-readable storage medium of claim 21 , wherein the actions further comprise: repeating actions d) to f) one or more times to further refine the fine range, wherein In each repetition, the light pulses used for that repetition are offset in time from the light pulses from each previously scanned channel. 25. The non-transitory processor-readable storage medium of claim 21, wherein the actions further comprise: repeating actions d) to f) one or more times, wherein, in each repetition, for This repeated light pulse has a different color than the light pulse from each previously scanned channel. 26. The non-transitory processor readable storage medium of claim 25 , the scanner comprising rows of pixels, wherein for each light pulse of a different color, one or more rows of pixels are configured to detect that color of light. 27. The non-transitory processor readable storage medium of claim 25, wherein the continuous light beam is a near-infrared light beam, and the receiver includes one or more rows of pixels configured to detect near-infrared light. 28. The non-transitory processor readable storage medium of claim 21 , wherein scanning the continuous light beam comprises: scanning the continuous light beam by an ordered combination of the first scanner and a slower second scanner at the scanning the continuous light beam over the field of view in a first scanning channel, wherein the second scanner scans along a different axis than the first scanner; and The plurality of light pulses across the field of view are scanned in the second scanning channel by an ordered combination of the first scanner and the second scanner. 29. The non-transitory processor readable storage medium of claim 28, further comprising one or more of: a) the combination of the first scanner and the second scanner is configured to scan a two-dimensional field of view in a focused mode; or b) The scanning period of the second scanner does not exceed 1% of the scanning period of the first scanner. 30. The non-transitory processor readable storage medium of claim 28, wherein the receiver comprises rows of pixels spaced apart to account for of the mobile.