RU2800694C2 - Method for predicting the trajectory of an agent near an unmanned vehicle based on the ranking - Google Patents

Abstract

FIELD: unmanned vehicles.

SUBSTANCE: invention relates to computer-implemented methods and systems for controlling an unmanned vehicle, in particular to methods and systems for predicting the trajectory of an agent, such as a pedestrian or a human-controlled vehicle, in the vicinity of an unmanned vehicle. The method includes predicting the trajectory of the agent near the unmanned vehicle. This is done by: receiving sensor data indicative of the current situation in the vicinity of the unmanned vehicle; generating a feature vector of the current situation based at least in part on the sensor data; searching the pre-built library feature vector bank to identify the resulting feature vector from the pre-built bank that is most relevant to the current situation feature vector, where each library feature vector in the pre-built bank is associated with an observed trajectory; and predicting the agent's trajectory based on the resulting feature vector. The method also includes planning a vehicle action based on the agent's predicted trajectory and controlling the vehicle in accordance with the planned action.

EFFECT: increases road safety during operation.

20 cl, 6 dwg

Description

The field of technology to which the invention relates

[0001] The present technology relates to computer-implemented methods and systems for controlling an unmanned vehicle, and more particularly to methods and systems for predicting the trajectory of an agent, such as a pedestrian or a human-controlled vehicle, in the vicinity of an unmanned vehicle.

State of the art

[0002] Unmanned vehicles are vehicles that can autonomously move around private and/or public areas. Using a system of sensors that detect the location and/or surroundings of the vehicle, logic within or associated with the vehicle controls the speed, propulsion, braking, and steering of the vehicle based on the location and surroundings of the unmanned vehicle detected by the sensor.

[0003] A variety of sensor systems may be used in an unmanned vehicle, such as, but not limited to, camera systems, radar systems, and LIDAR systems. Different sensor systems may be used to collect different information and/or information in different formats about the location and surroundings of the unmanned vehicle. For example, camera systems can be used to collect image data about the surroundings of an unmanned vehicle. In another example, LIDAR systems can be used to acquire point cloud data to build 3D mapping representations of the environment and other potential objects located in the vicinity of an unmanned vehicle.

[0004] One of the problems associated with driving an unmanned vehicle is correctly predicting the movement of agents such as pedestrians or human-driven vehicles. Overall traffic safety and passenger comfort can directly depend on the ability to accurately make such predictions. If such a prediction is incorrect, it is possible that this could lead to abrupt maneuvers or even a collision.

[0005] The problem of motion prediction also remains one of the most difficult problems in developments related to autonomous control, mainly due to the high variability in the possible future behavior of an agent, given the current situation at the time of prediction. The two phenomena responsible for variability are multimodality, caused by the ambiguity of the agent's intention (for example, turn right or move forward) and uncertainty in the implementation of a certain intention(For example, which lane to turn into.

[0006] To be useful in a real-time autonomous driving process, a motion prediction system must provide efficient ways to describe and quantify this uncertainty, such as calculating the most likely trajectories for an agent and their probabilities, or estimating the relative probability for a particular trajectory. It also should not pay much attention to physically impossible trajectories, since they can in some way confuse the system that processes the forecasts.

Disclosure of invention

[0007] Implementations of the disclosed technology predict the movement of agents, such as pedestrians and human-driven vehicles, in the vicinity of an unmanned vehicle. Such forecasts should be made with a low delay and avoid predicting implausible agent trajectories. In accordance with various implementations of the disclosed technology, this is achieved by using a pre-created trajectory bank containing feature vectors that are associated with previously observed agent trajectories. The unmanned vehicle collects sensor data indicative of the current situation in the vicinity of the unmanned vehicle. This sensor data is used to generate situation data that is mapped to a feature vector indicative of the current situation. Then, the pre-built trajectory bank is searched for feature vectors that are relevant (eg, being spatially close) to the feature vector representing the current situation, and the trajectory associated with the corresponding feature vector from the trajectory bank is used to predict the agent's trajectory.

[0008] Since the predicted trajectory is selected from a pre-created trajectory bank of the actual observed trajectories of the agents, the predicted trajectory will be physically plausible. In addition, by effectively transforming the agent trajectory prediction problem into a kind of search problem, the computational complexity of agent trajectory prediction is significantly reduced compared to trying to generate a prediction from scratch. This can lead to a decrease in the delay in predicting the trajectory of agents. Accordingly, implementations of the disclosed technology will improve the operation of unmanned vehicles.

[0009] In accordance with one aspect of the present invention, the technology is implemented in a method for controlling an unmanned vehicle. The method includes predicting, by means of a computing device associated with the unmanned vehicle, the trajectory of an agent in the vicinity of the unmanned vehicle. This is done by: receiving sensor data indicative of the current situation in the vicinity of the unmanned vehicle from a sensor associated with the unmanned vehicle; generating a feature vector of the current situation based at least in part on the sensor data; searching the pre-built library feature vector bank to identify the resulting feature vector from the pre-built bank that is most relevant to the current situation feature vector, each library feature vector in the pre-built bank being associated with an observed trajectory; and predicting the agent's trajectory at least in part based on the resulting feature vector. The method also includes planning, by means of the computing device, the action of the unmanned vehicle based at least in part on the predicted trajectory of the agent, and controlling the unmanned vehicle in accordance with the planned action.

[00010] In some implementations, generating the current situation feature vector includes using a pre-trained neural network to generate the current situation feature vector. In some of these implementations, using a pretrained neural network includes providing a feature map representing bird's eye view angles of the current situation in the vicinity of the unmanned vehicle based at least in part on sensor data as input to the pretrained neural network. In some implementations, the feature map is agent-centered and has an orientation based on the agent's orientation.

[00011] In some implementations, searching a pre-created library feature vector bank includes performing an approximate nearest neighbor search. In some of these implementations, the library's pre-built feature vector bank includes a pointer based on a traversable hierarchical cramped world graph, and performing an approximate nearest neighbor search involves traversing the traversable hierarchical cramped world graph.

[00012] In some implementations, searching a pre-created library feature vector bank includes generating a ranked list of feature vectors from the pre-created bank. In some of these implementations, generating a ranked list of feature vectors includes ranking the library feature vectors at least in part based on a measure of the relevance of the library feature vectors to the current situation feature vector.

[00013] In some implementations, the resulting feature vector includes the library feature vector closest to the current situation feature vector.

[00014] In accordance with other aspects of the present disclosure, a method is provided for generating a bank of library feature vectors for use in predicting the trajectory of an agent in the vicinity of an unmanned vehicle being used. The method includes: receiving at the server a plurality of observation samples, each observation sample obtained by a sensor on the data-acquisition unmanned vehicle, each observation sample including data on the trajectory of the observed agent near the data-acquisition unmanned vehicle for a given period time; fetching at the server at least a first portion of the plurality of observation samples to obtain a library of agent trajectories; formation on the server of a library feature vector for each trajectory in the library of agent trajectories; collecting, at the server of the library feature vectors, at least in part based on the library feature vectors; and sending the library feature vector bank to a computing device associated with the unmanned vehicle being used.

[00015] In some implementations of this aspect, sampling at the server at least a first portion of a plurality of observation samples to obtain a library of agent trajectories includes clustering agent trajectories from at least a first portion of a plurality of observation samples into a plurality of clusters, where each cluster in the plurality of clusters comprises similar agent trajectories. A cluster is randomly selected from a set of clusters, and the agent's trajectory is randomly selected from the selected cluster.

[00016] In some implementations, collecting the library feature vector bank server further includes generating an index of the library feature vectors in the library feature vector bank. In some of these implementations, generating the library feature vector index includes generating a proximity graph of the library feature vectors. In some implementations, the proximity graph includes a hierarchical roamable close world graph.

[00017] In some implementations, server generation of a library feature vector for each trajectory in the agent trajectory library includes using a neural network to generate a library feature vector from each trajectory in the agent trajectory library.

[00018] In some implementations, each observation sample further includes data about the situation surrounding the observed agent. In addition, the method further includes: using at least a second part of the plurality of observation samples to train the first neural network to generate the first feature vectors based on the input data of situations in the environment of the observed agents; and using at least a second part of the plurality of observation samples to jointly train the second neural network to generate second feature vectors based on the observed agent trajectory inputs. The first feature vectors and the second feature vectors will be close when the trajectory of the observed agent corresponds to the situation in the environment of this observed agent.

[00019] In accordance with a further aspect of the present invention, the technology is implemented on a server. The server includes a processor and a memory coupled to the processor, the memory containing programmed instructions that, when executed by the processor, cause the processor to: the sample of observations includes data on the trajectory of the observed agent near the unmanned vehicle collecting data during a given period of time; fetch at the server at least a first portion of the plurality of observation samples to obtain a library of agent trajectories; form on the server a library feature vector for each trajectory in the agent trajectory library; collect on the server a bank of library feature vectors based at least in part on the library feature vectors; and provide a bank of library feature vectors to a computing device associated with the unmanned vehicle.

[00020] In some implementations, the memory includes programmed instructions that, when executed by the processor, cause the processor to fetch at the server at least a first portion of a plurality of observation samples to obtain a library of agent trajectories by: clustering agent trajectories from at least a first portion of a plurality of observation samples into a plurality of clusters, each cluster in the plurality of clusters containing similar agent trajectories; to randomly select a cluster from a plurality of clusters; and perform a random selection of the agent's trajectory from the selected cluster.

[00021] In accordance with another aspect of the present invention, the technology is implemented in an unmanned vehicle. The unmanned vehicle includes at least one sensor. The unmanned vehicle further includes a computing device including a processor and a memory. The memory stores programmed instructions that, when executed by the processor, cause the computing device to predict the trajectory of an agent in the vicinity of the unmanned vehicle by: receiving sensor data indicative of the current situation in the vicinity of the unmanned vehicle from at least one sensor; generating a feature vector of the current situation based at least in part on the sensor data; searching the pre-created library feature vector bank to determine the resulting feature vector from the pre-created bank that is most relevant to the current situation feature vector, each library feature vector in the pre-created bank being associated with an observed trajectory; and predicting the agent's trajectory at least in part based on the resulting feature vector. The memory further stores programmed instructions that, when executed by the processor, cause the computing device to plan the action of the unmanned vehicle based at least in part on the predicted trajectory of the agent, and to control the unmanned vehicle in accordance with the planned action. In some implementations, the unmanned vehicle is an unmanned vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

[00022] These and other features, aspects, and advantages of the present technology will become more apparent from the following description, the appended claims, and the accompanying drawings, in which:

[00023] In FIG. 1 is a schematic diagram of an exemplary computer system for use in some implementations of the systems and/or methods of the present technology.

[00024] In FIG. 2 shows a networked computing environment including an unmanned vehicle suitable for use with some implementations of the present technology.

[00025] In FIG. 3 shows a general view of a situation in which various implementations of the disclosed technology may be used.

[00026] FIG. 4 is a flowchart depicting an exemplary agent trajectory prediction scenario in accordance with various implementations of the disclosed technology.

[00027] In FIG. 5 shows a flowchart of a trajectory bank generation method and machine learning model used in an agent trajectory prediction method in accordance with various implementations of the disclosed technology.

[00028] In FIG. 6 shows a flow diagram of a method for predicting an agent's trajectory in accordance with various implementations of the disclosed technology.

Implementation of the invention

[00029] Various exemplary implementations of the disclosed technology will be more fully described below with reference to the accompanying drawings. However, the present technology may be implemented in many different forms and should not be construed as being limited to the exemplary implementations set forth herein. In the drawings, the sizes and relative sizes of layers and regions may be enlarged for clarity. Throughout the document, like reference numerals refer to like elements.

[00030] The examples and conventions provided herein are intended primarily to assist the reader in understanding the principles of the present technology, and not to limit its scope to such specific examples and terms. It should be understood that those skilled in the art will be able to devise various configurations which, although not expressly described herein, nevertheless implement the principles of the present technology and are included within its spirit and scope.

[00031] In addition, for ease of understanding, in the following description, implementations of the present technology may be described in a relatively simplified manner. Those skilled in the art will appreciate that various implementations of the present technology may be more complex.

[00032] In some cases, what are considered useful examples of modifications to the present technology may also be set forth. This is only done to facilitate understanding and, again, not to define the scope or delimitation of the present technology. These modifications are not intended to be an exhaustive list, and other modifications may be made by one skilled in the art without departing from the scope of the present technology. In addition, cases where examples of modifications are not given should not be understood to mean that modifications are not possible and/or that what is described is the only way to implement such an element of the present technology.

[00033] It should be understood that although the terms "first", "second", "third", etc., may be used in this document. to describe the various elements, these elements should not be limited to these concepts. These concepts are used to distinguish one element from another. Thus, the first element described below could be called the second element without departing from the spirit of the present invention. As used herein, "and/or" includes any and all combinations of one or more of the related listed elements.

[00034] It should be understood that when an element is referred to as "connected" or "associated" with another element, it may be directly connected to or associated with another element, or intermediate elements may be present. In contrast, when an element is referred to as "directly connected" or "directly connected" to another element, there are no intermediate elements. Other words used to describe the relationship between elements should be interpreted in a similar way (eg "between" or " directly between", "adjacent" or "immediately adjacent", etc.).

[00035] The terminology used herein is only intended to describe specific exemplary implementations and is not intended to limit the present technology. The singular forms used herein are intended to include the plural forms, unless the context clearly indicates otherwise. In addition, it should be understood that the terms "comprises" and/or "comprising" when used in this description indicate the presence of the mentioned features, integers, steps, operations, elements and/or components, but do not exclude the presence or addition of one or more other signs, integers, stages, operations, elements, components and/or their groups.

[00036] The functions of the various elements shown in the drawings, including any functional unit labeled "processor", may be provided through the use of dedicated hardware as well as hardware capable of executing software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared. In some implementations of the present technology, the processor may be a general purpose processor, such as a central processing unit (CPU), or a processor dedicated to a specific purpose, such as a digital signal processor (DSP). In addition, explicit use of the term "processor" should not be construed as referring solely to hardware capable of executing software, and may implicitly include, but is not limited to, application specific integrated circuit (ASIC), field programmable gate array (FPGA), DC a memory device (ROM) for storing software, random access memory (RAM) and non-volatile storage device. Other hardware, conventional and/or specialized, may also be included.

[00037] Software modules, or simply modules or blocks that are meant to be software, may be represented herein as any combination of flowchart elements or other elements indicating the execution of process steps and/or textual description. Such modules may be executed by the hardware that is shown explicitly or implicitly. In addition, it should be understood that a module may include, for example, but not limited to, computer program logic, computer program instructions, software, stack, firmware, hardware circuits, or combinations thereof that provide the desired capabilities.

[00038] As used herein, a "database" is any structured collection of data, regardless of its particular structure, database management software, or computer hardware on which the data is stored, implemented, or otherwise provided for use. The database may reside on the same hardware as the process that stores or uses the information stored in the database, or it may reside on separate hardware such as a specialized server or multiple servers.

[00039] The present technology may be implemented as a system, method, and/or computer program product. The computer program product may include a computer-readable storage medium (or media) storing computer-readable program instructions that, when executed by a processor, cause the processor to perform aspects of the disclosed technology. The computer-readable storage medium may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disk, hard drive, random access memory (RAM), read only memory (ROM), flash memory, optical disk, memory card, floppy disk, mechanically or visually encoded media(For example,punch card or barcode) and/or any combination thereof. In the context of this document, a computer-readable storage medium should be considered as a permanent computer-readable medium. It should not be considered as an intermittent signal such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating along a waveguide or other transmission media. (for example, light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[00040] It should be understood that computer-readable program instructions may be downloaded to appropriate computing or processing devices from a computer-readable storage medium or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. A network interface in a computing/processing device may receive computer-readable program instructions via a network and forward the computer-readable program instructions for storage on a computer-readable storage medium in an associated computing or processing device.

[00041] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, machine instructions, firmware instructions, configuration data for an integrated circuit, or either source code or object code written in any combination of one or more programming languages. The machine-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network.

[00042] All statements in this document that set forth the principles, aspects, and implementations of the present technology, as well as specific examples thereof, are intended to cover both their structural and functional equivalents, whether they are currently known or will be developed in the future. . Thus, for example, those skilled in the art will appreciate that any block diagrams herein represent conceptual views of an illustrative circuit that implements the principles of the present technology. Likewise, it should be understood that any flow diagrams, process flow diagrams, state transition diagrams, pseudocode, and the like represent various processes that can be substantially represented in computer readable program instructions. These machine-readable program instructions may be transmitted to a processor or other programmable data processing device to form a mechanism such that the instructions that are executed through the processor of a computer or other programmable data processing device form a means for implementing the functions/actions indicated in the block diagram and/or block or blocks of the flowchart. These computer-readable program instructions may also be stored on a computer-readable storage medium that can control a computer, a programmable data processing device, and/or other devices to function in a certain manner, such that the computer-readable storage medium on which the instructions are stored includes an article of manufacture, including includes instructions that implement aspects of the function/action specified in flowcharts, process flowcharts, state transition diagrams, pseudocode, and the like.

[00043] Machine-readable program instructions may also be downloaded to a computer, other programmable processing device, or other devices to cause a series of functional steps to be executed on the computer, other programmable device, or other devices to form a computer-implemented process such that the instructions that are executed on a computer, other programmable device, or other device, implement the functions/actions specified in flowcharts, process flowcharts, state transition diagrams, pseudocode, and the like.

[00044] In some alternative implementations, functions indicated in flowcharts, process flow diagrams, state transition diagrams, pseudocode, and the like may occur out of the order indicated in the drawings. For example, two blocks shown sequentially in a block diagram may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each of the functions noted in the drawings, and combinations of such functions, may be implemented by specialized hardware systems that perform said functions or activities, or by combinations of specialized hardware and computer instructions.

[00045] Having established these fundamentals, we now consider some non-limiting examples illustrating various implementations of aspects of the present invention.

computer system

[00046] In FIG. 1 shows computer system 100. Computer system 100 can be a multi-user computer, single-user computer, laptop computer, tablet computer, smartphone, embedded control system, or any other computer system that is currently known or will be developed later. In addition, it should be understood that some or all of the components of computer system 100 may be virtualized and/or cloud-based. As shown in FIG. 1, computer system 100 includes one or more processors 102, memory 110, memory interface 120, and network interface 140. These system components are interconnected via bus 150, which may include one or more internal and/or external buses ( not shown) (eg, PCI bus, Universal Serial Bus, Firewire IEEE 1394 bus, SCSI bus, Serial-ATA bus, etc.) to which various hardware components are electronically connected.

[00047] Memory 110, which may be random access memory or any other type of memory, may contain data 112, an operating system 114, and program 116. Data 112 may be any data that serves as input to or output from any program on a computer system. 100. Operating system 114 is an operating system such as MICROSOFT WINDOWS or LINUX. Program 116 may be any program or set of programs that includes programmed instructions that can be executed by a processor to control the actions performed by computer system 100.

[00048] Memory interface 120 is used to connect storage devices such as storage device 125 to computer system 100. One type of storage device 125 is a solid state drive that can use an integrated circuit assembly for persistent data storage. Another type of storage device 125 is a hard drive, such as an electromechanical device, that uses a magnetic storage device to store and retrieve digital data. Similarly, storage device 125 can be an optical drive, a card reader that accepts a removable memory card such as an SD card, or a flash memory device that can be connected to computer system 100, such as via a universal serial bus (USB ).

[00049] In some implementations, computer system 100 may use well-known virtual memory technologies that allow programs of computer system 100 to behave as if they had access to a large, contiguous address space, instead of accessing many smaller storage spaces, such as memory 110 and storage device 125. Thus, while data 112, operating system 114, and programs 116 are shown to reside in memory 110, those skilled in the art will appreciate that these elements are not necessarily contained entirely simultaneously in memory 110.

[00050] Processors 102 may include one or more microprocessors and/or other integrated circuits. Processors 102 execute software instructions stored in memory 110. When computer system 100 starts up, processors 102 may initially execute the boot procedure and/or software instructions that make up operating system 114.

[00051] Network interface 140 is used to connect computer system 100 to other computer systems or network devices (not shown) via network 160. Network interface 140 may include a combination of hardware and software that allows communication on network 160. In some implementations, network interface 140 may be a wireless network interface. Software at network interface 140 may include software that uses one or more network protocols to communicate over network 160. For example, network protocols may include TCP/IP (Transmission Control Protocol/Internet Protocol).

[00052] It should be understood that computer system 100 is only an example, and that the disclosed technology may be used with computer systems or other computing devices having other configurations.

Networked Computing Environment

[00053] In FIG. 2 shows a networked computing environment 200 suitable for use with certain non-limiting implementations of the present technology. Networked computing environment 200 includes a computing device 210 associated with vehicle 220 and/or associated with a user (not shown) that is associated with vehicle 220 (such as an operator of vehicle 220). The networked computing environment 200 also includes a server 235 that communicates with the computing device 210 via a communications network 240 (eg, the Internet, and the like).

[00054] In at least some non-limiting implementations of the present technology, computing device 210 is communicatively coupled to control systems of vehicle 220. Computing device 210 can be configured and configured to control various operating systems of vehicle 220, including, but not limited to , ECU (engine control unit), steering systems, braking systems, and signaling and lighting systems (i.e. headlights, brake lights and/or direction indicators). In such an implementation, vehicle 220 may be an unmanned vehicle.

[00055] In some non-limiting implementations of the present technology, networked computing environment 200 may include a GPS satellite (not shown) transmitting and/or receiving a GPS signal to computing device 210/from it. It should be understood that the present technology is not limited to GPS and may use a location technology other than GPS. It should be noted that the GPS satellite may not be used at all.

[00056] Vehicle 220 with which computing device 210 is associated may be any vehicle, recreational or otherwise, such as a private or commercial vehicle, truck, motorcycle, and the like. Although vehicle 220 is depicted as a ground vehicle, this may not be the case in all non-limiting implementations of the present technology. For example, in some non-limiting implementations of the present technology, vehicle 220 may be a waterborne vehicle, such as a boat, or an aircraft, such as an unmanned aerial vehicle.

[00057] Vehicle 220 may be a user driven or unmanned vehicle. In some non-limiting implementations of the present technology, it is contemplated that vehicle 220 may be implemented as an unmanned vehicle (SDC). Vehicle 220 may be a fully autonomous vehicle or a partially autonomous vehicle in which the human driver controls some aspects of the vehicle's operation while other aspects ( e.g. " autopilot" on the highway) are automated. It should be noted that the specific parameters of the vehicle 220 are not limiting, and these specific parameters include, for example: vehicle manufacturer, vehicle model, vehicle year, vehicle weight, vehicle dimensions, vehicle weight distribution, area vehicle surface, vehicle height, transmission type (such as 2x or 4x), wheel type, brake system, fuel system, mileage, vehicle identification number, engine size, or other vehicle characteristics or parameters.

[00058] According to the present technology, the implementation of the computing device 210 is not specifically limited. For example, computing device 210 may be implemented as a vehicle engine control unit, vehicle CPU, vehicle navigation device. (eg, TomTom™, Garmin™), tablet computer, personal computer built into the vehicle 220, and the like. Thus, it should be noted that computing device 210 may or may not be permanently associated with vehicle 220. In addition or alternatively, computing device 210 may be implemented in a wireless communications device such as a mobile phone (e.g., smartphone or wireless phone). ). In some implementations, computing device 210 has a display 270.

[00059] Computing device 210 may include some or all of the components of computer system 100 depicted in FIG. 1, depending on the particular implementation. In some implementations, computing device 210 is an onboard computer device and includes processors 102, storage 125, and memory 110. In other words, computing device 210 includes hardware and/or software and/or firmware, or a combination thereof, to data processing, as will be described in more detail below.

[00060] In some non-limiting implementations of the present technology, communication network 240 is the Internet. In alternative non-limiting implementations of the present technology, communications network 240 may be implemented as any suitable local area network (LAN), wide area network (WAN), private communications network, and the like. It should be explicitly understood that implementations for communication network 240 are provided for purposes of illustration only. A communication channel (not individually numbered) is provided between computing device 210 and communication network 240, the implementation of which will depend on , among other things, how computing device 210 is implemented. By way of example only, and not limitation, in those non-limiting implementations of the present technology , in which the computing device 210 is implemented as a wireless communication device, such as a smartphone or navigation device, the communication channel may be implemented as a wireless communication link. Examples of wireless links may include, but are not limited to, a 3G network link, a 4G link, and the like. Communication network 240 may also use a wireless connection to server 235.

[00061] In some implementations of the present technology, server 235 is implemented as a computer server and may include some or all of the components of computer system 100 of FIG. In one non-limiting example, server 235 is implemented as a Dell™ PowerEdge™ server running the Microsoft™ Windows Server™ operating system, but may also be implemented in any other suitable hardware, software and/or firmware, or combinations thereof. In the depicted non-limiting implementations of the present technology, server 235 is a single server. In alternative non-limiting implementations of the present technology, the functionality of server 235 may be distributed and may be implemented across multiple servers (not shown).

[00062] In some non-limiting implementations of the present technology, processors 102 of computing device 210 may communicate with server 235 to receive one or more updates. Such updates may include, but are not limited to, software updates, map updates, route updates, weather updates, and the like. In some non-limiting implementations of the present technology, the computing device 210 may also be configured to transmit to the server 235 certain operational data, such as routes traveled, traffic data, performance data, and the like. Some or all of such data transmitted between vehicle 220 and server 235 may be encrypted and/or anonymized.

[00063] It should be noted that the computing device 210 may use a variety of sensors and systems to collect information about the environment 250 of the vehicle 220. As seen in FIG. 2, the vehicle 220 may be equipped with multiple sensor systems 280. It should be noted that different sensor systems from the plurality of sensor systems 280 may be used to collect different types of data regarding the environment 250 of the vehicle 220.

[00064] In one example, a plurality of sensor systems 280 may include various optical systems , including but not limited to one or more camera-type sensor systems that are mounted on vehicle 220 and coupled to processors 102 of computing device 210. Generally speaking, one or more camera-type sensor systems may be configured to collect image data about various parts of the environment 250 of vehicle 220. In some cases, image data provided by one or more camera-type sensor systems may be used by computing device 210 to perform procedures. object detection. For example, computing device 210 may be configured to input image data provided by one or more camera type sensor systems to an object detection neural network (ODNN) that is trained to locate and classify potential objects in the environment 250 of vehicle 220.

[00065] In another example, the plurality of sensor systems 280 may include one or more radar type sensor systems that are mounted on vehicle 220 and communicatively coupled to processors 102. Generally speaking, one or more radar type sensor systems may be configured to use radio waves to collect data about various parts of the environment 250 of the vehicle 220. For example, one or more radar-type sensor systems can be configured to collect radar data about potential objects in the environment 250 of the vehicle 220, such data potentially representing a distance objects from a system of sensors such as radar, the orientation of the objects, the speed of movement and/or the speed of the objects, and the like.

[00066] In a further example, a plurality of sensor systems 280 may include one or more LIDAR systems that are mounted on vehicle 220 and coupled to communicate with processors 102. Generally speaking, the LIDAR system is configured to collect data about the environment 250 of the vehicle. means 220 used, for example, to build a multidimensional map of objects in the environment 250 of the vehicle 220. The LIDAR system can be installed or provided as an additional equipment on the vehicle 220 in various locations and / or in various configurations to collect information about the environment 250 of the vehicle funds 220.

[00067] For example, depending on the implementation of the vehicle 220 and the LIDAR system, the LIDAR system may be installed on the inside, top of the windshield of the vehicle 220. However, other locations for installing the lidar system are within the scope of the present invention, in including on the rear window, side windows, front hood, roof, front grille, front bumper or side of the vehicle 220.

[00068] In the context of the present technology, computing device 210 is configured to detect one or more objects in environment 250 of vehicle 220 based on data received from one or more camera systems and one or more LIDAR systems. For example, computing device 210 configured to detect a specific object in the environment 250 of vehicle 220 may be configured to identify LIDAR data and camera data associated with the specific object, generate an "embedding" representing features associated with the specific object, and object detection by forming a bounding box for the object.

Agent trajectory prediction

[00069] Implementations of the disclosed technology predict the movement of agents (ie, other traffic participants) surrounding an unmanned vehicle, such as pedestrians and human-driven vehicles. As with any vehicle, a self-driving vehicle takes time to change its speed. Sudden changes in speed and acceleration caused by incorrect prediction of the agent's trajectory can be uncomfortable for the passengers of an unmanned vehicle. In some cases, incorrect prediction of the movement of agents can even lead to collisions. Thus, the AV motion planning module must have a good idea of where the nearest agents may be in a few seconds so that it can plan for the AV while maintaining a safe distance. The traffic planning module can also greatly benefit from understanding when a situation is inherently multi-modal and many different futures are likely. In such situations, an unmanned vehicle may take extra care until the situation is cleared up.

[00070] In accordance with various implementations of the disclosed technology, one way to describe the future movement of an agent is to represent it in terms of probabilities for possible future trajectories agent (for example, in the form of a probability distribution over possible trajectories) based on information about the situation available at the time of forecasting.

[00071] For practical use in an unmanned vehicle, these probabilities must be calculated with a low delay so that the prediction can be quickly changed based on new information, such as a sudden change in the agent's acceleration, and the planning subsystem of the unmanned vehicle can quickly receive information about the change in the situation. In addition, it must be possible to efficiently access the expected and most probable future positions of agents so that the planning subsystem can use this information in real time. In addition, there should be a convenient way to quantify future uncertainty and possible multimodality.

[00072] Non-limiting embodiments of the present technology are directed to performing agent future trajectory prediction by ranking possible future trajectories based on a metric representing the probability that the agent will follow each trajectory from a given set of possible future trajectories. Various implementations of the disclosed technology use a discriminatory approach, using metric-based machine learning methods in combination with approximate nearest neighbor methods for prediction. According to various implementations, trajectory predictions are selected by evaluating a pre-prepared dictionary (also called a "library" or "bank") of trajectories, such as a representative subset of agent trajectories observed in real life by an unmanned vehicle recognition system. The bank of trajectories can be pre-processed offline. This means that during the operation of an unmanned vehicle, the amount and complexity of calculations for predicting the future trajectory of the agent is significantly reduced. During operation, the unmanned vehicle encodes the situation surrounding the unmanned vehicle or agent and performs prediction using approximate nearest neighbor search methods in the trajectory bank. In addition, in some implementations, Monte Carlo estimation methods may be used in the prediction.

[00073] In FIG. 3 and 4 provide an overview of the situation in which various implementations of the disclosed technology may be used. In the example shown in FIG. 3, an unmanned vehicle 302 (in this case, an unmanned vehicle) is moving in situation 304. An agent 306 (in this case, a pedestrian) is also moving in situation 304. Unmanned vehicle 302 uses various sensors (e.g., LIDAR , radar, cameras, etc.). not shown) to gather information about the situation, and uses well-known techniques such as machine vision object recognition algorithms to process sensor inputs to form a set of situation feature maps representing various aspects of the situation. In some implementations, these feature maps may represent the elements of the situation as top or bird's eye views. In some implementations, these feature maps use the same reference frame. For example, feature maps can be generated such that the agent of interest is located in the center of the map during prediction, and the y-axis of the map is aligned with the agent's orientation. Each of these feature maps represents some aspect of the situation, such as the location of a lane or crosswalk, the presence of an agent, speed or acceleration at a particular time, lane availability caused by traffic light conditions, and/or other aspects that can be detected using sensors in the unmanned vehicle 302.

[00074] Feature maps for aspects that change over time are generated over a given period of time. For example, situation 304 can be sampled over several seconds(For example,three seconds) with a fixed sampling rate (for example, two samples per second). Positions 310, 312, 314 and 316 are shown in FIG. 3 for an unmanned vehicle 302 at times t-_3,t-_2,t-_1,t₀, respectively. Positions 320, 322, 324 and 326 are shown for agent 306 at times t_-3, t_-2, t_-1, t₀, respectively. Combination static feature map and these time dependent feature maps can be used to represent the current situation of the unmanned vehicle.

[00075] As shown in FIG. 4, based on the collected information, a computing device (not shown) associated with the unmanned vehicle will attempt to predict the agent's future trajectory over the next few seconds in agent trajectory prediction module 400. To perform this prediction, the static situation attribute map 404 can be combined with the time-dependent situation attribute maps 406 using the merge function 408 . In some implementations, the merge function 408 may merge the static situation feature map 404 and the time dependent situation feature maps 406, although it should be understood that other merge functions may be used. The combination of situation feature maps generated by pooling function 408 is then used as input to a trained machine learning model 402, such as a pre-trained artificial neural network, that generates a feature vector 410 representing the current situation in the vicinity of the unmanned vehicle.

[00076] The feature vector 410 is then used to search a pre-created bank 412 of library feature vectors 414, in which each of the library feature vectors represents a possible trajectory for the agent. In some implementation, this search may take the form of calculating the relevance of a set of library feature vectors (which may include all library feature vectors) to feature vector 410. In some implementations, this search may be performed using an approximate K-nearest neighbor search, which may use, for example, a pre-built hierarchical tight-world graph available for movement (vehicle) such as that described in Y. A. Malkov and D. A. Yashunin, “ Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, No. 4, pp. 824-836, April 1, 2020 and other known or later developed K-nearest neighbor search algorithms or methods, or the search may be performed using another known or later search algorithm.

[00077] This search of the feature vector 410 in the pre-built bank 412 yields a set of relevance scores 416 for the set of library feature vectors. These relevance scores may, for example, be based on the spatial distance ( eg , Euclidean distance) of the library feature vectors from the feature vector 410, or other measures of relevance. These relevancy measures may, in some implementations, represent the likelihood that an agent will follow a future trajectory corresponding to a library feature vector. Once these relevance scores have been computed, the most relevant library feature vector may be selected as the "outcome" feature vector 418, and the predicted agent trajectory 420 may be based on the agent trajectory corresponding to the resulting feature vector 418. In some implementations, a set of library feature vectors may be ranked according to their relevance scores, and this ranking may be used to select the resulting feature vector 418 to be used to predict the agent's trajectory. In some implementations where the relevance score represents the probability that an agent follows a trajectory corresponding to a library feature vector, Monte Carlo methods may be used to select the resulting feature vector 418 to be used to predict the agent's trajectory.

[00078] Once the resulting feature vector 418 is selected, the agent's trajectory prediction can be based on the agent's trajectory from the trajectory bank that corresponds to the resulting feature vector 418. In some implementations, the resulting feature vector 418 may be based on more than one of the library feature vectors, and the predicted agent trajectory 420 may be based on more than one agent trajectory from the trajectory bank. In some implementations, the prediction method described above may be repeated for each agent of interest found in the situation. In some implementations, multimodality can be handled by extending the machine learning model to generate multimodal output as a set of feature vectors and weights, and then matching the feature vectors for each mode, with the results possibly aggregated by mode.

[00079] Once the predicted agent trajectory 420 has been prepared, the predicted agent trajectory 420 may be sent to the planning subsystem 430 associated with the unmanned vehicle. The planning subsystem will take into account the predicted trajectory for the agent when planning the actions of the unmanned vehicle, and the control of the unmanned vehicle will be carried out in accordance with these actions.

[00080] As will be understood from the above description with reference to FIG. 3 and 4, agent trajectory prediction uses a pretrained machine learning model, such as a pretrained neural network, to generate feature vectors representing the current situation based on feature maps. In addition, the pre-created trajectory bank is searched for library feature vectors that are relevant to these feature vectors of the current situation, the library feature vectors being associated with possible trajectories for the agent. Thus, in accordance with various implementations of the disclosed technology, an offline trajectory bank generation method and a machine learning model are used. A flowchart of such a method is described with reference to FIG. 5.

[00081] Method 500 is divided into two main sections: section 502 training machine learning model; and a path bank generation section 504 . The machine learning model training section 502 jointly trains two machine learning models such as artificial neural networks, including a first machine learning model that generates feature vectors of the current situation based on feature maps, and a second machine learning model that generates feature vectors in the same space. features as well as the feature vectors generated by the first machine learning model based on the observed agent trajectories. As described above, the first machine learning model is used in driving an unmanned vehicle. The second machine learning model is used offline when generating a bank of trajectories. In some implementations, both the first and second machine learning models may be Convolutional Neural Networks (CNNs).

[00082] In block 510 of training section 502 of the machine learning model, a data set of an unmanned vehicle is received or accessed from a database. An unmanned vehicle data set typically includes a large amount of data collected by numerous unmanned vehicles over several months or even years. In some implementations, the data set includes observation samples that record sensor readings from sensors such as LIDAR , radar, cameras, etc. that are associated with an unmanned vehicle. Sensor readings are collected for the period of time during which the vehicle is running and at a given frequency (e.g. 10 samples per second).

[00083] In block 512, training data including training situations and training paths is prepared from the unmanned vehicle dataset. Moving agents, such as vehicles or pedestrians, that have been observed by an unmanned vehicle in motion for more than a given period of time (e.g., five seconds) are used as prediction targets and their trajectories over a given period time (for example, five seconds), and at a specified sampling rate (for example, five samples per second) are used as test data for training machine learning models. Each situation also includes historical data for a given period. time (for example, three seconds) to the data to be used as test data. The situations include static situation feature map data and time dependent feature map data. In some implementations, the sensor data will be processed to generate feature map data for the situation in the form of top or bird's eye views having an appropriate reference frame. In some implementations, this reference frame may place the prediction target (ie, the agent) at the center of the feature map data, and may orient the feature map data such that the y-axis of the maps is aligned with the orientation of the prediction target.

[00084] It should be understood that the preparation of training situations and training trajectories, such as described with reference to block 512, may be performed prior to execution of section 502 training of the machine learning model. If so, block 512 may be omitted and training situations and training paths may be received or accessed from the database rather than from the sensors on which they are based.

[00085] In block 514, training situations and training trajectories are used to jointly train the first and second machine learning models based on the agent trajectory test data and the corresponding situation data. Training is performed in such a way that the first machine learning model generates feature vectors of the current situation based on feature maps from the situation data, and the second machine learning model generates feature vectors based on agent trajectories. Machine learning models are trained in such a way that the feature vectors generated by the first and second machine learning models are in a common latent space and will be spatially close if the observed agent trajectory input to the second machine learning model matches the feature maps from the given situation that are input in the first machine learning model. The feature vectors generated by the first and second machine learning models will be spatially far away when there is no such match.

[00086] At block 516, the fully trained first machine learning model is sent or otherwise transmitted to the UAV agent trajectory prediction module, and the fully trained second machine learning model is sent or otherwise transmitted to the trajectory bank generation section 504, as described below.

[00087] At block 520 of trajectory bank generation section 504, the observed trajectories of agents in real situations are received or accessed (eg, from a database) by trajectory bank generation section 504. In some implementations, these agent trajectories are based on sensor data from sensors on unmanned vehicles such as LIDAR, radar, and cameras. In some implementations, camera images combined with LIDAR and radar data cover a given time period.(For example,five seconds) with the set frequency of data acquisition by the sensor (e.g. 10 sensor samples per second).

[00088] In block 522, observed agent trajectories are sampled to obtain a library of agent trajectories. When sampled from observed real agent trajectories, each of the trajectories in the agent trajectory library will be physically plausible, since this or that agent followed it at least once. This still saves the possible problem of over-representing high probability trajectories in the library, such as stationary trajectories or uniform forward motion, as well as the possible under-representation or exclusion of unusual trajectories that should nonetheless be represented. To solve this problem, some implementations may sample first by applying a clustering procedure to the observed agent trajectories, and then by sampling first by randomly selecting a cluster (or an index of clusters) and then randomly selecting a trajectory in the selected cluster. With this clustering method, trajectories that previously had a high probability will not be sampled as often, making the induced distribution of trajectories from the sample much closer to uniform. Some implementations use a mini-batch K-means algorithm for clustering, such as that described in Sculley, D., “Web-scale k-means clustering”, Proceedings of the 19th International Conference on World Wide Web, 1177-1178, 2010, with Euclidean distance in the space of trajectories. This clustering algorithm is fast and produces results that can work quite well in practice. It should be understood that other known clustering algorithms, such as fully connected clustering, may be used and may provide more uniform distributions.

[00089] In block 524, a library trajectory feature vector is generated for each of the trajectories in the agent trajectory library. In some implementations, this is done using a trained machine learning model, such as a trained artificial neural network that has been trained and provided by the training section 502 of the machine learning model described above. As discussed above, this machine learning model is trained to generate a feature vector given an input trajectory. The library vectors of trajectory signs are associated with the trajectories of the agents on the basis of which they were formed.

[00090] In block 526, library trajectory feature vectors are collected to form a library trajectory feature vector trajectory bank. In some implementations, this can be done by generating an index of library feature vectors, which will significantly speed up the search for library feature vectors when using the trajectory bank. Since the search will search for library feature vectors that are spatially close to the target vector representing the current situation for the unmanned vehicle, one structure that can be used for said pointer is a proximity graph. As is well known to those skilled in the art, a proximity graph is a graph in which two vertices are connected by an edge if the vertices are spatially close to each other using a given distance metric. According to some implementations, one of the well-known types of proximity graph is a hierarchical moveable close-world graph. Methods for constructing such a graph and using it to perform an approximate search for the K-nearest neighbor are described, for example, in the publication by Malkov and Yashunin, the link to which is given above. It should be understood that other well-known or later developed index generation methods and/or other proximity graph-based index generation methods may also be used.

[00091] At block 528, the trajectory bank is sent to or otherwise transmitted to the agent trajectory prediction module of the unmanned vehicle. The agent trajectory prediction module is typically executed on the computing device associated with the unmanned vehicle and is described below with reference to FIG. 6.

[00092] It should be understood that the machine learning model training section 502 and the path bank generation section 504 may be performed "offline" on a server or other computing device prior to using the trained machine learning model and/or the path bank they generate when used on unmanned vehicle. In addition, the machine learning model training section 502 and the path bank generation section 504 may be executed on different computing devices and at different times.

[00093] Since the set of trajectories is fixed in advance, it can be limited to contain only physically plausible agent trajectories. This can simplify the modeling task for machine learning models and prevent completely wrong predictions from being made in areas of the problem space where prediction quality is poor. In some implementations, the set of trajectories may be enriched with trajectories corresponding to complex maneuvers, thus ensuring that such maneuvers can be predicted and processed.

[00094] In addition, by creating a bank of trajectories with a pointer, it is possible to perform prediction by evaluating potential solutions. Computationally, this is much easier than generating a forecast from scratch. Thus, by formulating the problem of predicting the trajectories of agents as an estimation problem, it is possible to improve the quality of forecasting while maintaining a small delay associated with the implementation of forecasts.

[00095] In FIG. 6 shows a flowchart of a method 600 for predicting agent trajectories in accordance with various implementations of the disclosed technology. In some implementations, method 600 is implemented as an agent trajectory prediction module running on a computing device associated with an unmanned vehicle.

[00096] At block 602, sensor data is received from sensors associated with the unmanned vehicle. The sensor data may include data from sensors such as LIDAR, radar, cameras, and other sensors used in the unmanned vehicle and may represent the current situation in the vicinity of the unmanned vehicle for a predetermined period of time before predicting the agent's trajectory.

[00097] In block 604, the sensor data is processed to generate feature map data for the situation in the form of top or bird's eye views having an appropriate reference frame. In some implementations, this reference frame may place the agent for which the trajectory is being generated at the center of the feature map data (e.g., using the center of the agent's bounding box as the center of the feature maps) and may orient the feature map data so that the map's y-axis is aligned with the agent's orientation.

[00098] In block 606, a feature vector representing the current situation is generated based on the feature map data for the situation. In some implementations, this can be done using a pre-trained machine learning model such as an artificial neural network that maps feature map data to a feature vector situation. This pre-trained machine learning model may be trained and provided by the training section 502 of the machine learning model described above with reference to FIG. 5.

[00099] At block 608, a pre-built library feature vector trajectory bank 414, in which each of the library feature vectors represents a possible trajectory for the agent, is searched for feature vectors that are related to the feature vector representing the current situation. In some implementations, this may take the form of calculating a relevance score for a set of library feature vectors. In some implementations, the search may be performed using an approximate K-nearest neighbor search, which may use, for example, a pre-built hierarchical cramped-world graph available for movement (vehicle) such as that described in Malkov and Yashunin, referenced above. It should be understood that other known or later developed K-nearest neighbor search algorithms or methods may be used, or the search may be performed using another known or later matching or search algorithm.

[000100] This search may generate a set of relevance scores for a set of library feature vectors. These relevance scores may, for example, be based on the spatial distance between the library feature vectors and the feature vector representing the current situation, or other measures of relevance. Once these relevance scores have been computed, the most relevant library feature vector can be selected as the resulting feature vector. In some implementations, a set of library feature vectors may be ranked according to their relevance scores, forming a ranked list. This ranked list can be used to select the resulting feature vector to be used to predict the agent's trajectory.

[000101] In block 610, the agent's trajectory is predicted based on the agent's trajectory from the trajectory bank that corresponds to the resulting feature vector. In some implementations, the prediction method described above may be repeated for each agent of interest that is found in the vicinity of the unmanned vehicle.

[000102] At block 612, the predicted trajectory may be sent to a planning subsystem associated with the unmanned vehicle. In some implementations, the planning subsystem runs on the same computing device as the agent trajectory prediction module. The planning subsystem takes into account the predicted trajectory for the agent when planning the actions of an unmanned vehicle. At block 614, the unmanned vehicle is controlled in accordance with these planned actions.

[000103] Modifications and improvements to the above-described implementations of the present technology may become apparent to those skilled in the art. The above description is intended to be exemplary and not limiting. Therefore, it is intended that the scope of the present technology be limited only by the scope of the appended claims.

Claims (59)

1. A method for controlling an unmanned vehicle, the method comprising the steps of: by means of a computing device associated with the unmanned vehicle, the trajectory of the agent near the unmanned vehicle is predicted by: receiving sensor data indicative of a current situation in the vicinity of the unmanned vehicle from a sensor associated with the unmanned vehicle; generating a feature vector of the current situation based at least in part on the sensor data; searching the pre-built library feature vector bank to identify the resulting feature vector from the pre-built bank that is most relevant to the current situation feature vector, each library feature vector in the pre-built bank being associated with an observed trajectory; And predicting the agent's trajectory at least in part based on the resulting feature vector; planning, by means of the computing device, the action of the unmanned vehicle based at least in part on the predicted trajectory of the agent; And control the unmanned vehicle according to the planned action. 2. The method according to claim. 1, in which the generation of the vector of features of the current situation contains the use of a pre-trained neural network to generate a vector of features of the current situation. 3. The method of claim 2, wherein using the pretrained neural network comprises providing a feature map representing bird's eye view angles of the current situation in the vicinity of the unmanned vehicle based at least in part on sensor data as input. for a pretrained neural network. 4. The method of claim 3, wherein the feature map is agent-centered and has an orientation based on the agent's orientation. 5. The method of claim 1, wherein the search in a pre-created library feature vector bank comprises the step of performing an approximate nearest neighbor search. 6. The method of claim 5, wherein the pre-created library feature vector bank includes a pointer based on a hierarchical cramped world graph available for movement, and wherein performing an approximate nearest neighbor search comprises traversing the hierarchical cramped world graph. available for movement. 7. The method of claim. 1, wherein the resulting feature vector contains a library feature vector that is closest to the feature vector of the current situation. 8. The method according to claim. 1, in which the search in a pre-created bank of library feature vectors includes the step of generating a ranked list of feature vectors from a pre-created bank. 9. The method of claim 8, wherein generating the ranked list of feature vectors comprises ranking the library feature vectors at least in part based on a measure of relevance of the library feature vectors to the current situation feature vector. 10. A method for generating a bank of library feature vectors for use in predicting the trajectory of an agent near a used unmanned vehicle, the method comprising the steps of: receiving at the server a plurality of observation samples, each observation sample received by a sensor on the data-acquiring unmanned vehicle, each observation sample including trajectory data of an observed agent in the vicinity of the data-acquiring unmanned vehicle for a predetermined period of time; sampling on the server at least the first part of the plurality of observation samples to obtain a library of agent trajectories; forming on the server a library feature vector for each trajectory in the agent trajectory library; collecting on the server a bank of library feature vectors based at least in part on the library feature vectors, and sending the bank of library feature vectors to a computing device associated with the unmanned vehicle being used. 11. The method according to claim 10, wherein sampling at the server at least the first part of the plurality of observation samples to obtain a library of agent trajectories comprises the steps of: clustering agent trajectories from at least a first part of the plurality of observation samples into a plurality of clusters, each cluster in the plurality of clusters containing similar agent trajectories; randomly choose a cluster from a set of clusters and randomly choose the trajectory of the agent from the selected cluster. 12. The method of claim 10, wherein assembling the library feature vector bank at the server further comprises generating an index of the library feature vectors in the library feature vector bank. 13. The method of claim 12, wherein generating the library feature vector index comprises generating a proximity graph of the library feature vectors. 14. The method of claim 13, wherein the proximity graph comprises a hierarchical close-world graph available for movement. 15. The method of claim 10, wherein generating on the server a library feature vector for each trajectory in the agent trajectory library comprises using a neural network to generate a library feature vector from each trajectory in the agent trajectory library. 16. The method of claim 10, wherein each sample of observations further includes data about a situation surrounding the agent being observed, and wherein the method further comprises: using at least a second part of the plurality of observation samples to train the first neural network to generate first feature vectors based on input data from situations surrounding the observed agents; And using at least a second part of the plurality of observation samples to jointly train the second neural network to generate second feature vectors based on the input data of the trajectories of the observed agents; And in this case, the first feature vectors and the second feature vectors will be close if the trajectory of the observed agent corresponds to the situation surrounding this observed agent. 17. Server containing: CPU; memory coupled to the processor, the memory containing programmed instructions which, when executed by the processor, cause the processor to: access at the server a plurality of observation samples, each observation sample collected by a sensor in the data collecting unmanned vehicle, each observation sample including data about the trajectory of the observed agent in the vicinity of the data collecting unmanned vehicle for a predetermined period of time; fetch at the server at least a first portion of the plurality of observation samples to obtain a library of agent trajectories; form on the server a library feature vector for each trajectory in the agent trajectory library; collect on the server a bank of library feature vectors based at least in part on the library feature vectors; And transfer the bank of library feature vectors to the computing device associated with the unmanned vehicle. 18. The server of claim 17, wherein the memory contains programmed instructions that, when executed by the processor, cause the processor to fetch at the server at least a first portion of the plurality of observation samples to obtain a library of agent trajectories by: clustering agent trajectories from at least a first portion of the plurality of observation samples into a plurality of clusters, each cluster in the plurality of clusters containing similar agent trajectories; random selection of a cluster from a set of clusters and random choice of the agent's trajectory from the selected cluster. 19. Unmanned vehicle, containing: at least one sensor; a computing device including a processor and a memory, the memory storing programmed instructions which, when executed by the processor, cause the computing device to: predict the trajectory of an agent near an unmanned vehicle by: receiving sensor data indicative of a current situation in the vicinity of the unmanned vehicle from at least one sensor; generating a feature vector of the current situation based at least in part on the sensor data; searching the pre-built library feature vector bank to identify the resulting feature vector from the pre-built bank that is most relevant to the current situation feature vector, each library feature vector in the pre-built bank being associated with an observed trajectory; And predicting the agent's trajectory based at least in part on the resulting feature vector; planning the action of the unmanned vehicle based at least in part on the predicted trajectory of the agent; And drive the unmanned vehicle according to the planned action. 20. Unmanned vehicle according to claim 19, containing an unmanned vehicle.