CN118409532A - End-to-end automatic driving method, device, equipment and storage medium - Google Patents

Abstract

The application provides an end-to-end automatic driving method, device, equipment and storage medium, wherein the method comprises the following steps: acquiring sensor data of a self-vehicle at the current moment; processing the sensor data at the current moment to obtain sparse sensor feature vectors; performing interactive processing on the reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector; performing interactive processing on the sparse query quantity of the reference map and the sparse sensor feature vector to obtain a sparse query vector of the map; performing obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and performing map construction according to the map sparse query vector to obtain an online map; and automatically driving the vehicle based on the obstacle detection result and the online map. Therefore, all information is characterized as sparse query vectors, all subtasks of automatic driving are connected in series based on the sparse query vectors, and efficient and high-accuracy end-to-end automatic driving is achieved.

Description

End-to-end autonomous driving method, device, equipment and storage medium

Technical Field

The present application relates to the field of image processing technology, and in particular to an end-to-end autonomous driving method, apparatus, device and storage medium.

Background technique

Autonomous driving systems integrate multiple tasks such as detection, tracking, online mapping, motion prediction, and self-vehicle planning/control. Mainstream autonomous driving systems decouple each task into a separate model and optimize it independently. Manual post-processing steps are added between each model to handle redundant information or difficult cases, which makes the entire process very complicated. In addition, the raw sensor data cannot be used for downstream tasks, and errors gradually accumulate in the system information transmission, leading to potential safety issues.

To solve the above problems, the end-to-end autonomous driving system takes raw sensor data as input and returns planning results in a more concise way. In existing end-to-end autonomous driving methods, the entire driving scene is represented by dense bird's-eye view features, and multi-sensor data and time information are used as raw input to the end-to-end autonomous driving system and play a role in each subtask.

Although such methods can obtain planning results concisely and optimize the model end-to-end, solving the problems of cumulative error and inconsistent optimization targets, their performance in each subtask of autonomous driving lags far behind the corresponding single-task methods. In addition, the fusion of time series information and sensor data is achieved through dense bird's-eye view features, which significantly increases the computational cost and memory usage, making it difficult to deploy the model.

Summary of the invention

In view of the above problems, the embodiments of the present application provide an end-to-end autonomous driving method, apparatus, device and storage medium to overcome the above problems or at least partially solve the above problems.

According to a first aspect of an embodiment of the present application, an end-to-end autonomous driving method is disclosed, the method comprising:

Get the current sensor data of the vehicle;

Processing the current moment sensor data to obtain a sparse sensor feature vector;

Interactively processing a reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least includes: a historical detection sparse query vector determined based on historical sensor data of the vehicle;

Interactively processing the reference map sparse query quantity and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least includes: a historical map sparse query vector determined based on historical sensor data of the vehicle;

Performing obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and performing map construction according to the map sparse query vector to obtain an online map;

The vehicle is automatically driven based on the obstacle detection result and the online map.

Optionally, the method further comprises:

According to the historical position information of the obstacle, a sparse query vector for obstacle reference trajectory prediction is determined;

According to the historical position information of the ego vehicle, a sparse query vector for predicting the ego vehicle reference trajectory is determined;

Interactively processing the obstacle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector;

Interactively processing the ego-vehicle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an ego-vehicle trajectory prediction sparse query vector;

Performing trajectory prediction according to the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain an obstacle prediction trajectory and an ego vehicle prediction trajectory;

The autonomous vehicle is driven automatically based on the obstacle detection result and the online map, including:

The self-vehicle is automatically driven based on the obstacle detection result, the online map, the obstacle prediction trajectory and the self-vehicle prediction trajectory.

Optionally, the method further comprises:

Interactively processing the ego-vehicle trajectory prediction sparse query vector and ego-vehicle navigation command information to obtain an ego-vehicle trajectory prediction sparse query vector containing navigation information;

Utilizing the ego vehicle trajectory prediction sparse query vector containing navigation information, optimizing the ego vehicle prediction trajectory under multiple constraints to obtain an ego vehicle path planning result;

The autonomous vehicle is driven automatically based on the obstacle detection result and the online map, including:

The self-vehicle is automatically driven based on the self-vehicle path planning result, the obstacle detection result, the online map, the obstacle prediction trajectory and the self-vehicle prediction trajectory.

Optionally, when the multiple constraints include dynamic constraints, the ego vehicle trajectory prediction sparse query vector containing navigation information is used to optimize the ego vehicle predicted trajectory under multiple constraints to obtain an ego vehicle path planning result, including:

Using the ego vehicle trajectory containing navigation information to predict a sparse query vector, regressing the motion information of the ego vehicle, the motion information including the speed and acceleration of the ego vehicle;

The motion information is used to optimize the predicted trajectory of the ego vehicle to obtain a ego vehicle path planning result.

Optionally, when the multiple constraints include a safety distance constraint, the ego vehicle trajectory prediction sparse query vector containing navigation information is used to optimize the ego vehicle predicted trajectory under multiple constraints to obtain an ego vehicle path planning result, including:

Predicting a relative position relationship between the obstacle and the ego vehicle based on the ego vehicle trajectory prediction sparse query vector containing navigation information and the obstacle trajectory prediction sparse query vector;

The relative position relationship is used to perform distance constraint optimization processing on the predicted trajectory of the ego vehicle to obtain a ego vehicle path planning result.

Optionally, determining the obstacle reference trajectory prediction sparse query vector according to the obstacle historical position information includes:

Determine the detection frame whose confidence score in the obstacle detection result exceeds the score threshold as the tracking target detection frame;

Determine the sparse query vector corresponding to the tracking target detection box as the tracking sparse query vector;

Encoding the obstacle historical position information corresponding to the tracking sparse query vector into a historical position sparse query vector;

The historical position sparse query vector and the tracking sparse query vector are interactively processed to obtain an obstacle reference trajectory prediction sparse query vector.

Optionally, the reference detection sparse query vector is obtained by the following steps:

Obtaining a historical detection sparse query vector and a tracking sparse query vector at a previous moment, wherein the historical detection sparse query vector includes: historical detection sparse query vectors at N moments before the current moment, where N is an integer greater than 1;

A reference detection sparse query vector is obtained according to the historical detection sparse query vector and the tracking sparse query vector at the last moment.

Optionally, the current moment sensor data is processed to obtain a sparse sensor feature vector, including:

Performing feature extraction on the current moment sensor data to obtain a sparse initial sensor feature vector;

Calculating the spatial position information of the sparse initial sensor feature vector according to the current sensor data;

The spatial position information and the sparse initial sensor feature vector are fused to obtain a sparse sensor feature vector.

Optionally, end-to-end autonomous driving is achieved through a multi-tasking network;

In the case where the end-to-end autonomous driving includes obstacle detection and map construction, the labels of the training data of the multi-task processing network include: real position information of obstacles and real map information;

In the case where the end-to-end autonomous driving includes obstacle detection, map construction and trajectory prediction, the labels of the training data of the multi-task processing network include: real position information of obstacles, real map information, real trajectory of obstacles and real trajectory of the vehicle;

In the case where the end-to-end autonomous driving includes obstacle detection, map construction, trajectory prediction and path planning, the labels of the training data of the multi-task processing network include: the real position information of the obstacle, the real map information, the real trajectory of the obstacle, the real trajectory of the vehicle, the real motion information of the vehicle, and the real relative position relationship between the obstacle and the vehicle.

According to a second aspect of the embodiments of the present application, an end-to-end autonomous driving device is disclosed, the device comprising:

A data acquisition module is used to obtain the current sensor data of the vehicle;

A data processing module, used for processing the sensor data at the current moment to obtain a sparse sensor feature vector;

A first interaction module is used to interactively process a reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least includes: a historical detection sparse query vector determined based on historical sensor data of the vehicle;

A second interaction module is used to interactively process the reference map sparse query quantity and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least includes: a historical map sparse query vector determined based on historical sensor data of the vehicle;

A detection construction module, configured to perform obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and to perform map construction according to the map sparse query vector to obtain an online map;

The automatic driving module is used to automatically drive the vehicle based on the obstacle detection result and the online map.

According to a third aspect of an embodiment of the present application, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the end-to-end autonomous driving method described in the first aspect of the embodiment of the present application are implemented.

The fourth aspect of an embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the end-to-end autonomous driving method described in the first aspect of the embodiment of the present application are implemented.

The embodiments of the present application include the following advantages:

In an embodiment of the present application, all information in the spatial and temporal dimensions of the entire autonomous driving scene is represented by a sparse query vector without using any dense bird's-eye view features. The current moment sensor data of the vehicle is obtained, and the current moment sensor data of the vehicle is processed into a sparse sensor feature vector; and the reference detection sparse query vector and the sparse sensor feature vector are interactively processed to obtain a detection sparse query vector; the reference map sparse query vector and the sparse sensor feature vector are interactively processed to obtain a map sparse query vector; obstacle detection is performed according to the detection sparse query vector to obtain an obstacle detection result, and map construction is performed according to the map sparse query vector to obtain an online map, so that the vehicle is autonomously driven based on the obstacle detection result and the online map.

Since the reference detection sparse query vector includes a historical detection sparse query vector determined based on the historical sensor data of the vehicle, and the reference map sparse query vector includes a historical map sparse query vector determined based on the historical sensor data of the vehicle, the detection sparse query vector obtained based on the reference detection sparse query and the sparse sensor feature vector introduces historical time series information and current moment sensor data, and the map detection sparse query vector obtained based on the reference map sparse query vector and the sparse sensor feature vector introduces historical time series information and current moment sensor data. Thanks to the introduction of historical time series information and current moment sensor data, each task in end-to-end autonomous driving (such as obstacle detection and map construction) has better performance. In addition, through the sparse query vector, it is possible to more effectively utilize long-time historical information, expand to more modes and tasks, and reduce computing costs and memory usage. In this way, all information in different modes, different tasks, space and time is represented as a sparse query vector, and based on this, each subtask of autonomous driving is connected in series to achieve efficient and high-accuracy end-to-end autonomous driving.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

FIG1 is a flowchart of the steps of an end-to-end autonomous driving method provided by an embodiment of the present application;

FIG2 is a schematic diagram of the structure of a sparse sensing network provided in an embodiment of the present application;

FIG3 is a schematic diagram of the structure of a motion planning network provided in an embodiment of the present application;

FIG4 is a flowchart of another end-to-end autonomous driving method provided by an embodiment of the present application;

FIG5 is a schematic diagram of the structure of a multi-task processing network provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of an end-to-end autonomous driving device provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

Detailed ways

In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

In the related art, the entire driving scene is represented by dense bird's-eye view features, and the fusion of time series information and sensor data is achieved through dense bird's-eye view features, which significantly increases the computing cost and memory usage, making it difficult to implement model deployment, and the performance of each subtask in autonomous driving lags far behind the corresponding single task processing method. In order to overcome the limitations of the above problems, the applicant proposes the following technical concept: It is proposed to use a purely sparse method without relying on any dense bird's-eye view features for map construction and obstacle detection. All information in different modalities, different tasks, space and time are represented as sparse query vectors, and based on this, the various subtasks of autonomous driving are connected in series to achieve efficient and high-accuracy end-to-end autonomous driving.

Based on the above technical concept, the embodiment of the present application provides an end-to-end autonomous driving method, as shown in Figure 1, which is a step flow chart of an end-to-end autonomous driving method provided by the embodiment of the present application. As shown in Figure 1, the method may include steps S101 to S104:

Step S101: Acquire the current sensor data of the vehicle.

In an embodiment of the present application, the sensor data at the current moment is the environmental data collected at the current moment that reflects the entire autonomous driving scene of the vehicle, and the sensor data at the current moment includes different types of sensor data (i.e., multimodal data). In one embodiment, the sensor data at the current moment includes point cloud data collected by the radar and multi-view images collected by the on-board camera. Among them, the viewing angle and number of multi-view images will vary depending on the arrangement of the on-board cameras. The multi-view images may include the front view image, the left front view image, the right front view image, the rear view image, the left rear view image, and the right rear view image of the vehicle. In addition, the multi-view image can be a video frame image at the current moment extracted from the video, or it can be an image at the current moment directly collected by the on-board camera.

Step S102: Process the sensor data at the current moment to obtain a sparse sensor feature vector.

In an embodiment of the present application, the sparse sensor feature vector is a more discrete and sparse three-dimensional feature representation compared to the bird's-eye view feature. The sparse sensor feature vector represents the target object in the driving scene and does not contain redundant information in the driving scene. For example, the sparse sensor feature vector represents obstacles in the driving scene, or represents map elements in the driving scene. Since the sparse sensor feature vector represents the target object in the driving scene, the sparse sensor feature vector contains the semantic information and position information of the target object. In the specific implementation, different types of sensor data (for example, image data, point cloud data) are processed using different feature extraction networks to obtain sparse sensor feature vectors; support for multimodal data can be expanded by increasing the number of encoders. In addition, when the sensor data at the current moment is feature encoded, the position information of the data in three-dimensional space is also encoded, and then the features (semantic information) and positions are unified into the sparse sensor feature vector.

In an optional embodiment, the sensor data at the current moment is processed to obtain a sparse sensor feature vector, including: extracting features from the sensor data at the current moment to obtain a sparse initial sensor feature vector; calculating spatial position information of the sparse initial sensor feature vector based on the sensor data at the current moment; and fusing the spatial position information with the sparse initial sensor feature vector to obtain a sparse sensor feature vector.

In an embodiment of the present application, a sparse initial sensor feature vector refers to a feature vector containing semantic information of a target object. In order to accurately characterize the target object in a driving scene, after obtaining the sparse initial sensor feature vector, each sparse initial sensor feature vector is position-encoded to obtain a sparse sensor feature vector containing semantic information and position information.

The sensor data at the current moment includes multiple different types of sensor data. For different types of sensor data, different feature extraction networks are sampled to extract features of different types of sensor data, and the features of different types of sensor data are flattened and spliced together to obtain sparse initial sensor feature vectors; for each sparse initial sensor feature vector, the corresponding spatial position information is calculated, and the spatial position information and the sparse initial sensor feature vector are fused to obtain a sparse sensor feature vector.

For example, different types of sensor data include image data and cloud data. Image data is represented as I∈R ^{N ×H×W×3} , where N is the number of input image viewpoints; point cloud data is represented as Where _Np is the number of points in the point cloud data, and _Cp is the dimension corresponding to each point cloud data. Use the feature extraction network to extract features from the image data and point cloud data respectively, obtain image data features and point cloud data features, and flatten and splice the image data features and point cloud data features together to obtain a sparse initial sensor feature vector Where _Nf is the number of sparse initial sensor feature vectors, and C is the dimension of the sparse initial sensor feature vectors. Afterwards, for each sparse initial sensor feature vector, the corresponding spatial position information is calculated, and the spatial position information is fused with the sparse initial sensor feature vector _Ft to obtain a sparse sensor feature vector.

Step S103: interactively processing the reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least includes: a historical detection sparse query vector determined based on historical sensor data of the vehicle.

In the embodiment of the present application, the detection sparse query vector refers to the sparse query vector used for detection and tracking at the current moment, and each detection sparse query vector represents an obstacle object, for example, each detection sparse query vector corresponds to a vehicle in the driving scene. The historical detection sparse query vector refers to the historical detection sparse query vector of N moments before the current moment, and the historical detection sparse query vector will be updated as the autonomous driving progresses.

Specifically, interactively processing the reference detection sparse query vector and the sparse sensor feature vector means: interactively processing the reference detection sparse query vector and the sparse sensor feature vector through a self-attention mechanism and a cross-attention mechanism to obtain a detection sparse query vector. Since the reference detection sparse query vector includes a historical detection sparse query vector, historical timing information and current moment sensor data are introduced into the detection sparse query vector through interactive processing, and obstacle detection can be well implemented based on the detection sparse query vector. In addition, after obtaining the detection sparse query vector, the detection sparse query vector is saved for autonomous driving at the next moment.

In an optional embodiment, the reference detection sparse query vector is obtained by the following steps: obtaining a historical detection sparse query vector and a tracking sparse query vector at a previous moment, the historical detection sparse query vector comprising: historical detection sparse query vectors of N moments before the current moment, where N is an integer greater than 1; obtaining a reference detection sparse query vector based on the historical detection sparse query vector and the tracking sparse query vector at the previous moment.

In the embodiment of the present application, the tracking sparse query vector at the last moment is determined according to the obstacle detection result at the last moment, and is used to track obstacles with high credibility. In the specific implementation, the historical detection sparse query vector and the tracking sparse query vector at the last moment are stored in an end-to-end multi-task memory library, and thus the historical detection sparse query vector and the tracking sparse query vector at the last moment are obtained from the end-to-end multi-task memory library.

Specifically, according to the historical detection sparse query vector and the tracking sparse query vector at the last moment, it includes: concatenating the initialization detection sparse query vector, the historical detection sparse query vector and the tracking sparse query vector at the last moment to obtain a reference detection sparse query vector. The initialization detection sparse query vector is a learnable random initialization vector.

Step S104: interactively processing the reference map sparse query quantity and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least includes: a historical map sparse query vector determined based on historical sensor data of the vehicle.

In the embodiment of the present application, the map sparse query vector refers to the sparse query vector used for online mapping at the current moment, and each map sparse query vector represents a map element, for example, each map sparse query vector corresponds to a line segment in the map. The historical map sparse query vector refers to the historical map sparse query vector of N moments before the current moment, and the historical map sparse query vector will be updated as the autonomous driving progresses.

Specifically, interactively processing the reference map sparse query volume and the sparse sensor feature vector means: interactively processing the reference map sparse query volume and the sparse sensor feature vector through the self-attention mechanism and the cross-attention mechanism to obtain the map sparse query vector. Since the reference map sparse query volume includes the historical map sparse query vector, the historical time series information and the current moment sensor data are introduced into the map sparse query vector through interactive processing, and the map construction can be well realized based on the map sparse query vector. In addition, after obtaining the map sparse query vector, the map sparse query vector is saved for autonomous driving at the next moment.

In an optional embodiment, the reference map sparse query vector is obtained by the following steps: obtaining a historical map sparse query vector, the historical map sparse query vector comprising: historical map sparse query vectors of N moments before the current moment, where N is an integer greater than 1; concatenating the historical map sparse query vector with an initialization map sparse query vector to obtain a historical map sparse query vector, wherein the initialization map sparse query vector is a learnable random initialization vector.

Step S105: performing obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and performing map construction according to the map sparse query vector to obtain an online map.

In an embodiment of the present application, each detection sparse query vector represents an obstacle object, and the detection sparse query vector is processed by the detection network to obtain an obstacle detection result. The obstacle can be a vehicle in the driving scene or a pedestrian in the driving scene. Among them, the obstacle detection result is represented by a detection box and a detection box confidence score. The higher the detection box confidence score, the more reliable the corresponding obstacle detection result. Each map sparse query vector represents a map element, and the map sparse query vector is processed by the map construction network to obtain an online map. The online map includes information such as lane line information and road signs.

Step S106: Automatically drive the vehicle based on the obstacle detection result and the online map.

In the embodiment of the present application, the obstacle detection results and online maps are obtained by processing the sensor data at the current moment, and multiple subtasks of autonomous driving are realized based on the sensor data at the current moment. In this way, all information in different modes, different tasks, space and time are represented as sparse query vectors, and based on this, the various subtasks of autonomous driving are connected in series to achieve efficient and high-accuracy end-to-end autonomous driving.

In the embodiment of the present application, end-to-end autonomous driving is achieved through a multi-task processing network; when the end-to-end autonomous driving includes obstacle detection and map construction, the labels of the training data of the multi-task processing network include: the real position information of the obstacle and the real map information. By carrying the labels of the real position information and the real map information of the obstacle in the training data, end-to-end optimization of the obstacle detection task and end-to-end optimization of the map construction task are achieved. In this way, the obstacle detection task and the map construction task are connected in series to achieve efficient and highly accurate end-to-end autonomous driving.

Specifically, the multi-task processing network includes a feature extraction network and a sparse perception network. Among them, the feature extraction network is used to process the sensor data at the current moment to obtain a sparse sensor feature vector. The sparse perception network is used to perform obstacle detection tasks and map construction tasks. By way of example, Figure 2 is a structural schematic diagram of a sparse perception network provided in an embodiment of the present application, and the sparse perception network includes two time series encoders, an obstacle detection module (i.e., an obstacle output head) and a map construction module (i.e., a map construction output head) and a data update module.

The process of executing obstacle detection tasks and map construction tasks is as follows: obtaining historical detection sparse query vectors, historical map sparse query vectors, and tracking sparse query vectors at the previous moment from an end-to-end multi-task memory library; and obtaining a reference detection sparse query vector based on the historical detection sparse query vector, the tracking sparse query vector at the previous moment, and the initialization detection sparse query vector; obtaining a reference map sparse query vector based on the historical map sparse query vector and the initialization map sparse query vector; using a temporal encoder to interactively process the reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, and using an obstacle detection module to perform obstacle detection on the detection sparse query vector to obtain an obstacle detection result; using another temporal encoder to interactively process the reference map sparse query vector and the sparse sensor feature vector to obtain a map sparse query vector, and using a map construction module to construct a map on the map sparse query vector to obtain an online map. Finally, the data update module updates the historical map sparse query vector, historical detection sparse query vector, and tracking sparse query vector in the end-to-end multi-task memory library for end-to-end autonomous driving at the next moment.

In an optional embodiment, in order to obtain the obstacle prediction trajectory and the vehicle prediction trajectory, the vehicle is automatically driven based on the obstacle prediction trajectory and the vehicle prediction trajectory on the basis of the obstacle detection result and the online map. After step S105, the following steps S107 to S1012 are also included:

Step S107: Determine the obstacle reference trajectory prediction sparse query vector according to the obstacle historical position information.

In the embodiment of the present application, the obstacle historical position information refers to the obstacle historical position information used for tracking, and the obstacle historical position information is represented by the coordinate position in the map; the obstacle reference trajectory prediction sparse query vector refers to the initialization vector used for obstacle trajectory prediction.

Specifically, according to the historical position information of the obstacle, a sparse query vector for predicting the obstacle reference trajectory is determined, including: determining a detection frame in the obstacle detection result whose detection frame confidence score exceeds a score threshold as a tracking target detection frame; determining a sparse query vector corresponding to the tracking target detection frame as a tracking sparse query vector; encoding the obstacle historical position information corresponding to the tracking sparse query vector into a historical position sparse query vector; and interactively processing the historical position sparse query vector and the tracking sparse query vector to obtain a sparse query vector for predicting the obstacle reference trajectory.

In the embodiment of the present application, the score threshold is flexibly set according to the actual situation. The obstacle detection result is characterized by the detection frame and the detection frame confidence score. The higher the detection frame confidence score, the more reliable the corresponding obstacle detection result. Therefore, the detection frame whose detection frame confidence score exceeds the score threshold in the obstacle detection result is determined as the tracking target detection frame, and the sparse query vector corresponding to the tracking target detection frame is determined as the tracking sparse query vector. Only the obstacle trajectory corresponding to the obstacle detection result with high confidence is predicted.

Compared with the obstacle historical position information, the historical position sparse query vector is a high-dimensional historical position feature. By encoding the obstacle historical position information into a historical position sparse query vector, the obstacle historical position information and the tracking sparse query vector can be interactively processed based on the multi-head attention mechanism. In the specific implementation, a learnable embedding vector is introduced to determine the obstacle reference trajectory prediction sparse query vector, and the interactive processing results of the obstacle historical position information and the tracking sparse query vector are spliced with the learnable embedding vector to obtain the obstacle reference trajectory prediction sparse query vector.

Step S108: Determine a sparse query vector for predicting a reference trajectory of the vehicle according to the historical position information of the vehicle.

In the embodiment of the present application, the historical position information of the self-vehicle refers to the position of the self-vehicle in the driving scene, and the historical position information of the self-vehicle can be determined by the positioning device of the self-vehicle. The self-vehicle reference trajectory prediction sparse query vector refers to the initialization vector used for self-vehicle trajectory prediction. Similarly, the self-vehicle reference trajectory prediction sparse query vector is determined by introducing a learnable embedding vector; the self-vehicle reference trajectory prediction sparse query vector is determined according to the self-vehicle historical position information, including: encoding the self-vehicle historical position information to obtain the encoded self-vehicle historical position information; splicing the encoded self-vehicle historical position information with the self-vehicle learnable embedding vector to obtain the self-vehicle reference trajectory prediction sparse query vector.

Step S109: interactively processing the obstacle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector.

In an embodiment of the present application, the obstacle trajectory prediction sparse query vector includes obstacle position information and map information for predicting the obstacle trajectory; different obstacles correspond to different obstacle trajectory prediction sparse query vectors. The detection sparse query vector refers to the detection sparse query vector obtained in the above step S103, and the map sparse query vector refers to the map sparse query vector obtained in the above step S104. The obstacle reference trajectory prediction sparse query vector refers to the initialization vector used for obstacle trajectory prediction. Through the multi-head attention mechanism, the obstacle reference trajectory prediction sparse query vector is interactively processed with the detection sparse query vector and the map sparse query vector to interactively update the obstacle reference trajectory prediction sparse query vector to obtain the obstacle trajectory prediction sparse query vector.

Step S1010: interactively processing the ego-vehicle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an ego-vehicle trajectory prediction sparse query vector.

In the embodiment of the present application, the sparse query vector for predicting the trajectory of the vehicle includes the position information of the vehicle, the position information of the surrounding obstacles and the map information, so as to predict the trajectory of the vehicle. The detection sparse query vector refers to the detection sparse query vector obtained in the above step S103, and the map sparse query vector refers to the map sparse query vector obtained in the above step S104. Through the multi-head attention mechanism, the sparse query vector for predicting the reference trajectory of the vehicle is interactively updated with the detection sparse query vector and the map sparse query vector to obtain the sparse query vector for predicting the trajectory of the vehicle.

Step S1011: performing trajectory prediction according to the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain an obstacle prediction trajectory and an ego vehicle prediction trajectory.

In the embodiment of the present application, the obstacle prediction trajectory refers to the predicted driving trajectory of the obstacle in the future, and the vehicle prediction trajectory refers to the predicted driving trajectory of the vehicle in the future. By predicting the obstacle prediction trajectory and the vehicle prediction trajectory, path planning for the autonomous driving of the vehicle can be achieved. Specifically, the obstacle trajectory prediction sparse query vector and the vehicle trajectory prediction sparse query vector are processed by the trajectory prediction network to obtain the obstacle prediction trajectory and the vehicle prediction trajectory.

Step S1012: Automatically driving the vehicle based on the obstacle detection result, the online map, the obstacle prediction trajectory and the vehicle prediction trajectory.

In the embodiment of the present application, the obstacle detection results, online maps, obstacle prediction trajectories and vehicle prediction trajectories are obtained by processing the sensor data at the current moment, thereby realizing multiple subtasks of autonomous driving based on the sensor data at the current moment. In this way, all information in different modes, tasks, space and time is represented as sparse query vectors, and based on this, the various subtasks of autonomous driving are connected in series to achieve efficient and high-accuracy end-to-end autonomous driving.

In an embodiment of the present application, end-to-end autonomous driving is achieved through a multi-task processing network; in the case where the end-to-end autonomous driving includes obstacle detection, map construction, and trajectory prediction, the labels of the training data of the multi-task processing network include: the real position information of the obstacle, the real map information, the real trajectory of the obstacle, and the real trajectory of the vehicle. By carrying the labels of the real position information, real map information, real trajectory of the obstacle, and the real trajectory of the vehicle in the training data, end-to-end optimization of the obstacle detection task, end-to-end optimization of the map construction task, and end-to-end optimization of the trajectory prediction task are achieved. In this way, the obstacle detection task, the map construction task, and the trajectory prediction task are connected in series to achieve efficient and highly accurate end-to-end autonomous driving.

In an optional embodiment, in order to realize the trajectory planning of the self-vehicle, the self-vehicle is automatically driven based on the self-vehicle path planning result on the basis of the obstacle detection result, the online map, the obstacle prediction trajectory and the self-vehicle prediction trajectory. After step S1011, steps S1013 to S1015 are also included:

Step S1013: interactively process the vehicle trajectory prediction sparse query vector and the vehicle navigation command information to obtain the vehicle trajectory prediction sparse query vector containing navigation information.

Step S1014: using the ego vehicle trajectory prediction sparse query vector containing navigation information, optimizing the ego vehicle predicted trajectory under multiple constraints to obtain an ego vehicle path planning result.

Step S1015: Automatically drive the ego vehicle based on the ego vehicle path planning result, the obstacle detection result, the online map, the obstacle prediction trajectory and the ego vehicle prediction trajectory.

In the embodiment of the present application, the self-vehicle navigation command information refers to the navigation command for controlling the vehicle, such as navigation commands such as turning left, turning right, accelerating and decelerating. The various constraints include dynamic constraints and safe distance constraints, where the dynamic constraints include speed constraints and acceleration constraints; the safe distance constraint means that the relative position relationship between the self-vehicle and the predicted obstacle must meet the safe distance. In order to realize the trajectory planning of the self-vehicle, the self-vehicle navigation command information is introduced to optimize the predicted trajectory of the self-vehicle, thereby obtaining the self-vehicle path planning result.

Since the obstacle detection results, online maps, obstacle prediction trajectories, vehicle prediction trajectories, and vehicle path planning results are all obtained through processing the current sensor data, multiple subtasks of autonomous driving can be realized based on the current sensor data. In this way, all information in different modes, tasks, space, and time is represented as sparse query vectors, and based on this, the various subtasks of autonomous driving are connected in series to achieve efficient and high-accuracy end-to-end autonomous driving.

The following describes the optimization processing of the predicted trajectory of the ego-vehicle under dynamic constraints and safety distance constraints respectively.

(1) When the multiple constraints include dynamic constraints, the ego-vehicle trajectory containing navigation information is used to predict a sparse query vector, and the predicted trajectory of the ego-vehicle is optimized under multiple constraints to obtain an ego-vehicle path planning result, including: using the ego-vehicle trajectory containing navigation information to predict the sparse query vector, regressing the motion information of the ego-vehicle, the motion information including the speed and acceleration of the ego-vehicle; and using the motion information to optimize the predicted trajectory of the ego-vehicle to obtain an ego-vehicle path planning result.

Among them, regressing the motion information of the ego vehicle is equivalent to predicting the motion information (speed and acceleration) of the ego vehicle under the ego vehicle predicted trajectory, so as to optimize the ego vehicle predicted trajectory with the motion information. For example, if the ego vehicle speed exceeds the safe driving speed, the ego vehicle predicted trajectory is adjusted so that the adjusted trajectory meets the safe driving speed constraint. For another example, if the ego vehicle acceleration exceeds the safe driving acceleration (i.e., the acceleration is too fast), the ego vehicle predicted trajectory is adjusted so that the adjusted trajectory meets the safe driving acceleration constraint.

(2) When the multiple constraints include a safety distance constraint, the ego vehicle trajectory prediction sparse query vector containing navigation information is used to optimize the ego vehicle predicted trajectory under multiple constraints to obtain an ego vehicle path planning result, including: predicting a relative position relationship between the obstacle and the ego vehicle based on the ego vehicle trajectory prediction sparse query vector containing navigation information and the obstacle trajectory prediction sparse query vector; and performing distance constraint optimization processing on the ego vehicle predicted trajectory using the relative position relationship to obtain an ego vehicle path planning result.

Specifically, a certain safety distance needs to be met between the ego vehicle and the obstacle. Using the relative position relationship to perform distance constraint optimization processing on the ego vehicle predicted trajectory means adjusting the ego vehicle predicted trajectory so that the adjusted trajectory meets the safety distance constraint.

In an embodiment of the present application, end-to-end autonomous driving is achieved through a multi-task processing network; when the end-to-end autonomous driving includes obstacle detection, map construction, trajectory prediction and path planning, the labels of the training data of the multi-task processing network include: the real position information of the obstacle, the real map information, the real trajectory of the obstacle, the real trajectory of the vehicle, the real motion information of the vehicle, and the real relative position relationship between the obstacle and the vehicle.

By carrying the real position information of obstacles, real map information, real trajectory of obstacles, real trajectory of the vehicle, real motion information of the vehicle, and labels of the real relative position relationship between obstacles and the vehicle in the training data, end-to-end optimization of obstacle detection tasks, end-to-end optimization of map construction tasks, end-to-end optimization of trajectory prediction tasks, and end-to-end optimization of path planning are achieved. In this way, the obstacle detection tasks, map construction tasks, trajectory prediction tasks, and path planning tasks are connected in series to achieve efficient and highly accurate end-to-end autonomous driving.

Among them, the multi-task processing network includes a feature extraction network, a sparse perception network and a motion planning network. The motion planning network is used to perform trajectory prediction tasks and path planning tasks. For example, Figure 3 is a structural diagram of a motion planning network provided in an embodiment of the present application, and the motion planning network includes a trajectory prediction module and a path planning module.

Specifically, the process of executing the trajectory prediction task and the path planning task is as follows: according to the historical position information of the obstacle, determine the obstacle reference trajectory prediction sparse query vector; and according to the historical position information of the vehicle, determine the ego vehicle reference trajectory prediction sparse query vector; then, obtain the historical obstacle reference trajectory prediction sparse query vector, the historical ego vehicle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector from the end-to-end multi-task memory library, and input the obstacle reference trajectory prediction sparse query vector and the ego vehicle reference trajectory prediction sparse query vector into the interactive prediction module for interactive processing and trajectory prediction, and obtain the ego vehicle trajectory prediction sparse query vector. Among them, the interactive processing includes: the obstacle reference trajectory prediction sparse query vector interacts with the corresponding historical obstacle reference trajectory prediction sparse query vector, the obstacle reference trajectory prediction sparse query vector interacts with the detection sparse query vector and the map sparse query vector; the ego vehicle reference trajectory prediction sparse query vector interacts with the corresponding historical ego vehicle reference trajectory prediction sparse query vector, and the ego vehicle reference trajectory prediction sparse query vector interacts with the detection sparse query vector and the map sparse query vector.

Next, the path planning module interactively processes the ego-vehicle trajectory prediction sparse query vector and the ego-vehicle navigation command information to obtain the ego-vehicle trajectory prediction sparse query vector containing navigation information; and uses the ego-vehicle trajectory prediction sparse query vector containing navigation information to optimize the ego-vehicle prediction trajectory under multiple constraints to obtain the ego-vehicle path planning result.

The present application is described below in conjunction with a specific implementation method.

FIG. 4 is a flowchart of another end-to-end autonomous driving method provided in an embodiment of the present application. As shown in FIG. 4 , the method includes steps S401 to S4013:

Step S401: Acquire the current sensor data of the vehicle.

Step S402: Process the sensor data at the current moment to obtain a sparse sensor feature vector.

Step S403: interactively process the reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least includes: a historical detection sparse query vector determined based on historical sensor data of the vehicle.

Step S404: interactively processing the reference map sparse query quantity and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least includes: a historical map sparse query vector determined based on historical sensor data of the vehicle.

Step S405: performing obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and performing map construction according to the map sparse query vector to obtain an online map.

Step S406: Determine the obstacle reference trajectory prediction sparse query vector according to the obstacle historical position information.

Step S407: Determine a sparse query vector for predicting a reference trajectory of the vehicle according to the historical position information of the vehicle.

Step S408: interactively process the obstacle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector.

Step S409: interactively processing the ego-vehicle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an ego-vehicle trajectory prediction sparse query vector.

Step S4010: performing trajectory prediction according to the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain an obstacle prediction trajectory and an ego vehicle prediction trajectory.

Step S4011: interactively process the vehicle trajectory prediction sparse query vector and the vehicle navigation command information to obtain the vehicle trajectory prediction sparse query vector containing navigation information.

Step S4012: using the ego vehicle trajectory prediction sparse query vector containing navigation information, optimizing the ego vehicle predicted trajectory under multiple constraints to obtain an ego vehicle path planning result.

Step S4013: Automatically drive the ego vehicle based on the ego vehicle path planning result, the obstacle detection result, the online map, the obstacle prediction trajectory and the ego vehicle prediction trajectory.

In the embodiment of the present application, the obstacle detection task, the map construction task, the trajectory prediction task and the path planning task are connected in series, and efficient and highly accurate end-to-end autonomous driving can be achieved based on the current sensor data of the self-vehicle. Specifically, obstacle detection and map construction are achieved by executing steps S401 to S405; trajectory prediction is achieved by executing steps S406 to S4010 to obtain the obstacle prediction trajectory and the self-vehicle prediction trajectory; in order to achieve path planning, steps S4011 to S4012 are executed to obtain the self-vehicle path planning result that meets multiple constraints (such as dynamic constraints and safety distance constraints), and then step S4013 is executed to perform autonomous driving on the self-vehicle.

In summary, the embodiment of the present application represents all the information in the spatial and temporal dimensions of the entire autonomous driving scene by a sparse query vector, without using any dense bird's-eye view features, effectively utilizing long-time historical information, expanding to more modes and tasks, and reducing computing costs and memory usage. Thanks to the introduction of historical time series information and current sensor data, each task in end-to-end autonomous driving has better performance. In this way, the problems of lagging single-task performance, high computing costs, large memory usage, and difficult model deployment of the end-to-end autonomous driving model are solved.

In an embodiment of the present application, end-to-end autonomous driving is achieved through a multi-task processing network. For example, FIG5 is a schematic diagram of a structure of a multi-task processing network provided in an embodiment of the present application, and the multi-task processing network includes: a feature extraction network, a sparse perception network, and a motion planning network.

Specifically, the current sensor data of the vehicle (including image data and point cloud data) is input into the feature extraction network. After the feature extraction network extracts features and positions the sensor data at the current moment, it inputs a sparse sensor feature vector.

The sparse perception network obtains the historical detection sparse query vector, the historical map sparse query vector, and the tracking sparse query vector at the previous moment from the end-to-end multi-task memory library, and obtains the reference detection sparse query vector based on the historical detection sparse query vector and the tracking sparse query vector at the previous moment, and obtains the reference map sparse query volume based on the historical map sparse query vector; then, the reference detection sparse query vector and the sparse sensor feature vector are interactively processed to obtain the detection sparse query vector; the reference map sparse query volume and the sparse sensor feature vector are interactively processed to obtain the map sparse query vector; and obstacle detection is performed based on the detection sparse query vector to obtain the obstacle detection result, and the map is constructed based on the map sparse query vector to obtain the online map.

The motion planning network takes the detection sparse query vector, the map sparse query vector and the tracking sparse query vector as input, and determines the obstacle reference trajectory prediction sparse query vector according to the obstacle historical position information; and determines the ego vehicle reference trajectory prediction sparse query vector according to the ego vehicle historical position information. Then the obstacle reference trajectory prediction sparse query vector is interactively processed with the detection sparse query vector and the map sparse query vector to obtain the obstacle trajectory prediction sparse query vector; the ego vehicle reference trajectory prediction sparse query vector is interactively processed with the detection sparse query vector and the map sparse query vector to obtain the ego vehicle trajectory prediction sparse query vector; then, trajectory prediction is performed based on the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain the obstacle prediction trajectory and the ego vehicle prediction trajectory. Finally, based on the ego vehicle navigation command information, the ego vehicle prediction trajectory is optimized to obtain the ego vehicle path planning result that meets multiple constraints.

The embodiment of the present application further provides an end-to-end autonomous driving device, as shown in FIG6 , which is a schematic diagram of the structure of an end-to-end autonomous driving device provided by the embodiment of the present application, the device comprising:

The data acquisition module 610 is used to acquire the current sensor data of the vehicle;

A data processing module 620 is used to process the sensor data at the current moment to obtain a sparse sensor feature vector;

A first interaction module 630 is configured to interactively process a reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least includes: a historical detection sparse query vector determined based on historical sensor data of the vehicle;

The second interaction module 640 is used to interactively process the reference map sparse query quantity and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least includes: a historical map sparse query vector determined based on historical sensor data of the vehicle;

A detection construction module 650, configured to perform obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and to perform map construction according to the map sparse query vector to obtain an online map;

The autonomous driving module 660 is used to perform autonomous driving of the vehicle based on the obstacle detection result and the online map.

In an optional embodiment, the device further includes:

A first determination module is used to determine the obstacle reference trajectory prediction sparse query vector according to the obstacle historical position information;

The second determination module is used to determine the sparse query vector for predicting the reference trajectory of the ego vehicle according to the historical position information of the ego vehicle;

A third interaction module is used to interactively process the obstacle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector;

A fourth interaction module, configured to interactively process the ego-vehicle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an ego-vehicle trajectory prediction sparse query vector;

The trajectory prediction module is used to perform trajectory prediction according to the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain an obstacle prediction trajectory and an ego vehicle prediction trajectory.

In an optional embodiment, the device further includes:

A fifth interactive module, configured to interactively process the ego-vehicle trajectory prediction sparse query vector and the ego-vehicle navigation command information to obtain the ego-vehicle trajectory prediction sparse query vector containing navigation information;

The trajectory optimization module is used to use the ego vehicle trajectory containing navigation information to predict the sparse query vector, optimize the ego vehicle predicted trajectory under multiple constraints, and obtain the ego vehicle path planning result.

In an optional embodiment, when the multiple constraints include dynamic constraints, the trajectory optimization module includes:

A regression module, configured to use the ego vehicle trajectory containing navigation information to predict a sparse query vector and regress the ego vehicle's motion information, wherein the motion information includes the ego vehicle's speed and acceleration;

The first optimization submodule is used to optimize the predicted trajectory of the ego vehicle using the motion information to obtain a path planning result of the ego vehicle.

In an optional embodiment, when the multiple constraints include a safety distance constraint, the trajectory optimization module includes:

A position prediction module, used for predicting a relative position relationship between the obstacle and the ego vehicle based on the ego vehicle trajectory prediction sparse query vector containing navigation information and the obstacle trajectory prediction sparse query vector;

The second optimization submodule is used to perform distance constraint optimization processing on the predicted trajectory of the ego vehicle by utilizing the relative position relationship to obtain a path planning result of the ego vehicle.

In an optional embodiment, the first determining module includes:

A first determination submodule is used to determine the detection frame whose confidence score of the detection frame in the obstacle detection result exceeds the score threshold as the tracking target detection frame;

A second determination submodule is used to determine the sparse query vector corresponding to the tracking target detection box as a tracking sparse query vector;

A position information encoding module, used for encoding the obstacle historical position information corresponding to the tracking sparse query vector into a historical position sparse query vector;

The sixth interactive module is used to interactively process the historical position sparse query vector and the tracking sparse query vector to obtain an obstacle reference trajectory prediction sparse query vector.

In an optional embodiment, the device further includes:

A first acquisition module is used to acquire a historical detection sparse query vector and a tracking sparse query vector at a previous moment, wherein the historical detection sparse query vector includes: a historical detection sparse query vector at N moments before the current moment, where N is an integer greater than 1;

The reference detection vector module is used to obtain a reference detection sparse query vector according to the historical detection sparse query vector and the tracking sparse query vector at the previous moment.

In an optional embodiment, the data processing module includes:

An extraction module, used for performing feature extraction on the sensor data at the current moment to obtain a sparse initial sensor feature vector;

A calculation module, used for calculating the spatial position information of the sparse initial sensor feature vector according to the sensor data at the current moment;

The fusion module is used to fuse the spatial position information with the sparse initial sensor feature vector to obtain a sparse sensor feature vector.

In an optional embodiment, end-to-end autonomous driving is achieved through a multi-tasking network;

In the case where the end-to-end autonomous driving includes obstacle detection and map construction, the labels of the training data of the multi-task processing network include: real position information of obstacles and real map information;

In the case where the end-to-end autonomous driving includes obstacle detection, map construction and trajectory prediction, the labels of the training data of the multi-task processing network include: real position information of obstacles, real map information, real trajectory of obstacles and real trajectory of the vehicle;

In the case where the end-to-end autonomous driving includes obstacle detection, map construction, trajectory prediction and path planning, the labels of the training data of the multi-task processing network include: the real position information of the obstacle, the real map information, the real trajectory of the obstacle, the real trajectory of the vehicle, the real motion information of the vehicle, and the real relative position relationship between the obstacle and the vehicle.

The embodiment of the present application also provides an electronic device, with reference to FIG7, which is a schematic diagram of the structure of an electronic device provided by the embodiment of the present application. As shown in FIG7, the electronic device 700 includes: a memory 710 and a processor 720, the memory 710 and the processor 720 are connected via a bus communication, the memory 710 stores a computer program, and the computer program can be run on the processor 720, thereby implementing the steps of the end-to-end autonomous driving method described in the embodiment of the present application.

An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the end-to-end autonomous driving method described in the embodiment of the present application are implemented.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

The embodiments of the present application are described with reference to the flowcharts and/or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

The above is a detailed introduction to an end-to-end autonomous driving method, device, equipment and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims (12)

1. An end-to-end autonomous driving method, characterized in that the method comprises: Get the current sensor data of the vehicle; Processing the current moment sensor data to obtain a sparse sensor feature vector; Interactively processing a reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least includes: a historical detection sparse query vector determined based on historical sensor data of the vehicle; Interactively processing the reference map sparse query quantity and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least includes: a historical map sparse query vector determined based on historical sensor data of the vehicle; Performing obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and performing map construction according to the map sparse query vector to obtain an online map; The vehicle is automatically driven based on the obstacle detection result and the online map. 2. The method according to claim 1, characterized in that the method further comprises: According to the historical position information of the obstacle, a sparse query vector for obstacle reference trajectory prediction is determined; According to the historical position information of the ego vehicle, a sparse query vector for predicting the ego vehicle reference trajectory is determined; Interactively processing the obstacle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector; Interactively processing the ego-vehicle reference trajectory prediction sparse query vector, the detection sparse query vector and the map sparse query vector to obtain an ego-vehicle trajectory prediction sparse query vector; Performing trajectory prediction according to the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain an obstacle prediction trajectory and an ego vehicle prediction trajectory; The autonomous vehicle is driven automatically based on the obstacle detection result and the online map, including: The self-vehicle is automatically driven based on the obstacle detection result, the online map, the obstacle prediction trajectory and the self-vehicle prediction trajectory. 3. The method according to claim 2, characterized in that the method further comprises: Interactively processing the ego-vehicle trajectory prediction sparse query vector and ego-vehicle navigation command information to obtain an ego-vehicle trajectory prediction sparse query vector containing navigation information; Utilizing the ego vehicle trajectory prediction sparse query vector containing navigation information, optimizing the ego vehicle prediction trajectory under multiple constraints to obtain an ego vehicle path planning result; The autonomous vehicle is driven automatically based on the obstacle detection result and the online map, including: The self-vehicle is automatically driven based on the self-vehicle path planning result, the obstacle detection result, the online map, the obstacle prediction trajectory and the self-vehicle prediction trajectory. 4. The method according to claim 3, characterized in that, when the multiple constraints include dynamic constraints, the ego vehicle trajectory prediction sparse query vector containing navigation information is used to optimize the ego vehicle predicted trajectory under multiple constraints to obtain the ego vehicle path planning result, including: Using the ego vehicle trajectory containing navigation information to predict a sparse query vector, regressing the motion information of the ego vehicle, the motion information including the speed and acceleration of the ego vehicle; The motion information is used to optimize the predicted trajectory of the ego vehicle to obtain a ego vehicle path planning result. 5. The method according to claim 3 or 4, characterized in that, when the multiple constraints include a safety distance constraint, the ego vehicle trajectory prediction sparse query vector containing navigation information is used to optimize the ego vehicle predicted trajectory under multiple constraints to obtain an ego vehicle path planning result, including: Predicting a relative position relationship between the obstacle and the ego vehicle based on the ego vehicle trajectory prediction sparse query vector containing navigation information and the obstacle trajectory prediction sparse query vector; The relative position relationship is used to perform distance constraint optimization processing on the predicted trajectory of the ego vehicle to obtain a ego vehicle path planning result. 6. The method according to claim 2, characterized in that determining the obstacle reference trajectory prediction sparse query vector according to the obstacle historical position information comprises: Determine the detection frame whose confidence score in the obstacle detection result exceeds the score threshold as the tracking target detection frame; Determine the sparse query vector corresponding to the tracking target detection box as the tracking sparse query vector; Encoding the obstacle historical position information corresponding to the tracking sparse query vector into a historical position sparse query vector; The historical position sparse query vector and the tracking sparse query vector are interactively processed to obtain an obstacle reference trajectory prediction sparse query vector. 7. The method according to claim 1, wherein the reference detection sparse query vector is obtained by the following steps: Obtaining a historical detection sparse query vector and a tracking sparse query vector at a previous moment, wherein the historical detection sparse query vector includes: historical detection sparse query vectors at N moments before the current moment, where N is an integer greater than 1; A reference detection sparse query vector is obtained according to the historical detection sparse query vector and the tracking sparse query vector at the last moment. 8. The method according to claim 1, characterized in that the processing of the current moment sensor data to obtain a sparse sensor feature vector comprises: Performing feature extraction on the current moment sensor data to obtain a sparse initial sensor feature vector; Calculating the spatial position information of the sparse initial sensor feature vector according to the current sensor data; The spatial position information and the sparse initial sensor feature vector are fused to obtain a sparse sensor feature vector. 9. The method according to any one of claims 1 to 8, characterized in that end-to-end autonomous driving is achieved through a multi-tasking processing network; In the case where the end-to-end autonomous driving includes obstacle detection and map construction, the labels of the training data of the multi-task processing network include: real position information of obstacles and real map information; In the case where the end-to-end autonomous driving includes obstacle detection, map construction and trajectory prediction, the labels of the training data of the multi-task processing network include: real position information of obstacles, real map information, real trajectory of obstacles and real trajectory of the vehicle; In the case where the end-to-end autonomous driving includes obstacle detection, map construction, trajectory prediction and path planning, the labels of the training data of the multi-task processing network include: the real position information of the obstacle, the real map information, the real trajectory of the obstacle, the real trajectory of the vehicle, the real motion information of the vehicle, and the real relative position relationship between the obstacle and the vehicle. 10. An end-to-end autonomous driving device, characterized in that the device comprises: A data acquisition module is used to obtain the current sensor data of the vehicle; A data processing module, used for processing the sensor data at the current moment to obtain a sparse sensor feature vector; A first interaction module is used to interactively process a reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least includes: a historical detection sparse query vector determined based on historical sensor data of the vehicle; A second interaction module is used to interactively process the reference map sparse query quantity and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least includes: a historical map sparse query vector determined based on historical sensor data of the vehicle; A detection construction module, configured to perform obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and to perform map construction according to the map sparse query vector to obtain an online map; The automatic driving module is used to automatically drive the vehicle based on the obstacle detection result and the online map. 11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the end-to-end autonomous driving method according to any one of claims 1 to 9 are implemented. 12. A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the end-to-end autonomous driving method described in any one of claims 1-9 are implemented.