CN114359334A - Target tracking method, apparatus, computer equipment and storage medium - Google Patents

Abstract

The application relates to a target tracking method, a device, computer equipment and a storage medium, wherein the method comprises the steps of obtaining first current frame data of a target scene collected by a first sensor and second current frame data of the target scene collected by at least one second sensor different from the first sensor in type, obtaining actual measurement characteristic information of each candidate detection object in the current frame, wherein each detection object in the first current frame data and each detection object in the second current frame data are successfully associated and matched, and then tracking each candidate detection object according to the actual measurement characteristic information of each candidate detection object in the current frame and a predicted detection frame of each target to be tracked in the current frame. The method can accurately determine the target to be tracked of each detection object, and effectively complete target tracking of each frame.

Description

Target tracking method, apparatus, computer equipment and storage medium

technical field

The present application relates to the field of tracking technology, and in particular, to a target tracking method, device, computer equipment and storage medium.

Background technique

With the development of sensor technology and computer technology, simultaneous positioning and map construction solutions based on various sensors have been widely used in the fields of robot autonomous navigation, unmanned driving, mobile measurement and battlefield environment construction.

For example, during target tracking, the data information of the target can be detected by the sensor, and the target tracking can be realized after analyzing the detected data information. Usually, the data information obtained by different sensors has different dimensions, different architectures, and different emphases of features (contour, size, trajectory, category, color, texture, etc.), and for the same target, the extracted features are different. However, in the traditional technology, a single sensor is often used to track the target, which is easy to cause insufficient feature dimension when detecting the target, resulting in low accuracy of the target tracking result.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide a target tracking method, device, computer equipment and storage medium that can improve the accuracy of target tracking results in view of the above technical problems.

In a first aspect, an embodiment of the present application provides a target tracking method, which includes:

Obtain the first current frame data of the target scene collected by the first sensor, and the second current frame data of the target scene collected by at least one second sensor; the first sensor and the second sensor are different types of sensors;

According to the first current frame data and the second current frame data, the measured feature information of each candidate detection object in the current frame is obtained; the candidate detection objects are the detection objects in the first current frame data and the detection objects in the second current frame data Associating successfully matched detection objects; the measured feature information represents the inherent characteristic information of each candidate detection object;

According to the historical detection frame of each target to be tracked in the previous frame, predict the detection frame of each target to be tracked in the current frame;

Each candidate detection object is tracked according to the measured feature information of each candidate detection object in the current frame and the predicted detection frame of each target to be tracked in the current frame.

In one of the embodiments, the above-mentioned measured feature information includes the measured three-dimensional detection frame and the measured track feature; the above-mentioned measured feature information of each candidate detection object in the current frame and each target to be tracked predict the detection frame in the current frame, right Each candidate detection object is tracked, including:

Obtain the intersection ratio between the measured 3D detection frame of each candidate detection object in the current frame and the 3D detection frame predicted by each target to be tracked in the current frame;

Obtain the similarity between the measured track features of the remaining detection objects and the historical track features of each target to be tracked in the previous frame; the remaining detection objects are candidate detection objects whose intersection ratio is less than a preset intersection ratio threshold;

Track each candidate detection object whose intersection ratio is greater than the intersection ratio threshold, and each remaining detection object whose similarity is greater than the preset similarity threshold.

In one embodiment, the above-mentioned first current frame data of the target scene collected by the first sensor and the second current frame data of the target scene collected by at least one second sensor include:

For the data collected by any frame sensor, obtain the timestamp T1 of the first current frame data and the timestamp T2 of the second current frame data;

If the interval between T1 and T2 is less than the preset interval threshold, then determine that the first current frame data and the second current frame data are the data collected by the current frame;

If the interval between T1 and T2 is greater than the preset threshold, discard the first current frame data and the second current frame data, and re-acquire the first current frame data timestamp T1 and the second current frame data timestamp T2 of the next frame .

In one embodiment, obtaining the timestamp T1 of the first current frame data and the timestamp T2 of the second current frame data above includes:

If the first current frame data and the second current frame data do not carry a time stamp, then the acquisition time of the first current frame data and the acquisition time of the second current frame data are converted into the same time axis to obtain T1 and T2.

In one of the embodiments, before obtaining the first current frame data of the target scene collected by the first sensor and the second current frame data of the target scene collected by at least one second sensor, the method further includes:

The sampling frequency of the first sensor is adjusted to be the same as the sampling frequency of the second sensor, and the external parameter information between the first sensor and the second sensor is calibrated.

In one of the embodiments, the first sensor is a camera device; the second sensor is a lidar;

Then the above-mentioned calibration of the external parameter information between the first sensor and the second sensor includes:

Adjust the relative pose information between the camera device and the lidar to the target pose information, and obtain the calibrated external parameter information of the camera device according to the preset calibration algorithm;

The external parameter information of the camera device is calibrated according to the calibrated external parameter information.

In one embodiment, the above-mentioned first current frame data is pixel data collected by a camera device, and the second current frame data is point cloud data collected by lidar; the above-mentioned measured feature information includes the measured three-dimensional detection frame and the measured trajectory feature;

Then above-mentioned according to the first current frame data and the second current frame data, determine the measured feature information of each candidate detection object in the current frame, including:

Obtain the 2D detection frame of each detection object in the pixel data and the measured 3D detection frame of each detection object in the point cloud data;

Matching the two-dimensional detection frame of each detection object in the pixel data and the measured three-dimensional detection frame of each detection object in the point cloud data to determine candidate detection objects;

According to the two-dimensional detection frame and feature information of each candidate detection object in the pixel data, and the measured 3D detection frame and feature information of each detection object in the point cloud data, the measured trajectory characteristics of each candidate detection object are determined.

In one of the embodiments, the above-mentioned two-dimensional detection frame of each detection object in the pixel data and the measured three-dimensional detection frame of each detection object in the point cloud data are matched, and the candidate detection object is determined, including:

mapping each measured 3D detection frame to a corresponding 2D mapped detection frame;

Obtain the intersection ratio between each two-dimensional detection frame and each two-dimensional mapping detection frame, and determine the detection object whose intersection ratio is greater than the threshold of the intersection ratio as a candidate detection object.

In one of the embodiments, according to the two-dimensional detection frame and feature information of each candidate detection object in the pixel data, and the measured 3D detection frame and feature information of each detection object in the point cloud data, determine the detection object of each candidate. Measured trajectory features, including:

The two-dimensional detection frame and feature information of each candidate detection object in the pixel data are comprehensively determined as the two-dimensional trajectory feature of each candidate detection object; the measured three-dimensional detection frame and feature information of each candidate detection object in the point cloud data are comprehensively determined is the three-dimensional trajectory feature of each candidate detection object;

Convert the three-dimensional trajectory features of each candidate detection object into the corresponding two-dimensional mapping trajectory features;

According to the fusion data of each 2D mapped trajectory feature and each 2D trajectory feature, the measured trajectory feature of each candidate detection object is extracted.

In one of the embodiments, the above-mentioned three-dimensional trajectory features of each candidate detection object are converted into corresponding two-dimensional mapping trajectory features, including:

Convert the three-dimensional coordinates of the point cloud points in the measured three-dimensional detection frame corresponding to each three-dimensional trajectory feature into two-dimensional coordinates;

According to the two-dimensional coordinates of each point cloud point and the z-axis coordinate of each point cloud point in the three-dimensional coordinates, obtain the bird's-eye view corresponding to each measured three-dimensional detection frame;

According to the two-dimensional coordinates of each point cloud point and the intensity of each point cloud point, the intensity map corresponding to each measured three-dimensional detection frame is obtained;

According to the density of each point cloud point in the z-axis direction in the bird's-eye view, the density map corresponding to each measured 3D detection frame is obtained;

The bird's-eye view, the intensity map and the density map are combined to obtain the 2D mapping trajectory feature corresponding to each 3D trajectory feature.

In one of the embodiments, the above-mentioned according to the two-dimensional coordinates of each point cloud point and the z-axis coordinate of each point cloud point in the three-dimensional coordinates, obtain the bird's-eye view corresponding to each measured three-dimensional detection frame, including:

The z-axis coordinates of each point cloud point in the three-dimensional coordinates are normalized, and the normalized z-axis coordinate is determined as the pixel value of each point cloud point;

Taking the pixel value corresponding to each point cloud point as the pixel value of the corresponding two-dimensional coordinate position, the bird's-eye view corresponding to each measured three-dimensional detection frame is obtained.

In one embodiment, according to the two-dimensional coordinates of each point cloud point and the intensity of each point cloud point, the intensity map corresponding to each measured three-dimensional detection frame is obtained, including:

The intensity of each point cloud point is normalized, and the normalized intensity is determined as the pixel value of each point cloud point;

Taking the pixel value corresponding to each point cloud point as the pixel value of the corresponding two-dimensional coordinate position, the intensity map corresponding to each measured three-dimensional detection frame is obtained.

In one of the embodiments, above-mentioned according to the density of each point cloud point in the z-axis direction in the bird's-eye view, obtain the density map corresponding to each measured three-dimensional detection frame, including:

According to the number of point cloud points in the z-axis direction in the two-dimensional coordinate position of each point cloud point, the maximum value of the number of point cloud points in all coordinate positions, and the minimum value of the number of point cloud points in all coordinate positions, Determine the density of the point cloud points at each location in the z-axis direction;

The density of the point cloud points in each coordinate position in the z-axis direction is taken as the pixel value of the corresponding two-dimensional coordinate position, and the density map corresponding to each measured three-dimensional detection frame is obtained.

In one of the embodiments, above-mentioned according to the fusion data of each two-dimensional mapping track feature and each two-dimensional track feature, extract the measured track feature of each candidate detection object, including:

After compressing each two-dimensional trajectory feature and each two-dimensional mapping trajectory feature into the same scale, the fusion data matrix is obtained by splicing;

The feature information extracted from the fusion data matrix is determined as the measured trajectory feature of each candidate detection object.

In one embodiment, above-mentioned according to the historical detection frame of each target to be tracked in the previous frame, predict that each target to be tracked predicts the detection frame in the current frame, including:

Through the preset tracking algorithm model, according to the historical 3D detection frame of each target to be tracked in the previous frame, predict the 3D detection frame of each target to be tracked in the current frame; wherein, the tracking algorithm model is based on the space of uniform speed motion state Equations are constructed.

In a second aspect, an embodiment of the present application provides a target tracking device, the device comprising:

an acquisition module for acquiring the first current frame data of the target scene collected by the first sensor, and the second current frame data of the target scene collected by at least one second sensor; the first sensor and the second sensor are different types of sensors;

The feature acquisition module is used to acquire the measured feature information of each candidate detection object in the current frame according to the first current frame data and the second current frame data; the candidate detection objects are each detection object in the first current frame data and the second current frame data. Each detection object in the frame data is successfully associated and matched to the detection object; the measured feature information represents the inherent characteristic information of each candidate detection object;

The prediction module is used to predict the detection frame of each target to be tracked in the current frame according to the historical detection frame of each target to be tracked in the previous frame;

The tracking module is used for tracking each candidate detection object according to the measured feature information of each candidate detection object in the current frame and the prediction detection frame of each target to be tracked in the current frame.

In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the processor implements the steps of any one of the methods provided in the embodiments of the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of any one of the methods provided in the above-mentioned first aspect embodiment.

A target tracking method, apparatus, computer equipment, and storage medium provided by the embodiments of the present application, and the target tracking method, apparatus, computer equipment, and storage medium are obtained by acquiring the first current frame data of the target scene collected by the first sensor and at least one The second current frame data of the target scene collected by the second sensor of a different type from the first sensor is obtained, and each candidate detection object determined according to the successful correlation matching between each detection object in the first current frame data and each detection object in the second current frame data is obtained. The measured feature information of the object in the current frame, and then the detection frame is predicted in the current frame according to the measured feature information of each candidate detection object in the current frame and the predicted target to be tracked in the current frame, and each candidate detection object is tracked. In this method, when each detection object in the current frame is tracked, the measured feature information of each detection object is determined according to the data collected by two or more different types of sensors, that is, the data collected by various types of sensors is synthesized to Determining the measured feature information of the detected objects in each frame can accurately and completely reflect the features of the detected objects in each frame. In this way, when the measured feature information of each candidate detection object is matched with the accurate data of each target to be tracked , the target to be tracked to which each detection object belongs can be accurately determined, and the target tracking of each frame can be effectively completed. In addition, the candidate detection objects are selected according to the successful correlation matching between the detection objects in the first current frame data and the detection objects in the second current frame data, which ensures the consistency of detection objects in the data collected by various types of sensors.

Description of drawings

1 is an application environment diagram of a target tracking provided by an embodiment;

Fig. 1a is a positional relationship between a lidar and a camera device in one embodiment;

Figure 1b is an internal structure diagram of a computer device in one embodiment;

2 is a schematic flowchart of a target tracking method provided by an embodiment;

3 is a schematic flowchart of a target tracking method provided by another embodiment;

4 is a schematic flowchart of a target tracking method provided by another embodiment;

5 is a schematic flowchart of a target tracking method provided by another embodiment;

6 is a schematic flowchart of a target tracking method provided by another embodiment;

7 is a schematic flowchart of a target tracking method provided by another embodiment;

8 is a schematic flowchart of a target tracking method provided by another embodiment;

9 is a schematic flowchart of a target tracking method provided by another embodiment;

10 is a schematic flowchart of a target tracking method provided by another embodiment;

11 is a schematic flowchart of a target tracking method provided by another embodiment;

12 is a flowchart of a target tracking method provided by another embodiment;

FIG. 13 is a structural block diagram of a target tracking apparatus provided by an embodiment.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

A target tracking method provided by the present application can be applied to the application environment shown in FIG. Among them, laser radar 01, camera equipment 02 and computer equipment can communicate; laser radar 01 includes but is not limited to pulse radar, continuous wave radar, meter wave radar, decimeter wave radar, centimeter wave radar, etc., including 8 Line, 16 line, 24 line, 32 line, 64 line, 128 line lidar; camera equipment 02 includes but is not limited to professional camera, CCD camera, network camera, camcorder, black and white camera, color camera, infrared camera, X-ray Cameras, unannounced cameras, etc.; computer equipment 03 includes but is not limited to servers, various terminals: personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, etc. camera.

Among them, the position between the lidar 01 and the camera device 02 is relatively fixed, and the installation method can be that the radar and the camera are installed on the roadside pole, and the position is not limited. Schematic diagram of the installation of the laser radar 01 and the camera equipment 02. Figure 1b provides a diagram of the internal structure of a computer device 03 comprising a processor, memory and network interface connected by a system bus. Among other things, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The computer device's database is used to store target tracking related data. The network interface of the computer device is used to communicate with external terminals through a network connection. The computer program is executed by a processor to implement a target tracking method.

The embodiments of the present application provide a target tracking method, apparatus, computer equipment and storage medium, which can improve the accuracy of target tracking results. The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems will be described in detail below through embodiments and in conjunction with the accompanying drawings. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. It should be noted that, in a target tracking method provided by the present application, the execution subject of FIG. 2 to FIG. 12 is a computer device, wherein the execution subject may also be a target tracking device, wherein the device can be implemented by software, hardware or software and hardware. The combined implementation becomes part or all of the computer equipment.

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of the present application, but not all of the embodiments.

In one embodiment, FIG. 2 provides a target tracking method, which involves a computer device according to the data of the same scene collected by two or more different types of sensors, and the association of detected objects according to the data collected by different types of sensors After matching, the prediction data and the measured data of the successfully matched detection objects are matched and analyzed, and then the successfully matched detection objects in the matching analysis result correspond to the target to be tracked, and the specific process of tracking the target to be tracked is shown in Figure 2. shown, the method includes:

S101: Acquire first current frame data of a target scene collected by a first sensor, and second current frame data of a target scene collected by at least one second sensor; the first sensor and the second sensor are different types of sensors.

Among them, the target scene refers to the scene where the target to be tracked is located. For example, if the target is a vehicle and the vehicle is driving on a certain road, then a certain range of the road is the target scene.

The first sensor includes, but is not limited to, a camera device or a lidar. Similarly, the second sensor also includes, but is not limited to, a camera device or a lidar; however, the first sensor and the second sensor are different types of sensors. If the sensor is a lidar, then the second sensor is a camera device. Please refer to the installation method of the lidar and the camera device as shown in Figure 1a.

The current frame data of the target scene collected by the first sensor is the first current frame data. Taking the first sensor as a lidar as an example, the first current frame data is the collected target scene. The current data is 3D point cloud data, for example, the target For the vehicle, the vehicle is driving on a certain road, and the lidar set on the roadside scans a certain range of the road to obtain the three-dimensional point cloud data in the spatial scene. Likewise, if the second sensor is a camera device, then the second current frame data is the two-dimensional pixel data of the current frame of the target scene collected by the camera device, such as video data of the target scene.

S102, according to the first current frame data and the second current frame data, obtain the measured feature information of each candidate detection object in the current frame; the candidate detection objects are each detection object in the first current frame data and each detection object in the second current frame data. The detected objects are associated with the detected objects that have been successfully matched; the measured feature information represents the inherent characteristic information of each candidate detected object.

Among them, the candidate detection objects are detection objects that are successfully correlated and matched in the first current frame data and the second current frame data. For example, there are 5 detection objects in the first current frame data, and there are also 5 detection objects in the second current frame data. , but there are only 4 correlation matches, so these 4 detection objects are candidate detection objects.

In practical applications, after the first current frame data and the second current frame data are obtained, the detection objects in the first current frame data and the second current frame data are correlated and matched to obtain candidate detection objects, and then the detection objects are obtained from the first current frame data and the second current frame data. The measured feature information of the current frame of each candidate detection object is obtained from the first current frame data and the second current frame data.

In addition, when the computer device determines the measured three-dimensional detection frame and the measured trajectory feature of the current frame of the candidate detection object, the first current frame data and the second current frame data must be valid frames. Optionally, the valid frame refers to the synchronization of the collection time of the first current frame data and the second current frame data, for example, the collection time interval between the two is less than a preset threshold.

The measured feature information represents the inherent characteristic information of each candidate detection object. Optionally, the measured feature information includes the measured 3D detection frame and the measured trajectory feature, wherein the measured 3D detection frame and the measured trajectory feature refer to the data based on the first current frame data and the measured trajectory feature. The three-dimensional detection frame of each target and the trajectory feature of each target calculated from the actual information in the second current frame data. Among them, the three-dimensional detection frame is the detection frame of each candidate detection object in the point cloud, and the trajectory feature may be a feature such as a directional gradient histogram feature that can reflect various information of the detection object; 2. The position information of the detection frame of each candidate detection object in the current frame data, the color and texture information in the detection frame, etc., comprehensively determine the trajectory feature of each candidate detection object, as the measured trajectory feature of the current frame of each candidate detection object. Of course, in practical application, the actually measured feature information may also include a two-dimensional detection frame and a two-dimensional trajectory feature of a candidate detection object, which is not limited in this embodiment of the present application.

S103, according to the historical detection frame of each target to be tracked in the previous frame, predict the detection frame of each target to be tracked in the current frame.

For each target to be tracked, the position changes of its motion trajectory in consecutive frames are related, and some prediction methods can be used to predict the target's future trajectory based on the target's trajectory. Now what needs to be predicted is the predicted detection frame of each target to be tracked in the current frame, and the feature information (including detection frame and trajectory features) of each target to be tracked in the previous frame is already known, so the target can be tracked according to each target to be tracked. In the historical detection frame of the previous frame, each target to be tracked is predicted to predict the detection frame in the current frame. In practical applications, the detection frame may include a two-dimensional detection frame or a three-dimensional detection frame, which is not limited in this embodiment of the present application. The 3D detection frame is predicted in the frame, and the historical 3D detection frame here refers to the known 3D detection frame that has occurred in the previous frame of the target to be tracked. For example, the computer equipment can use the preset Kalman filter to predict the three-dimensional detection frame of each target to be tracked in the current frame according to the historical three-dimensional detection frame of each target to be tracked in the previous frame.

S104: Track each candidate detection object according to the measured feature information of each candidate detection object in the current frame and the predicted detection frame of each target to be tracked in the current frame.

Since the historical detection frame of each target to be tracked in the previous frame is already known, the historical detection frame of each target to be tracked in the previous frame is the standard, and the predicted detection frame of each target to be tracked in the current frame can be used as the standard. The identification basis of the target to be tracked in the current frame; for example, taking the 3D detection frame as an example, by judging the difference between the measured 3D detection frame of each candidate detection object in the current frame and the predicted 3D detection frame of each target to be tracked in the current frame Whether it matches or not, it can be determined which target to be tracked each candidate detection object belongs to, so that each candidate detection object can be tracked. It should be noted here that there are multiple targets to be tracked in each frame of data during target tracking, so it is necessary to determine which target to be tracked each detection object in the current frame data belongs to, and determine which target to be tracked belongs to each detection object in the current frame data. After the object belongs to the target to be tracked, the trajectory of each to-be-tracked mark in the current frame data can be determined in turn. Therefore, in all the embodiments of the present application, those in the tracking process (that have not been successfully tracked) are called detection objects, and those that have been successfully tracked are called objects to be tracked (or targets), which will not be described in detail later.

In the target tracking method provided in this embodiment, the first current frame data of the target scene collected by the first sensor and the second current frame data of the target scene collected by at least one second sensor of a different type from the first sensor are obtained. Each detection object in the first current frame data and each detection object in the second current frame data are correlated and matched successfully. The measured feature information of each candidate detection object in the current frame, and then according to the actual measurement of each candidate detection object in the current frame The feature information and each predicted target to be tracked predict a detection frame in the current frame, and track each candidate detection target. In this method, when tracking each detection object in the current frame, the measured characteristic information of each detection object is determined according to the data collected by more than two different types of sensors, that is, the data collected by various types of sensors is synthesized to Determining the measured feature information of the detected objects in each frame can accurately and completely reflect the features of the detected objects in each frame. In this way, when the measured feature information of each candidate detection object is matched with the accurate data of each target to be tracked, The target to be tracked to which each detection object belongs can be accurately determined, and the target tracking of each frame can be effectively completed. In addition, the candidate detection objects are screened out according to the successful correlation matching between the detection objects in the first current frame data and the detection objects in the second current frame data, which ensures the consistency of detection objects in the data collected by various types of sensors.

An embodiment is provided to describe the process of predicting the detection frame of each target to be tracked in the current frame according to the feature information of each target to be tracked in the previous frame in the above step S103. In this embodiment, the three-dimensional detection frame is used as an example. Note, this embodiment includes: predicting the three-dimensional detection frame of each target to be tracked in the current frame according to the historical three-dimensional detection frame of each target to be tracked in the previous frame through a preset tracking algorithm model; wherein, the tracking algorithm The model is constructed based on the space equations of uniformly variable motion states.

Among them, the state space equation is established according to the different motion states of the target in space, which can reflect the changes of the target's trajectory at different times, motion information and other expressions; the tracking algorithm model constructed based on the state space equation, for example, Carl The Mann filter can be closer to the real information when the target moves in space. After the tracking algorithm model is constructed, the predicted 3D detection frame of each target to be tracked in the current frame is predicted by the tracking algorithm model according to the historical 3D detection frame of each target, which can make the predicted 3D detection frame of each target to be tracked more accurate.

Since the space equation of the uniformly variable motion state couples the motion of the target in space with time, and the influence of acceleration is considered, the trajectory prediction error can be smaller, and the tracking effect with large speed changes can be improved. Therefore, the tracking algorithm model based on the space equation of the uniform speed motion state is very accurate in predicting the 3D detection frame of each target to be tracked in the current frame based on the historical 3D detection frame of each target to be tracked in the previous frame.

Illustratively, taking each target detection frame at each moment as a rectangular frame as an example, the detection result of each frame of data is the target frame (for example, the point cloud is a three-dimensional target frame, and the image is a two-dimensional target frame); The motion in the image is regarded as a uniformly variable motion, and the following state space equation (1) is constructed to reflect the information change of the target when the uniformly variable motion is in the image.

代表时间t之前同 一目标在图像x轴、y轴方向的速度，

代表时间t之前同一目标在图像x轴、y轴方向的加速度；α′、h′代表当前帧图像中目标的轨迹的长宽比和高度，α、h代表时间t之前目 标的轨迹的长宽比和高度，

代表同一目标在时间t之前长宽比变化率、高度变化率。Among them, in the above formula, x', y' represent the coordinates of the detection frame center point of the target in the current frame image on the x-axis and y-axis of the image, x, y represent the target's detection frame center point before time t on the image x-axis, y-axis the coordinates of the y-axis,

represents the speed of the same target in the x-axis and y-axis directions of the image before time t,

Represents the acceleration of the same target in the x-axis and y-axis directions of the image before time t; α', h' represent the aspect ratio and height of the target's trajectory in the current frame image, α, h represent the length and width of the target's trajectory before time t ratio and height,

It represents the change rate of aspect ratio and height change rate of the same target before time t.

Among them, the above t represents the time when the tracking fails, then the coordinates of the center point of the detection frame before t are based on the last image in the collected data that has been tracked, and the speed and acceleration refer to the average speed and acceleration of a certain time period before time t. average acceleration. Then, based on the above state space equation (1), the detection frame of the target in the image at each moment (each frame of data collected by each sensor) can be predicted, and the real-time position tracking of each target can be performed. Since the state space equation (1) is coupled with time, considering the acceleration, the influence of acceleration is considered in the trajectory prediction stage, which can make the trajectory prediction error smaller and improve the tracking effect with large speed changes.

On the basis of the above embodiments, the embodiments of the present application also provide a target tracking method, which involves a computer device predicting the current frame according to the measured feature information of each candidate detection object in the current frame and each target to be tracked in the current frame Detection frame, the specific process of tracking each candidate detection object. In this embodiment, the measured feature information includes the measured 3D detection frame and the measured trajectory feature, and according to the historical 3D detection frame of each target to be tracked in the previous frame, predict each to be tracked. The target predicts the three-dimensional detection frame in the current frame as an example to illustrate, as shown in Figure 3, the above-mentioned S104 steps include:

S201: Obtain the intersection ratio between the actually measured 3D detection frame of each candidate detection object in the current frame and the 3D detection frame predicted by each target to be tracked in the current frame.

This embodiment describes the matching process of the measured 3D detection frame of each candidate detection object in the current frame and the predicted 3D detection frame of each target to be tracked in the current frame. The operation is to match the similarity between the measured 3D detection frame of each candidate detection object in the current frame and each target to be tracked in the current frame to predict the similarity between the 3D detection frame in the current frame.

For example, the measured 3D detection frame of each candidate detection object in the current frame is A1, and the predicted 3D detection frame of each target to be tracked in the current frame is B1, then the similarity between A1 and B1 is determined by calculating the intersection ratio of A1 and B1. . Among them, the intersection and union ratio is the ratio of the intersection area and the union area of A1 and B1.

After the intersection ratio is determined, if the intersection ratio is higher than the preset intersection ratio threshold, the two are very similar, and it can be determined that the two match successfully, while the intersection ratio is lower than the preset intersection ratio threshold. There is a large difference between the two, and the two fail to match.

S202, obtain the similarity between the measured track features of the remaining detection objects and the historical track features of each target to be tracked in the previous frame; the remaining detection objects are candidate detection objects whose intersection ratio is less than a preset intersection ratio threshold.

After the above-mentioned calculation of the intersection ratio between the measured 3D detection frame of each candidate detection object and the predicted 3D detection frame of each target to be tracked, compare the intersection ratio of the intersection ratio smaller than the preset intersection ratio threshold to the corresponding candidate detection object. It is called the remaining detection object.

Each target to be tracked is tracked frame by frame in consecutive frames of the video. When tracking each target in the current frame, the trajectory features of the targets that have been tracked in the previous frame are known, then the previous The trajectory features of each target in the frame are called historical trajectory features. For the remaining detected objects, the similarity between the measured trajectory features of the remaining detected objects in the current frame and the historical trajectory features of each target to be tracked in the previous frame is calculated. to determine whether the two match. For example, to obtain the cosine similarity between the two, the distance measurement, such as calculating the Euclidean distance, may also be used, which is not limited in this embodiment. For example, the trajectory feature may be a Histogram of Oriented Gradient (HOG) feature, or other features, which are not limited in this embodiment.

After the similarity is determined, if the similarity is higher than the preset similarity threshold, it means that the two are very similar, and it can be determined that the two are successfully matched, while the similarity lower than the preset similarity threshold indicates that there is a difference between the two. larger, the two will fail to match.

S203, track each candidate detection object whose intersection ratio is greater than the intersection ratio threshold, and each remaining detection object whose similarity is greater than a preset similarity threshold.

The above-mentioned intersection ratio is greater than the preset intersection ratio threshold and the similarity is greater than the preset similarity threshold, indicating that the matching is successful. For all candidate detection objects that are successfully matched by the intersection ratio, it can be determined that the candidate detection object corresponds to the to-be-tracked object. The identification of the target is the identification of the known target to be tracked corresponding to the predicted 3D detection frame that is successfully matched; and for the similarity greater than the preset similarity threshold, it can be determined that the identification of the candidate detection object corresponding to the target to be tracked is a successful match. The identification of the known target to be tracked corresponding to the historical trajectory feature of . After the identification of each candidate detection object that has been successfully matched is determined, each candidate detection object is associated with the corresponding identification. After the association, it is possible to determine which target to be tracked each candidate detection object is according to the associated identification, thus completing Mark and track successfully matched candidate detection objects.

Of course, in practical applications, the candidate detection objects that are successfully matched can also be determined only by the method of cross-combination ratio, or the candidate detection objects that are successfully matched can be determined only by the method of similarity, which is not limited in the embodiment of the present application.

Optionally, after marking and tracking the successfully matched candidate detection objects, the current frame data of the target to be tracked to which each candidate detection object belongs can be updated according to the measured three-dimensional detection frame and the measured trajectory features of each candidate detection object. Detection box and trajectory information. It can be understood that the measured 3D detection frame and the measured trajectory feature of the target to be tracked in the updated current frame data can be used as the known historical 3D detection frame and historical trajectory feature in the previous frame data of the next frame of data.

The target tracking method provided by this embodiment firstly matches the measured 3D detection frame of each candidate detection object in the current frame with the predicted 3D detection frame of the target to be tracked in the current frame based on the intersection and comparison, and then detects the candidates that have not been successfully matched. Similarity matching is performed between the measured trajectory features of the object and the historical trajectory features of the target to be tracked in the previous frame. The whole process uses different matching methods to mix and match, that is, matching from different dimensions, which can effectively match the candidates in the current frame. The detection objects are all matched to the corresponding to-be-tracked targets, which avoids losing tracking due to the fact that the target is not matched successfully when the target is occluded, so that each target to be tracked can be tracked stably during the tracking process. And because the three-dimensional detection frame can more closely and accurately reflect the physical form of the target, in this embodiment, the three-dimensional detection frame of the detection object is used for cross-combination matching, which improves the accuracy of target matching and further increases the stability of target tracking. sex.

As mentioned above, the data of any frame (at any time) acquired by the first sensor and the second sensor needs to be a valid frame, and the valid frame refers to the time synchronization between the first current frame data and the second current frame data, so it is necessary to Determine whether the time before the first current frame data and the second current frame data in each frame is synchronized. Based on this, an embodiment is provided for description. As shown in FIG. 4 , the embodiment includes:

S301, for the data collected by any frame sensor, obtain the time stamp T1 of the first current frame data and the time stamp T2 of the second current frame data.

Specifically, it can be determined by acquiring the timestamp T1 of the first current frame data and the timestamp T2 of the second current frame data. In practical applications, when the first sensor and the second sensor collect data, a timestamp may or may not exist on the collected data, so the first current frame data and the second current frame data carry the time. In the case of stamping, for example, the first current frame data and the second current frame data both carry the time when the GPS module of the sensor collects the data. In this case, the computer device can directly obtain the time stamp T1 and the first current frame data 2. Timestamp T2 of the current frame data.

Optionally, for the situation that the first current frame data and the second current frame data do not carry timestamps, then the collection time of the first current frame data and the collection time of the second current frame data are converted into the same time axis. , to obtain the timestamp T1 of the first current frame data and the timestamp T2 of the second current frame data, respectively.

Specifically, if there is no GPS module in the sensor, and the precise time given by the GPS module cannot be obtained, then the system time of each sensor itself is converted into the system time of the computer equipment (that is, the processing platform), and unified under the system time of the processing platform. , the timestamps of the first current frame data and the second current frame data can be accurately judged. For example, at the same time, the system time of the processing platform is 4:00, the time of the first sensor system is 4:03, and the time of the second sensor system is 4:02. The time of the first current frame data is converted into 4:00 (timestamp T1), and the time of the second current frame data collected by the second sensor is converted into 4:00 (timestamp T2), thereby obtaining the first current frame The data timestamp T1 and the timestamp T2 of the second current frame data.

S302, if the interval between T1 and T2 is less than a preset interval threshold, determine that the first current frame data and the second current frame data are the data collected by the current frame; if the interval between T1 and T2 is greater than the preset threshold, discard For the first current frame data and the second current frame data, the time stamp T1 of the first current frame data and the time stamp T2 of the second current frame data of the next frame are obtained again.

After obtaining T1 and T2, determine the relationship between the interval between T1 and T2 (the time interval is the absolute value |T1-T2|) and the preset interval threshold (for example, 10ms). If the interval between T1 and T2 is smaller than the predetermined interval If the interval threshold is set, then it can be determined that the first current frame data and the second current frame data are time-synchronized, and if they are valid frames, the first current frame data and the second current frame data can be directly determined as the current moment (current frame). data. However, if the interval between T1 and T2 is greater than the preset threshold, it means that the time of the first current frame data and the second current frame data are not synchronized, and it is an invalid frame, then discard the first current frame data and the second current frame data, and re-acquire The judgment process is performed on the timestamp T1 of the first current frame data of the next frame and the timestamp T2 of the second current frame data. For example, according to a certain frame rate (such as 10Hz) to find the next frame of data.

Optionally, if the interval between T1 and T2 is smaller than the preset interval threshold, discard the smaller value in T1 and T2, keep the larger value, and then acquire the smaller value in the next frame of data collected by the sensor with the smaller value. Timestamp T, compare the new T with the retained larger value to determine whether the interval between the two is less than the preset interval threshold, if so, retain the new T value and the larger value, otherwise, continue to compare the next time The new T and the retained larger value, until the interval between the latest T and the retained larger value is less than the preset interval threshold. For example, the time stamp of the data collected by the lidar is T1, the time stamp of the data collected by the camera device is T2, and the preset interval threshold is 10Hz; if T1>T2 at the beginning, then discard T2, and then use the new T2 collected by the camera device. Compared with T1, if the interval between new T2 and T1 is less than 10 Hz, keep new T2 and T1; but if the interval between new T2 and T1 is > 10 Hz, continue down to the interval between T2 and T1 newly obtained by the imaging device.

In this embodiment, whether the data collected by each sensor is synchronous is judged by the time stamp of the data collected by each sensor, and the asynchronous data is discarded, and only the synchronous data is retained, so that the data during target tracking is more accurate.

In addition, before the above-mentioned acquisition of the first current frame data of the target scene collected by the first sensor and the second current frame data of the target scene collected by at least one second sensor, some preprocessing work may be performed to further ensure that each sensor collects Synchronization of data. Then optionally, in one embodiment, the method further includes: adjusting the sampling frequency of the first sensor to be the same as the sampling frequency of the second sensor, and calibrating the external parameter information between the first sensor and the second sensor.

The preprocessing preparation includes adjusting the sampling frequency of each sensor and calibrating the external parameter information of each sensor.

Among them, the adjustment of the sampling frequency of each sensor may be to adjust the frequency of use of each sensor when installing each sensor; it may also be to implant the adjustment program in the computer equipment in advance, and periodically adjust the program (the specified sampling frequency can be carried) Sent to each sensor, instructing each sensor to adjust its own sampling frequency.

Optionally, using the first sensor as a camera device; the second sensor as a lidar; calibrating the external parameter information between the first sensor and the second sensor, including: comparing the relative pose between the camera device and the lidar The information is adjusted to target pose information, and the calibrated external parameter information of the camera device is obtained according to a preset calibration algorithm; the external parameter information of the camera device is calibrated according to the calibrated external parameter information.

The external parameter information includes: pose information (relative position and relative angle) and external parameter information of the camera device

In practical applications, in order to enable the lidar and camera equipment to obtain the surrounding environment information comprehensively and effectively, when installing the lidar and the camera equipment, there must be an appropriate relative position and relative angle between them. For example, referring to the installation angle of the lidar and the camera device shown in Figure 1a, the relative position and relative angle between the lidar and the camera device should be guaranteed to ensure the relative position and relative angle between the lidar and the camera device. The surrounding environment information can be obtained comprehensively and effectively.

For example, acquiring target pose information, which includes a preset relative position and relative angle of the target, then the computer device instructs the camera device and the lidar to adjust its pose according to the relative position and relative angle of the target.

其中，该映射关系中，

为摄像设备的内参矩阵，

为摄像设备的 外参矩阵，

为世界坐标系中各点的坐标矩阵，

为像素坐标系中点的坐标矩阵。Based on the laser radar and camera equipment after adjusting the pose information, the computer equipment calibrates the external parameter information of the camera equipment. For example, the external parameter information of the camera equipment can be calibrated according to the mapping relationship between the point cloud and the image. Among them, the mapping relationship between the point cloud and the image represents the relationship between the world coordinate system (the coordinate system used by the lidar) and the pixel coordinate system (the coordinate system used by the pixels in the image of the camera device):

Among them, in this mapping relationship,

is the internal parameter matrix of the camera device,

is the extrinsic parameter matrix of the camera device,

is the coordinate matrix of each point in the world coordinate system,

is the coordinate matrix of the point in the pixel coordinate system.

Then, the coordinate matrix of each point in the world coordinate system and the coordinate matrix of the point in the pixel coordinate system can be determined by obtaining the point cloud data collected by the actual lidar and the pixel data collected by the camera device. The internal parameter information of the camera device can be obtained directly, and then the external parameter matrix of the camera device is obtained from the mapping relationship, and then the obtained external parameter matrix is used as the new external parameter information of the camera device, that is, the camera device is completed. Calibration.

In this embodiment, the external parameter information of the camera device and the laser radar is calibrated through the preprocessing preparation work, so that the camera device and the laser radar can comprehensively and effectively obtain the surrounding environment information, thereby making the collected data of the target scene more accurate. In addition to the above-mentioned calibration of the external parameter information of the camera device, the external parameter information (relative position and relative angle) between the camera device and the lidar can also be calibrated. The information calibration can ensure that the spatial uniformity of the data collected by the camera equipment and the lidar can be maintained and the accuracy of the data conversion can be improved.

The process of “determining the measured feature information of each candidate detection object in the current frame according to the first current frame data and the second current frame data” in the above step S102 will be described through the following embodiments. In this embodiment, the measured features are still used. The information includes the measured three-dimensional detection frame and the measured trajectory feature as an example to illustrate, as shown in FIG. 5 , in one embodiment, step S102 includes:

S401: Acquire a two-dimensional detection frame of each detection object in the pixel data, and a measured three-dimensional detection frame of each detection object in the point cloud data.

In this embodiment, the first current frame data is pixel data collected by a camera device, and the second current frame data is point cloud data collected by lidar as an example for description.

Obtain the two-dimensional detection frame of each detection object from the pixel data collected by the camera equipment. For example, through a preset deep learning algorithm model, such as the YOLOv3 model, etc., output the pixel rectangular frame location where each detection object is located, target confidence, and target classification. Among them, the rectangular frame positioning can reflect the two-dimensional detection frame of each detected object, and the target confidence can reflect the accuracy of each detected object relative to the target to be tracked; the target classification information reflects the category of each detected object, such as , the target is a car, a person or an animal.

Obtain the 3D detection frame (actually measured 3D detection frame) of each detection object from the point cloud data. For example, through a preset deep learning algorithm model, such as SECOND model, etc., output the location and size of each detection object in the point cloud coordinate system. , heading angle and other information; among them, the location, size, heading angle, etc. of each detection object in the point cloud coordinate system can reflect the measured 3D detection frame of each detection object. It should be noted that the measured 3D detection frame and the 2D detection frame in all the embodiments of this application refer to the actual measured 3D or 2D detection frame of the detection object, but the 3D detection frame is called the measured 3D detection frame for the purpose of To maintain the unity with the previous description, the previous description is called the measured 3D detection frame to distinguish it from the predicted 3D detection frame.

S402: Match the two-dimensional detection frame of each detection object in the pixel data with the measured three-dimensional detection frame of each detection object in the point cloud data to determine candidate detection objects.

After obtaining the 2D detection frame of each detection object in the pixel data and the measured 3D detection frame of each detection object in the point cloud data, first perform correlation matching between the detection objects in the pixel data and the detection objects in the point cloud data , and determine the detection object with successful association matching as the candidate detection object.

Optionally, as shown in FIG. 6 , an embodiment of determining candidate detection objects includes:

S501: Map each measured three-dimensional detection frame to a corresponding two-dimensional mapped detection frame.

Map the measured 3D detection frame of each detection object to the corresponding 2D mapping detection frame. For example, each measured 3D detection frame in the point cloud data can be input into the pre-trained conversion network model as input data, and the output result That is, to obtain the two-dimensional image corresponding to each measured three-dimensional detection frame in the point cloud data. For another example, firstly, based on the mapping relationship between the preset three-dimensional coordinate system and the two-dimensional coordinate system, after converting the three-dimensional coordinates of each point cloud point in each measured three-dimensional detection frame into two-dimensional coordinates, the measured three-dimensional coordinates are obtained. Detect the two-dimensional coordinates of each point cloud point in the frame, and then determine the corresponding pixel value on the point cloud point corresponding to each two-dimensional coordinate. After determining the pixel value of each two-dimensional coordinate position, each measured three-dimensional detection frame is obtained. the corresponding 2D image. In this way, the pixel values of each point at the two-dimensional coordinate position of the obtained two-dimensional detection frame are also different, and the characteristics of the three-dimensional point cloud point are also retained.

S502, obtain the intersection ratio between each two-dimensional detection frame and each two-dimensional mapping detection frame, and determine the detection object whose intersection ratio is greater than the threshold value of the intersection ratio as a candidate detection object.

After obtaining the two-dimensional mapping detection frame corresponding to the measured three-dimensional detection frame of each detection object, the intersection ratio between each two-dimensional mapping detection frame and the two-dimensional detection frame of each detection object in the pixel data is obtained, and then the intersection ratio is greater than A detection object with a preset intersection ratio threshold is determined as a candidate detection object. It can be understood that the candidate detection objects are essentially a pair of detection frames. It should be noted that, the preset intersection ratio threshold here and the preset intersection ratio threshold for matching in the foregoing embodiment may be the same or different, which is not limited in this embodiment.

By correlating and matching the 2D detection frame of each detection object in the pixel data and the measured 3D detection frame of each detection object in the point cloud data, the candidate detection objects are screened out, and the detection objects in the data collected by various types of sensors are ensured. consistency.

S403, according to the two-dimensional detection frame and feature information of each candidate detection object in the pixel data, and the measured three-dimensional detection frame and feature information of each detection object in the point cloud data, determine the measured trajectory feature of each candidate detection object.

After each candidate detection object is determined, further determine the measured trajectory feature of each candidate detection object. For example, extract feature information in the two-dimensional detection frame according to the two-dimensional detection frame of each candidate detection object; for example, extract the Histogram of Oriented Gradient (HOG) feature of the two-dimensional detection frame as the corresponding two-dimensional detection frame The feature of the detection frame; according to the measured 3D detection frame of each candidate detection object, extract the feature information in the measured 3D detection frame; then comprehensively determine each candidate according to the feature information in the 2D detection frame and the measured 3D detection frame Detect the measured trajectory features of the object. Among them, the measured 3D detection frame of each candidate detection object is the 3D detection frame of each detection object in the point cloud data.

In this embodiment, candidate detection objects are first determined based on the two-dimensional detection frame of each detection object in the pixel data and the measured three-dimensional detection frame of each detection object in the point cloud data, and then the candidate detection objects are determined according to the two-dimensional detection frame of each candidate detection object. The feature information and the feature information in the measured three-dimensional detection frame determine the measured trajectory characteristics of each candidate detection object. When determining the measured trajectory characteristics of each detection object, it is determined according to the data collected by more than two different types of sensors. The data collected by various types of sensors is combined to determine the measured trajectory characteristics of the detection objects in each frame, which can accurately , which completely reflects the characteristics of the detection objects in each frame, so that when matching the measured trajectory characteristics of each candidate detection object with the historical trajectory characteristics of each to-be-tracked target, it is possible to accurately determine the to-be-tracked belonging of each detection object. target, effectively complete the target tracking of each frame.

Provide a realizable way of determining the measured trajectory features of each candidate detection object in the above-mentioned S403 step, as shown in Figure 7, this embodiment includes:

S601, comprehensively determine the two-dimensional detection frame and feature information of each candidate detection object in the pixel data as the two-dimensional trajectory feature of each candidate detection object; determine the measured three-dimensional detection frame and feature information of each candidate detection object in the point cloud data The three-dimensional trajectory features of each candidate detection object are comprehensively determined.

The feature information extracted from the two-dimensional detection frame in the pixel data of each candidate detection object and the two-dimensional detection frame are comprehensively determined as the two-dimensional trajectory feature of each candidate detection object, that is, the two-dimensional trajectory feature can be regarded as each candidate detection object. The 2D detection frame carries the feature information of the pixels in the 2D detection frame.

The measured 3D detection frame of each candidate detection object in the point cloud data and the feature information extracted from the measured 3D detection frame are comprehensively determined as the 3D trajectory feature of each candidate detection object. Similarly, the 3D trajectory feature can be regarded as each candidate detection frame. The measured 3D detection frame of the object carries the feature information of the point cloud points in the measured 3D detection frame.

S602: Convert the three-dimensional trajectory feature of each candidate detection object into a corresponding two-dimensional mapping trajectory feature.

Since the 3D trajectory feature is the measured 3D detection frame in the form of display, when the 3D trajectory feature of each candidate detection object is converted into the corresponding 2D mapped trajectory feature, it can be converted according to the mapping relationship between the point cloud and the midpoint of the image, where , after converting the three-dimensional coordinates of each point cloud point of the measured three-dimensional detection frame into two-dimensional coordinates based on the mapping relationship, the two-dimensional coordinates of each point cloud point in the measured three-dimensional detection frame are obtained, and then the corresponding two-dimensional coordinates are determined. After determining the pixel value of each two-dimensional coordinate position corresponding to the pixel value on the point cloud point, the two-dimensional detection frame corresponding to the measured three-dimensional detection frame is obtained. In this way, the pixel value of each point in the two-dimensional coordinate position of the obtained two-dimensional detection frame is also different, and the characteristics of the point cloud points in the measured three-dimensional detection frame are also retained. In this way, the converted two-dimensional mapping trajectory The features include the detection frame in the three-dimensional trajectory feature and the feature information of the points in the detection frame.

S603, according to each two-dimensional mapping trajectory feature and the fusion data of each two-dimensional trajectory feature, extract the measured trajectory feature of each candidate detection object. Optionally, after compressing each two-dimensional trajectory feature and each two-dimensional mapping trajectory feature into the same scale, a fusion data matrix is obtained by splicing; the feature information extracted from the fusion data matrix is determined as the measured trajectory feature of each candidate detection object. .

Each 2D mapping trajectory feature is the feature information of each candidate detection object determined from the point cloud data, and each 2D trajectory feature is the feature information of each candidate detection object determined from the pixel data, extracted from the two fusion data. The feature information is the measured trajectory feature of each candidate detection object.

Still taking the above example to illustrate, the two-dimensional trajectory feature information of each candidate detection object and each two-dimensional mapping trajectory feature are adjusted to the same ratio, for example, the two-dimensional trajectory feature information of each candidate detection object is compressed to N*N*3 size , the two-dimensional mapping trajectory features are scaled to the same scale. After the scale adjustment, the two are spliced to obtain a fusion data matrix. Then, the fusion data matrix is input into the convolutional neural network for feature extraction, and the extracted features are the measured trajectory features of each candidate detection object.

In this embodiment, the three-dimensional trajectory features of each candidate detection object in the point cloud data are converted into two-dimensional mapped trajectory features, and then the two-dimensional mapped trajectory features and the two-dimensional trajectory features of each candidate detection object in the pixel data are spliced Then, a fusion data matrix is formed, and the feature information extracted from the fusion data matrix is determined as the measured trajectory characteristics of each candidate detection object. Determining the measured trajectory characteristics of the detection objects in each frame from multi-dimensional data can accurately and completely reflect the characteristics of the detection objects in each frame. When accurate data is matched, the target to be tracked to which each detection object belongs can be accurately determined, and the target tracking of each frame can be effectively completed.

Below with a specific embodiment, the process of converting the three-dimensional trajectory feature of each candidate detection object into the corresponding two-dimensional mapping trajectory feature in the above-mentioned S602 will be described, as shown in Figure 8, S602 includes:

S701: Convert the three-dimensional coordinates of the point cloud points in the measured three-dimensional detection frame corresponding to each three-dimensional trajectory feature into two-dimensional coordinates.

The conversion of the three-dimensional coordinates to the two-dimensional coordinates can be performed based on the mapping relationship between the three-dimensional coordinate system and the two-dimensional coordinate system. For example, the mapping relationship between the three-dimensional coordinate system and the two-dimensional coordinate system can be shown in the following formula (1):

In the above formula (1), a and b represent the coordinates of the point cloud point in the three-dimensional coordinate system, a _t and b _t represent the coordinates of the point cloud point in the two-dimensional coordinate system after mapping, and h represents the point cloud boundary to the y-axis. Distance, w represents the distance from the point cloud boundary to the x-axis, where the x-axis and the y-axis are the coordinate axes with the lidar as the origin.

Based on the mapping relationship of formula (1), the three-dimensional coordinates of each point cloud point in the three-dimensional detection frame are converted into two-dimensional coordinates.

S702, according to the two-dimensional coordinates of each point cloud point and the z-axis coordinate of each point cloud point in the three-dimensional coordinates, obtain a bird's-eye view corresponding to each measured three-dimensional detection frame; according to the two-dimensional coordinates of each point cloud point and each point cloud point The intensity map corresponding to each measured 3D detection frame is obtained; according to the density of each point cloud point in the z-axis direction in the bird's-eye view, the density map corresponding to each measured 3D detection frame is obtained.

In this step, the bird's-eye view, intensity map and density map corresponding to each measured three-dimensional detection frame are obtained respectively, and these three maps are all two-dimensional.

The bird's-eye view corresponding to the measured three-dimensional detection frame is obtained according to the two-dimensional coordinates of each point cloud point and the z-axis coordinate of each point cloud point in the three-dimensional coordinates. The two-dimensional coordinates of the point cloud points are obtained by conversion in the above steps, and the two-dimensional coordinates of each point cloud point determine the position of the point cloud point on the two-dimensional plane. Among them, the z-axis coordinate of the point cloud point in the three-dimensional coordinate is the Z coordinate of the point cloud point in the three-dimensional coordinate where the point cloud data is located, and the Z coordinate can also be regarded as the height of the point cloud point in the two-dimensional coordinate. Based on the measured 3D detection frame, the position of each point cloud point on the 2D plane and the height of the point cloud point in the 2D coordinates, the bird's-eye view corresponding to each measured 3D detection frame can be obtained.

Optionally, as shown in Figure 9, the process of acquiring the bird's-eye view corresponding to each measured 3D detection frame includes:

S801, normalize the z-axis coordinates of each point cloud point in the three-dimensional coordinates, and determine the normalized z-axis coordinate as the pixel value of each point cloud point.

First, determine the pixel value of each point cloud point in the bird's-eye view according to the z-axis coordinate of the point cloud point in the three-dimensional coordinate. Because the range of pixel value is 0-255, it is necessary to first determine the z of the point cloud point in the three-dimensional coordinate. The axis coordinates are normalized and normalized to between 0 and 255. The normalized value is the pixel value of each point cloud point in the bird's-eye view.

S802, take the pixel value corresponding to each point cloud point as the pixel value of the corresponding two-dimensional coordinate position, and obtain the bird's-eye view corresponding to each actually measured three-dimensional detection frame.

After obtaining the pixel value of each point cloud point in the bird's eye view, the coordinate position of each point cloud point can be determined by combining the two-dimensional coordinates of each point cloud point, and then the pixel value of the corresponding point cloud point is filled to the two-dimensional coordinate position. , to get the final bird's-eye view.

The acquired intensity map is determined according to the two-dimensional coordinates of each point cloud point and the intensity of each point cloud point, wherein the intensity of each point cloud point is obtained from the intensity of each point cloud when the lidar collects point cloud data. Based on the position of each point cloud point in the measured 3D detection frame on the 2D plane and the intensity of the point cloud point, the intensity map corresponding to each measured 3D detection frame can be obtained.

Optionally, as shown in FIG. 10 , the process of acquiring the intensity map corresponding to each measured 3D detection frame includes:

S901, normalize the intensity of each point cloud point, and determine the intensity after the normalization process as the pixel value of each point cloud point.

First, determine the pixel value of each point cloud point in the bird's eye view according to the intensity of the point cloud point. Similarly, the range of pixel value is 0-25 5, so the intensity of the point cloud point should be normalized first. The normalized value is the pixel value of each point cloud point in the intensity map.

S902, using the pixel value corresponding to each point cloud point as the pixel value of the corresponding two-dimensional coordinate position, obtain the intensity map corresponding to each actually measured three-dimensional detection frame.

After obtaining the pixel value of each point cloud point in the intensity map, the coordinate position of each point cloud point can be determined by combining the two-dimensional coordinates of each point cloud point, and then the pixel value of the corresponding point cloud point is filled to the two-dimensional coordinate position. , to get the final intensity map.

The obtained density map is determined according to the density of each point cloud point in the z-axis direction in the bird's-eye view, where the density of the point cloud point in the z-axis direction refers to the three-dimensional coordinates of the point cloud, each x-axis and y-axis The position determined by the axis is the density of the number of point cloud points in the z-axis direction.

Optionally, as shown in FIG. 11 , the process of obtaining the density map corresponding to each measured 3D detection frame includes:

S1001, according to the number of point cloud points in the z-axis direction in the two-dimensional coordinate position of each point cloud point, the maximum value of the number of point cloud points in all coordinate positions, and the minimum number of point cloud points in all coordinate positions value, which determines the density of point cloud points in the z-axis direction at each location.

The two-dimensional coordinate position of a pixel is determined by the x coordinate and the y coordinate, then each point cloud point in the three-dimensional detection frame corresponds to a two-dimensional coordinate position, and the point cloud point of each two-dimensional coordinate position in the z-axis direction is obtained. , then for one of the point cloud points, according to the number of point cloud points in the z-axis direction in the two-dimensional coordinate position of the point cloud point, the maximum value of the number of point cloud points in all coordinate positions, all The minimum value of the number of point cloud points in the coordinate position can determine the density of the point cloud points in the z-axis direction.

For example, the density of point cloud points in the z-axis direction can be determined by the following formula (2).

Among them, ρ _i is the density of the ith pixel (that is, the two-dimensional coordinate position), ci is the number of point clouds of the _ith pixel, c _min is the minimum value of the number of point clouds in each pixel, and c _max represents The maximum number of point clouds in each pixel.

S1002 , taking the density of the point cloud points at each coordinate position in the z-axis direction as the pixel value of the corresponding two-dimensional coordinate position, and obtaining a density map corresponding to each measured three-dimensional detection frame.

After obtaining the density of each point cloud point, the density is used as the pixel point of the point cloud point, and the coordinate position of each point cloud point can be determined by combining the two-dimensional coordinates of each point cloud point, and then the pixel corresponding to the point cloud point can be determined. The values are filled to the 2D coordinate positions to get the final density map.

The process of obtaining the bird's-eye view, intensity map and density map is as above, and the following is the process of merging the bird's-eye view, intensity map and density map.

S705, combine the bird's-eye view, the intensity map, and the density map to obtain a two-dimensional mapping trajectory feature corresponding to each three-dimensional trajectory feature.

After obtaining the bird's-eye view, intensity map and density map corresponding to each measured three-dimensional detection frame in the point cloud data, the three two-dimensional images are merged. Optionally, the bird's-eye view, intensity map and density map are used as R, The images of the three channels G and B are combined to obtain the two-dimensional mapping detection frame corresponding to each measured three-dimensional detection frame, and the feature information of each point cloud point in the measured three-dimensional detection frame is reflected in the two-dimensional mapping detection frame. On each pixel point in , each 3D trajectory feature corresponding to each measured 3D detection frame, and the 2D mapping trajectory feature corresponding to each 2D mapping detection frame, the 2D mapping trajectory feature corresponding to each 3D trajectory feature is obtained.

In this embodiment, since the bird's-eye view, the intensity map and the density map corresponding to each measured 3D detection frame are obtained first, the pixel value of each point in the three figures is converted based on the actual data of the point cloud data. The information of the point cloud points in the point cloud is preserved. When the three images are merged to obtain the pixel value of each point in the two-dimensional mapping trajectory feature, it can also accurately reflect the point cloud point in the three-dimensional point cloud. Therefore, the two-dimensional mapping trajectory features can more accurately reflect the three-dimensional trajectory features.

In one embodiment, as shown in FIG. 12, an embodiment of a target tracking method is provided, and the embodiment includes:

S1101, adjusting the sampling frequency of the first sensor to be the same as the sampling frequency of the second sensor, and calibrating the external parameter information between the first sensor and the second sensor;

S1102, for the data collected by any frame sensor, obtain the time stamp T1 of the first current frame data and the time stamp T2 of the second current frame data;

S1103, if the interval between T1 and T2 is less than a preset interval threshold, determine that the first current frame data and the second current frame data are the data collected by the current frame; if the interval between T1 and T2 is greater than the preset threshold, discard the first current frame data and the second current frame data. a current frame data and a second current frame data, re-acquire the first current frame data timestamp T1 and the second current frame data timestamp T2 of the next frame;

S1104, obtain the two-dimensional detection frame of each detection object in the pixel data, and the measured three-dimensional detection frame of each detection object in the point cloud data;

S1105, according to the two-dimensional detection frame of each detection object in the pixel data and the measured three-dimensional detection frame of each detection object in the point cloud data, determine the candidate detection object;

S1106, comprehensively determine the two-dimensional detection frame and feature information of each candidate detection object in the pixel data as the two-dimensional trajectory feature of each candidate detection object; Comprehensively determine the three-dimensional trajectory features of each candidate detection object;

S1107, convert the three-dimensional coordinates of the point cloud points in the measured three-dimensional detection frame corresponding to each three-dimensional trajectory feature into two-dimensional coordinates;

S1108: Obtain a bird's-eye view corresponding to each measured 3D detection frame; obtain a density map corresponding to each measured 3D detection frame; obtain an intensity map corresponding to each measured 3D detection frame; The 2D mapping trajectory feature corresponding to the 3D trajectory feature;

S1109, according to the fusion data of each two-dimensional mapping track feature and each two-dimensional track feature, extract the measured track feature of each candidate detection object;

S1110, by a preset tracking algorithm model, according to the historical three-dimensional detection frame of each target to be tracked in the previous frame, predict that each target to be tracked predicts a three-dimensional detection frame in the current frame;

S1111, obtain the intersection ratio between the measured three-dimensional detection frame of each candidate detection object in the current frame and the predicted three-dimensional detection frame of each target to be tracked in the current frame;

S1112, obtain the similarity between the measured track features of the remaining detection objects and the historical track features of each target to be tracked in the previous frame; the remaining detection objects are candidate detection objects whose intersection ratio is less than a preset intersection ratio threshold;

S1113: Track each candidate detection object whose intersection ratio is greater than the intersection ratio threshold, and each remaining detection object whose similarity is greater than a preset similarity threshold.

The implementation principles and technical effects of the steps in the target tracking method provided in this embodiment are similar to those in the previous embodiments of the target tracking method, and are not repeated here. The implementation manner of each step in the embodiment of FIG. 12 is only an example, and each implementation manner is not limited, and the order of each step can be adjusted in practical application, as long as the purpose of each step can be achieved.

It should be understood that although the various steps in the flowcharts of Figures 2-12 are shown in sequence as indicated by the arrows, these steps are not necessarily performed sequentially in the sequence indicated by the arrows. Unless explicitly stated herein, there is no strict order in the execution of these steps, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIGS. 2-12 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed and completed at the same time, but may be executed at different times. These sub-steps or stages are not necessarily completed at the same time. The order of execution of the steps is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of sub-steps or stages of other steps.

In one embodiment, as shown in Figure 8, a target tracking device is provided, comprising: an acquisition module 10, a determination module 11, a prediction module 12 and a tracking module 13, wherein,

The acquiring module 10 is configured to acquire the first current frame data of the target scene collected by the first sensor and the second current frame data of the target scene collected by at least one second sensor; the first sensor and the second sensor are different types of sensors ;

The feature acquisition module 11 is used to acquire the measured feature information of each candidate detection object in the current frame according to the first current frame data and the second current frame data; the candidate detection objects are each detection object in the first current frame data and the second current frame data. The detected objects in the current frame data are successfully correlated and matched; the measured feature information represents the inherent characteristic information of each candidate detection object;

Prediction module 12, for predicting each target to be tracked predicts the detection frame in the current frame according to the historical detection frame of each target to be tracked in the previous frame;

The tracking module 13 is configured to track each candidate detection object according to the measured feature information of each candidate detection object in the current frame and the prediction detection frame of each candidate detection object in the current frame.

In one embodiment, the above-mentioned tracking module 13 includes:

The first acquisition unit is used to obtain the intersection ratio between the measured three-dimensional detection frame of each candidate detection object in the current frame and the three-dimensional detection frame predicted by each target to be tracked in the current frame;

The second obtaining unit is used to obtain the similarity between the measured track features of the remaining detection objects and the historical track features of each target to be tracked in the previous frame; the remaining detection objects are candidates whose intersection ratio is less than a preset intersection ratio threshold Detection object;

The first determining unit is configured to track each candidate detection object whose intersection ratio is greater than the intersection ratio threshold, and each remaining detection object whose similarity is greater than a preset similarity threshold.

In one embodiment, the above-mentioned obtaining module 10 includes:

The third acquisition unit is used for acquiring the first current frame data timestamp T1 and the second current frame data timestamp T2 for the data collected by any frame sensor;

The second determination unit is configured to determine that the first current frame data and the second current frame data are the data collected by the current frame if the interval between T1 and T2 is less than the preset interval threshold; if the interval between T1 and T2 is greater than the predetermined interval threshold If the threshold is set, the first current frame data and the second current frame data are discarded, and the time stamp T1 of the first current frame data and the time stamp T2 of the second current frame data of the next frame are re-acquired.

In one embodiment, the above-mentioned third acquisition unit is specifically configured to, if the first current frame data and the second current frame data do not carry a time stamp, collect the collection time of the first current frame data and the collection time of the second current frame data Time is converted into T1 and T2 under the same time axis.

In one embodiment, the apparatus further includes: an adjustment module configured to adjust the sampling frequency of the first sensor to be the same as the sampling frequency of the second sensor, and to calibrate the external parameter information between the first sensor and the second sensor.

In one embodiment, the first sensor is a camera device; the second sensor is a lidar;

The above adjustment module is specifically used to adjust the relative pose information between the camera device and the lidar to the target pose information, and obtain the calibrated external parameter information of the camera device according to the preset calibration algorithm; The parameter information is used to calibrate the external parameter information of the camera device.

In one embodiment, the above-mentioned first current frame data is the pixel data collected by the camera equipment, and the second current frame data is the point cloud data collected by the lidar; Then the above-mentioned feature acquisition module 11 includes:

A detection frame acquisition unit, used for acquiring the two-dimensional detection frame of each detection object in the pixel data, and the measured three-dimensional detection frame of each detection object in the point cloud data;

The candidate detection object determination unit is used to match the two-dimensional detection frame of each detection object in the pixel data and the measured three-dimensional detection frame of each detection object in the point cloud data to determine the candidate detection object;

The measured feature determination unit is used to determine the actual measurement of each candidate detection object according to the two-dimensional detection frame and feature information of each candidate detection object in the pixel data, and the measured three-dimensional detection frame and feature information of each detection object in the point cloud data. Trajectory features.

In one embodiment, the above-mentioned candidate detection object determination unit is specifically configured to map each 3D detection frame to a corresponding 2D mapped detection frame; obtain the intersection ratio between each 2D detection frame and each 2D mapped detection frame , and the detection object whose intersection ratio is greater than the threshold of the intersection ratio is determined as a candidate detection object.

In one embodiment, the above-mentioned measured feature determination unit includes:

The trajectory feature determination subunit is used to comprehensively determine the two-dimensional detection frame and feature information of each candidate detection object in the pixel data as the two-dimensional trajectory feature of each candidate detection object; the actual measurement of each candidate detection object in the point cloud data The three-dimensional detection frame and feature information are comprehensively determined as the three-dimensional trajectory features of each candidate detection object;

a conversion subunit, used to convert the three-dimensional trajectory features of each candidate detection object into the corresponding two-dimensional mapping trajectory features;

The feature extraction sub-unit is used to extract the measured track features of each candidate detection object according to the fusion data of each two-dimensional mapping track feature and each two-dimensional track feature.

In one embodiment, the above-mentioned conversion subunit includes:

The coordinate conversion subunit is used to convert the three-dimensional coordinates of the point cloud points in the measured three-dimensional detection frame corresponding to each three-dimensional trajectory feature into two-dimensional coordinates;

The bird's-eye view subunit is used to obtain the bird's-eye view corresponding to each measured three-dimensional detection frame according to the two-dimensional coordinates of each point cloud point and the z-axis coordinate of each point cloud point in the three-dimensional coordinates;

The intensity map subunit is used to obtain the intensity map corresponding to each measured three-dimensional detection frame according to the two-dimensional coordinates of each point cloud point and the intensity of each point cloud point;

The density map subunit is used to obtain the density map corresponding to each measured three-dimensional detection frame according to the density of each point cloud point in the z-axis direction in the bird's-eye view;

It is used to combine the bird's-eye view, intensity map and density map to obtain the 2D mapping trajectory feature corresponding to each 3D trajectory feature.

In one embodiment, the above-mentioned bird's-eye view subunit is specifically used to normalize the z-axis coordinates of each point cloud point in the three-dimensional coordinates, and determine the normalized z-axis coordinate as each point cloud point The pixel value of each point cloud point is taken as the pixel value of the corresponding two-dimensional coordinate position, and the bird's-eye view corresponding to each measured three-dimensional detection frame is obtained.

In one embodiment, the above-mentioned intensity map subunit is specifically used to normalize the intensity of each point cloud point, and determine the intensity after the normalization process as the pixel value of each point cloud point; The corresponding pixel value is taken as the pixel value of the corresponding two-dimensional coordinate position, and the intensity map corresponding to each measured three-dimensional detection frame is obtained.

In one embodiment, the above-mentioned density map subunit is specifically used for the maximum number of point cloud points in the z-axis direction in the two-dimensional coordinate position of each point cloud point and the maximum number of point cloud points in all coordinate positions. value, the minimum value of the number of point cloud points in all coordinate positions, determine the density of point cloud points at each position in the z-axis direction; take the density of point cloud points at each coordinate position in the z-axis direction as the corresponding two-dimensional The pixel value of the coordinate position is obtained, and the density map corresponding to each measured 3D detection frame is obtained.

In one embodiment, the above-mentioned feature extraction subunit is specifically used for compressing each two-dimensional trajectory feature and each two-dimensional mapping trajectory feature into the same scale, and then splicing to obtain a fusion data matrix; the features extracted from the fusion data matrix The information is determined as the measured trajectory features of each candidate detection object.

In one embodiment, the above-mentioned prediction module 12 is specifically configured to predict the 3D detection frame of each target to be tracked in the current frame according to the historical 3D detection frame of each target to be tracked in the previous frame through a preset tracking algorithm model ; Among them, the tracking algorithm model is constructed based on the space equation of uniformly variable motion state.

For the specific limitation of the target tracking device, reference may be made to the limitation on the target tracking method above, which will not be repeated here. Each module in the above-mentioned target tracking device can be implemented in whole or in part by software, hardware and combinations thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer equipment in the form of hardware, and can also be stored in the memory in the computer equipment in the form of software, so that the processor can call and execute the corresponding operations of the above-mentioned various modules.

In one embodiment, a computer device is provided, the computer device may be a terminal, and the internal structure diagram of which may be as shown in Fig. 1b. The computer equipment includes a processor, memory, a network interface, a display screen, and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media, internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with external terminals through a network connection. The computer program when executed by a processor implements a target tracking method. The display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball or a touchpad set on the shell of the computer equipment , or an external keyboard, trackpad, or mouse.

Those skilled in the art can understand that the structure shown in FIG. 1b is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, a computer program is stored in the memory, and the processor implements the following steps when executing the computer program:

Obtain the first current frame data of the target scene collected by the first sensor, and the second current frame data of the target scene collected by at least one second sensor; the first sensor and the second sensor are different types of sensors;

According to the first current frame data and the second current frame data, the measured feature information of each candidate detection object in the current frame is obtained; the candidate detection objects are the detection objects in the first current frame data and the detection objects in the second current frame data Associating successfully matched detection objects; the measured feature information represents the inherent characteristic information of each candidate detection object;

According to the historical detection frame of each target to be tracked in the previous frame, predict the detection frame of each target to be tracked in the current frame;

Each candidate detection object is tracked according to the measured feature information of each candidate detection object in the current frame and the predicted detection frame of each target to be tracked in the current frame.

The implementation principle and technical effect of the computer device provided by the above-mentioned embodiment are similar to those of the above-mentioned method embodiment, which will not be repeated here.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program implements the following steps when executed by a processor:

Obtain the first current frame data of the target scene collected by the first sensor, and the second current frame data of the target scene collected by at least one second sensor; the first sensor and the second sensor are different types of sensors;

According to the first current frame data and the second current frame data, the measured feature information of each candidate detection object in the current frame is obtained; the candidate detection objects are the detection objects in the first current frame data and the detection objects in the second current frame data Associating successfully matched detection objects; the measured feature information represents the inherent characteristic information of each candidate detection object;

According to the historical detection frame of each target to be tracked in the previous frame, predict the detection frame of each target to be tracked in the current frame;

Each candidate detection object is tracked according to the measured feature information of each candidate detection object in the current frame and the predicted detection frame of each target to be tracked in the current frame.

The implementation principle and technical effect of the computer-readable storage medium provided by the above-mentioned embodiments are similar to those of the above-mentioned method embodiments, and details are not described herein again.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided in this application may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, all It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are more specific and detailed, but should not therefore be construed as limiting the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application should be determined by the appended claims.

Claims (14)

1. a target tracking method, is characterized in that, described method comprises: Acquire the first current frame data of the target scene collected by the first sensor, and the second current frame data of the target scene collected by at least one second sensor; the first sensor and the second sensor are different types of sensors ; According to the first current frame data and the second current frame data, the actual measurement feature information of each candidate detection object in the current frame is obtained; the candidate detection object is each detection object and all detection objects in the first current frame data. Each detection object in the second current frame data is successfully correlated and matched to the detection object; the measured feature information represents the inherent characteristic information of each candidate detection object; According to the historical detection frame of each target to be tracked in the previous frame, predict the detection frame of each target to be tracked in the current frame; Each candidate detection object is tracked according to the measured feature information of each candidate detection object in the current frame and the predicted detection frame of each to-be-tracked object in the current frame. 2. The method according to claim 1, wherein the measured feature information comprises an actual measured three-dimensional detection frame and an actual measured trajectory feature; Described according to the measured feature information of each described candidate detection object in the current frame, and the predicted detection frame of each described target to be tracked in the current frame, each described candidate detection object is tracked, including: Obtain the intersection ratio between the measured 3D detection frame of each candidate detection object in the current frame and the 3D detection frame predicted by each of the to-be-tracked targets in the current frame; Obtain the similarity between the measured track features of the remaining detection objects and the historical track features of the respective targets to be tracked in the previous frame; the remaining detection objects are candidate detection objects whose intersection ratio is less than a preset intersection ratio threshold; Track each candidate detection object whose intersection ratio is greater than the intersection ratio threshold, and each remaining detection object whose similarity is greater than a preset similarity threshold. 3. The method according to claim 1 or 2, wherein the first current frame data is pixel data collected by a camera device, and the second current frame data is point cloud data collected by lidar; the The measured feature information includes the measured 3D detection frame and the measured trajectory feature; Then, determining the measured feature information of each candidate detection object in the current frame according to the first current frame data and the second current frame data, including: Obtain the two-dimensional detection frame of each detection object in the pixel data, and the measured three-dimensional detection frame of each detection object in the point cloud data; Matching the two-dimensional detection frame of each detection object in the pixel data and the measured three-dimensional detection frame of each detection object in the point cloud data to determine the candidate detection object; According to the two-dimensional detection frame and feature information of each candidate detection object in the pixel data, and the measured 3D detection frame and feature information of each detection object in the point cloud data, determine each candidate detection frame The measured trajectory features of the object. 4. The method according to claim 3, wherein the two-dimensional detection frame of each detection object in the pixel data is matched with the measured three-dimensional detection frame of each detection object in the point cloud data, and the determined The candidate detection objects include: mapping each of the measured three-dimensional detection frames to corresponding two-dimensional mapped detection frames; The intersection ratio between each of the two-dimensional detection frames and each of the two-dimensional mapping detection frames is acquired, and a detection object whose intersection ratio is greater than the threshold of the intersection ratio is determined as the candidate detection object. 5 . The method according to claim 3 , wherein, according to the two-dimensional detection frame and feature information of each candidate detection object in the pixel data, and the point cloud of each detection object The measured 3D detection frame and feature information in the data determine the measured trajectory characteristics of each candidate detection object, including: The two-dimensional detection frame and feature information of each candidate detection object in the pixel data are comprehensively determined as the two-dimensional trajectory feature of each candidate detection object; each candidate detection object is included in the point cloud data. The measured 3D detection frame and feature information are comprehensively determined as the 3D trajectory feature of each candidate detection object; Converting the three-dimensional trajectory feature of each candidate detection object into a corresponding two-dimensional mapping trajectory feature; According to each of the two-dimensional mapped trajectory features and the fusion data of each of the two-dimensional trajectory features, the measured trajectory features of each of the candidate detection objects are extracted. 6. The method according to claim 5, wherein the converting the three-dimensional trajectory feature of each candidate detection object into a corresponding two-dimensional mapping trajectory feature comprises: Converting the three-dimensional coordinates of the point cloud points in the measured three-dimensional detection frame corresponding to each of the three-dimensional trajectory features into two-dimensional coordinates; According to the two-dimensional coordinates of each of the point cloud points and the z-axis coordinates of each of the point cloud points in the three-dimensional coordinates, obtain a bird's-eye view corresponding to each of the measured three-dimensional detection frames; According to the two-dimensional coordinates of each of the point cloud points and the intensity of each of the point cloud points, an intensity map corresponding to each of the measured three-dimensional detection frames is obtained; Obtain a density map corresponding to each of the measured three-dimensional detection frames according to the density of each of the point cloud points in the z-axis direction in the bird's-eye view; The bird's-eye view, the intensity map, and the density map are combined to obtain a two-dimensional mapped trajectory feature corresponding to each of the three-dimensional trajectory features. 7 . The method according to claim 6 , wherein, according to the two-dimensional coordinates of each of the point cloud points and the z-axis coordinates of each of the point cloud points in the three-dimensional coordinates, the measured three-dimensional coordinates are obtained. 8 . The bird's-eye view corresponding to the detection frame, including: Normalize the z-axis coordinate of each of the point cloud points in the three-dimensional coordinates, and determine the normalized z-axis coordinate as the pixel value of each of the point cloud points; Taking the pixel value corresponding to each of the point cloud points as the pixel value of the corresponding two-dimensional coordinate position, a bird's-eye view corresponding to each of the measured three-dimensional detection frames is obtained. 8 . The method according to claim 6 , wherein the intensity map corresponding to each of the measured three-dimensional detection frames is obtained according to the two-dimensional coordinates of each of the point cloud points and the intensity of each of the point cloud points. 9 . ,include: Normalize the intensity of each of the point cloud points, and determine the intensity after the normalization process as the pixel value of each of the point cloud points; Taking the pixel value corresponding to each of the point cloud points as the pixel value of the corresponding two-dimensional coordinate position, the intensity map corresponding to each of the measured three-dimensional detection frames is obtained. 9 . The method according to claim 6 , wherein the obtaining a density map corresponding to each of the measured three-dimensional detection frames according to the density of each of the point cloud points in the z-axis direction in the bird’s-eye view, comprising: 10 . : According to the number of point cloud points in the z-axis direction in the two-dimensional coordinate position of each point cloud point, the maximum value of the number of point cloud points in all coordinate positions, and the minimum value of the number of point cloud points in all coordinate positions, Determine the density of the point cloud points at each location in the z-axis direction; The density of the point cloud points at each coordinate position in the z-axis direction is taken as the pixel value of the corresponding two-dimensional coordinate position, and a density map corresponding to each of the measured three-dimensional detection frames is obtained. 10 . The method according to claim 9 , wherein, according to the fusion data of each of the two-dimensional mapped trajectory features and each of the two-dimensional trajectory features, the measured trajectory features of each of the candidate detection objects are extracted, 10 . include: After compressing each of the two-dimensional trajectory features and each of the two-dimensional mapping trajectory features into the same scale, splicing to obtain a fusion data matrix; The feature information extracted from the fusion data matrix is determined as the measured trajectory feature of each candidate detection object. 11. The method according to claim 1 or 2, wherein, predicting the detection frame of each target to be tracked in the current frame according to the historical detection frame of each target to be tracked in the previous frame, comprising: Through the preset tracking algorithm model, according to the historical 3D detection frame of each target to be tracked in the previous frame, predict the 3D detection frame of each target to be tracked in the current frame; wherein, the tracking algorithm model is based on It is constructed from the space equation of the uniformly variable motion state. 12. A target tracking device, wherein the device comprises: an acquisition module, configured to acquire the first current frame data of the target scene collected by the first sensor, and the second current frame data of the target scene collected by at least one second sensor; the first sensor and the second sensor for different types of sensors; A feature acquisition module, configured to acquire the measured feature information of each candidate detection object in the current frame according to the first current frame data and the second current frame data; the candidate detection object is the first current frame data each detection object in the second current frame data and each detection object in the second current frame data are correlated and successfully matched; the measured feature information represents the inherent characteristic information of each candidate detection object; The prediction module is used to predict the detection frame of each target to be tracked in the current frame according to the historical detection frame of each target to be tracked in the previous frame; The tracking module is configured to track each candidate detection object according to the measured feature information of each candidate detection object in the current frame and the prediction detection frame of each of the to-be-tracked objects in the current frame. 13. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 11 when the processor executes the computer program . 14. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images