CN114724131A - A vehicle tracking method, device, electronic device and storage medium - Google Patents

Abstract

The present application relates to a vehicle tracking method, device, electronic device and storage medium. The method obtains the bounding box information of the vehicle in each of the video streams by performing vehicle detection processing on a plurality of video streams. Perform license plate recognition processing to obtain the license plate recognition results of vehicles in each of the video streams; based on the bounding box information of the vehicles in each of the video streams and the license plate recognition results, obtain the corresponding license plate recognition results for each of the video streams. Track segment; according to the license plate recognition result contained in each of the track segments, fuse a plurality of the track segments to generate a vehicle tracking result, obtain the track segment based on the bounding box information and the license plate recognition result, and based on the track The multi-track segment is merged to improve the accuracy and robustness of vehicle tracking.

Description

A vehicle tracking method, device, electronic device and storage medium

technical field

The present application relates to the field of computers, and in particular, to a vehicle tracking method, device, electronic device and storage medium.

Background technique

The new infrastructure has brought new opportunities to smart security. At present, the security monitoring and defense system of the smart industrial park mainly shoots high-definition images at close distances of road nodes, and then uses license plate recognition to perform illegal, illegal capture, and vehicle tracking on the video obtained, which improves the efficiency of management and control. However, due to the large area of the industrial park, the license plate is easily blurred or obscured in rainy and foggy weather or queuing at intersections, which makes it unsatisfactory to rely only on license plate recognition for continuous vehicle tracking across cameras.

SUMMARY OF THE INVENTION

The present application provides a vehicle tracking method, device, electronic device and storage medium to solve the problem of unsatisfactory tracking effect caused by continuous vehicle tracking across cameras through license plate recognition in the related art.

In a first aspect, the present application provides a vehicle tracking method. The vehicle tracking method includes: performing vehicle detection processing on multiple video streams to obtain bounding box information of vehicles in each of the video streams; Perform license plate recognition processing on the video stream to obtain the license plate recognition result of the vehicle in each of the video streams; based on the bounding box information of the vehicle in each of the video streams and the license plate recognition result, obtain each of the video track segments corresponding to the stream; and according to the license plate recognition results included in each of the track segments, fuse a plurality of the track segments to generate a vehicle tracking result.

In some examples of the present application, according to the license plate recognition result included in each of the trajectory segments, fusing a plurality of the trajectory segments to generate a vehicle tracking result includes: determining the Whether the included vehicle recognition results are accurate; when the ratio of the trajectory segments with accurate results to all the trajectory segments is not lower than the target ratio, directly fuse a plurality of the trajectory segments based on the license plate recognition results, to generate vehicle tracking results.

In some examples of the present application, according to the license plate recognition result included in each of the trajectory segments, fusing a plurality of the trajectory segments to generate a vehicle tracking result includes: determining the Whether the included vehicle identification results are accurate; when the ratio of the track segments with accurate results to all the track segments is lower than the target ratio, determine the moving direction of the vehicle in each of the track segments; according to each of the track segments The movement direction of the trajectory segment filters each of the trajectory segments to filter out the trajectory segments with different moving directions; the remaining trajectory segments after filtering are fused to generate a vehicle tracking result.

In some examples of the present application, determining the moving direction of the vehicle in each of the trajectory segments includes: acquiring each timestamp included in each of the trajectory segments; according to the timestamps in each of the trajectory segments and the position of the vehicle corresponding to each of the time stamps to determine the moving direction of the vehicle in each of the track segments.

In some examples of the present application, filtering each of the trajectory segments according to the moving direction of each of the trajectory segments, so as to filter out the trajectory segments with different moving directions, includes: adding multiple video streams into The included vehicles are taken as target vehicles in turn, and the movement direction of the target vehicle is determined; the trajectory segment corresponding to the target vehicle is determined, and the movement direction of the vehicle in the determined trajectory segment is compared with the movement direction of the target vehicle. compare; when the difference between the motion direction of the vehicle in any of the determined trajectory segments and the motion direction of the target vehicle reaches the target difference value, remove the trajectory segment that reaches the target difference value to filter out the motion direction different of the trajectory segments.

In some examples of the present application, fusing the remaining trajectory segments after filtering to generate a vehicle tracking result includes: merging the remaining trajectory segments after filtering and having the same license plate recognition result, Obtain at least two merged track segments; obtain the similarity between all the merged track segments, and perform track fusion according to the similarity among all the merged track segments to generate a vehicle tracking result.

In some examples of the present application, acquiring the trajectory segment corresponding to each of the video streams based on the bounding box information of the vehicle in each of the video streams and the license plate recognition includes: performing the process on the bounding box information. Vehicle re-identification processing to obtain the vehicle characteristics corresponding to each vehicle; respectively fuse the bounding box information, the vehicle characteristics and the license plate recognition results of the vehicles in each of the video streams to obtain each of the video The trajectory segment corresponding to the stream.

In a second aspect, the present application provides a vehicle tracking device. The vehicle tracking device includes: a detection module, where the inspection module is configured to perform vehicle detection processing on a plurality of video streams to obtain vehicles in each of the video streams. the bounding box information, perform license plate recognition processing on a plurality of the video streams, and obtain the license plate recognition results of the vehicles in each of the video streams; a determination module, the determination module is used to identify the vehicle based on each of the video streams. The bounding box information and the license plate recognition are used to obtain the trajectory segment corresponding to each of the video streams; the fusion module is configured to, according to the license plate recognition result contained in each of the trajectory segments, identify multiple The trajectory segments are fused to generate vehicle tracking results.

In a third aspect, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

memory for storing computer programs;

The processor is configured to implement the steps of the vehicle tracking method described in any one of the embodiments of the first aspect when executing the program stored in the memory.

In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the vehicle tracking method according to any one of the embodiments of the first aspect.

Compared with the prior art, the above-mentioned technical solutions provided in the embodiments of the present application have the following advantages:

This solution can be applied to computer vision processing in the field of deep learning technology. The method provided by the embodiment of the present application obtains the bounding box information of the vehicle in each video stream by performing vehicle detection processing on multiple video streams, so as to obtain the bounding box information of the vehicle in each of the video streams. Perform license plate recognition processing on a plurality of the video streams to obtain the license plate recognition results of the vehicles in each of the video streams; based on the bounding box information of the vehicles in each of the video streams and the license plate recognition results, obtain each The trajectory segment corresponding to the video stream; according to the license plate recognition result contained in each of the trajectory segments, a plurality of the trajectory segments are fused to generate a vehicle tracking result, which is obtained based on the bounding box information and the license plate recognition result Track segments, and combine multiple track segments based on track segments to improve the accuracy and robustness of vehicle tracking.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. In other words, on the premise of no creative labor, other drawings can also be obtained from these drawings.

1 is a schematic flowchart of an optional vehicle tracking method provided by an embodiment of the present application;

FIG. 2 provides a basic flow chart of another optional vehicle tracking method according to an embodiment of the present application;

FIG. 3 is a basic schematic diagram of a trajectory fusion provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of the basic structure of a vehicle tracking device provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present application.

FIG. 1 is a schematic flowchart of a vehicle tracking method provided by an embodiment of the application. As shown in FIG. 1 , the vehicle tracking method includes but is not limited to:

S101. Perform vehicle detection processing on multiple video streams to obtain bounding box information of vehicles in each of the video streams, and perform license plate recognition processing on multiple video streams to obtain license plate recognition of vehicles in each of the video streams result;

S102, based on the bounding box information of the vehicle in each of the video streams and the license plate recognition result, obtain a trajectory segment corresponding to each of the video streams;

S103. According to the license plate recognition result included in each of the trajectory segments, fuse a plurality of the trajectory segments to generate a vehicle tracking result.

It should be understood that the vehicle tracking method provided in this embodiment is applied to a multi-camera vehicle tracking system, where the multi-camera vehicle tracking system includes, but is not limited to, multiple cameras; Obtain the shooting images. At the same time, this embodiment does not limit the model and function of each camera in the multi-camera vehicle tracking system, and can be set flexibly by relevant personnel. For example, the camera has the function of high-definition shooting, and also has recording and sound playback. , infrared scanning and other functions.

It can be understood that the multiple video streams are captured by the cameras in the above-mentioned multi-camera vehicle tracking system, each video stream has at least one vehicle, and among the multiple video streams, each video stream may be different in different time periods. For example, there are N road sections in an industrial park, and each road section corresponds to a camera. When any vehicle travels to any road section, the camera on this road section captures the vehicle to form a video stream. When reaching other road sections, the camera corresponding to the other road sections will take pictures of the vehicle to form a video stream, reaching multiple video streams, each video stream being captured by different cameras in different time periods;

In some examples, among the multiple video streams, each video stream may also be captured by the same camera in different time periods. For example, there is a road segment, and a camera is set on the road segment. In a time period, the vehicle travels through the road segment. , the camera shoots the vehicle to form a video stream, and in another time period, the vehicle drives through the road section again, the camera shoots the vehicle to form a video stream, and reaches multiple video streams, and each video stream can also be in different time periods taken by the same camera;

In some examples, each video stream may also be captured by different cameras in the same time period. For example, there is a road segment, and multiple cameras are set at different positions on the road segment. When any vehicle travels to the road segment, the road segment is The vehicle is captured by multiple cameras to form multiple video streams, and each video stream is captured by different cameras in the same time period.

Following the above example, that is, the types of video streams contained in multiple video streams can be at least one of the following: video streams captured by the same camera in different time periods, video streams captured by different cameras in different time periods, different in the same time period The video stream captured by the camera.

In some examples of this embodiment, performing vehicle detection processing on multiple video streams to obtain bounding box information of vehicles in each of the video streams includes: inputting the multiple video streams into a vehicle detection model to The vehicle detection model performs vehicle detection processing on multiple video streams, and obtains the bounding box information of vehicles contained in each video stream; among them, each video stream is input into the vehicle detection model after sampling one frame per second. The model outputs the BBox information of each vehicle in a frame until every frame in the video stream is processed.

It can be understood that, in some examples, the output branch of the vehicle detection model includes three branches: one is to output the position of the predicted target box, that is, the vehicle BBox bounding box information (LOC), and the other is to output the target is the foreground. (true target) or background (invalid target) (Obj), a probability of what kind of target the target is output (i.e. the probability that the target is a vehicle), it should be understood that in some examples, the vehicle detection model after the output , it is necessary to remove the result that the probability that the target is a vehicle is lower than the preset probability, wherein the preset probability can be flexibly set by the relevant personnel, for example, the preset probability is 25%-0.45%, preferably, the preset probability is 35%, or That is, after the vehicle detection model is output, the bounding box information whose probability of the target being a vehicle is lower than 35% is removed, that is, not every frame can output an accurate vehicle bounding box information;

Following the above example, it can be understood that before the vehicle detection processing is performed on multiple video streams to obtain the bounding box information of the vehicles in each of the video streams, the method further includes: pretraining an initial vehicle detection model to obtain the vehicle detection model. Detection model, in this example, the initial vehicle detection model is pre-trained through the target detection data set, where the target detection data set can be an open source data set, such as the open source MS-COCO data set; wherein, the initial vehicle detection model can be based on YOLOX, Yolov5, The model proposed by the frameworks such as yolo, Yolov4, Yolov3, etc., this embodiment does not limit the framework of the initial vehicle detection model, which can be flexibly set by relevant personnel, as long as the bounding box information of the vehicle can be obtained;

In some examples, in order to enable the trained vehicle detection model to more accurately and quickly identify the vehicle in the usage scenario, the vehicle detection model can also be fine-tuned by using the free dataset corresponding to the scenario. Specifically, For example, the vehicle tracking method is used in an industrial park. In order to enable the vehicle detection model to more accurately and quickly identify vehicles in the industrial park, the monitoring cameras distributed in the industrial park are used to collect the data of multiple vehicles appearing in the industrial park. Image and video data are used as their own data set (preferably, the own data set contains 500 videos, 2000-10000 pictures), and the vehicle detection model is fine-tuned through the own data set (Fine-tune), after the fine-tuning is obtained It is more suitable for the vehicle detector model of the industrial park, and then more accurately detects the bounding box information of vehicles in different video streams captured from the industrial park.

In some examples of this embodiment, performing license plate recognition processing on a plurality of the video streams to obtain a license plate recognition result of a vehicle in each of the video streams includes: inputting the plurality of video streams into a license plate recognition model, In order to make the license plate recognition model perform license plate recognition processing on vehicles in multiple video streams, and obtain license plate recognition results; it is understandable that, similarly, each video stream is sampled one frame per second and then input into the license plate recognition model. The license plate recognition model outputs the license plate recognition results of each vehicle in a frame until every frame in the video stream is processed.

It should be understood that, in some examples, not every frame of the license plate recognition result can obtain accurate license plate information. Based on different shooting resolutions and different shooting angles, some frames of the video stream can identify accurate license plates. , some frames of the video stream cannot identify the exact license plate. Among them, in order to better perform license plate recognition on vehicles in use scenarios, the trained vehicle recognition model can also be fine-tuned by using the scene's own data set, which will not be repeated here.

In some examples of this embodiment, based on the bounding box information of the vehicle in each of the video streams and the license plate recognition result, acquiring a trajectory segment corresponding to each of the video streams includes:

Perform vehicle re-identification processing on the bounding box information to obtain vehicle features (ReID features) corresponding to each vehicle;

The bounding box information, the vehicle feature and the license plate recognition result of the vehicles in each of the video streams are respectively fused to obtain a trajectory segment corresponding to each of the video streams.

It can be understood that, performing vehicle re-identification processing on the bounding box information corresponding to the vehicle in each video stream includes: inputting the bounding box information corresponding to the vehicle in each video stream into the vehicle re-identification model, so that the vehicle re-identification model The vehicle re-identification process is performed on the bounding box information to obtain vehicle characteristics; the vehicle re-identification model is to identify the same vehicle from multiple different cameras according to the vehicle appearance characteristics. For example, at an intersection, there are 4 cameras that capture the same car from different angles, and the vehicle re-identification model needs to recognize the same vehicle in these 4 images.

In some examples, the vehicle re-identification model is a model obtained by pre-training a transformer-based initial vehicle re-identification model on an open source data set. This embodiment does not limit the basic framework of the initial vehicle re-identification model to be a transformer. It can be set flexibly by relevant personnel. Similarly, the open source data set is not limited by this embodiment, for example, the open source data set is the VeRi data set. In some examples, after pre-training the initial vehicle re-identification model based on the transformer on the open source dataset, in order to enable the trained vehicle re-identification model to more accurately retrieve the same vehicle between different cameras in the current scene, the same The pre-trained vehicle recognition model can be fine-tuned through its own data set to obtain a vehicle retrieval model that can more accurately identify vehicle features.

It can be understood that the bounding box information, vehicle features and license plate recognition results corresponding to the vehicles in each video stream are input into the single-camera vehicle tracking model, and the single-camera vehicle tracking model fuses and analyzes the above information to obtain the corresponding information of each video stream. Each track segment contains bounding box information, frame timestamps, vehicle features and license plate recognition results, as well as a vehicle ID generated for each vehicle, which is uniquely identified.

Continuing the above example, similarly, the single-camera vehicle tracking model is obtained by pre-training the initial single-camera vehicle tracking model. In some examples, in order to make the single-camera vehicle tracking model more suitable for the usage scenario, it is also possible to obtain the auto-tracking model obtained by shooting according to the usage scenario. There are datasets that fine-tune the single-camera vehicle tracking model, for example, fine-tune the Deep-SORT model on the own dataset to obtain a single-camera vehicle tracking model.

In some examples of this embodiment, according to the license plate recognition result included in each of the trajectory segments, fusing a plurality of the trajectory segments to generate a vehicle tracking result includes: determining each of the trajectory segments Whether the vehicle recognition result contained in the The license plate recognition result fuses a plurality of the trajectory segments to generate a vehicle tracking result. Wherein, by determining each trajectory segment, it is determined whether the vehicle recognition result contained in each of the trajectory segments is accurate, the number of trajectory segments with accurate license plate recognition results is determined as A, and the number of all trajectory segments is recorded as B, then compare the value of A/B with the target ratio, and then determine whether the ratio of the trajectory segment with accurate results to all the trajectory segments is lower than the target ratio; if the trajectory segment with accurate results accounts for all the trajectory segments When the ratio of the trajectory segments is not lower than the target ratio, fuse a plurality of the trajectory segments directly based on the license plate recognition result to generate a vehicle tracking result; fuse a plurality of the trajectory segments directly based on the license plate recognition result , to generate the vehicle tracking result. Specifically, the trajectory segments with the same license plate recognition result are fused to generate the vehicle tracking result. It should be noted that when merging multiple said trajectory segments, it will delete the ones that do not contain accurate The trajectory segment of the license plate recognition result is discarded; the target ratio can be flexibly set by relevant personnel, preferably, the target ratio is set to any of 85% to 95%, and preferably, the target ratio is 90%.

Following the previous example, for example, it is determined whether the vehicle recognition results contained in each of the trajectory segments are accurate. If the license plate recognition results of more than 90% of the trajectory segments contain accurate license plate information, the multi-camera tracklets are merged based on the license plates (the remaining ones). Tracklets that do not contain accurate license plate information are discarded to obtain multi-camera tracking trajectories.

In some examples of this embodiment, according to the license plate recognition result included in each of the trajectory segments, fusing a plurality of the trajectory segments to generate a vehicle tracking result includes: determining each of the trajectory segments determine whether the vehicle identification result contained in the vehicle identification result is accurate; when the ratio of the track segments with accurate results to all the track segments is lower than the target ratio, determine the moving direction of the vehicle in each track segment; according to each track segment Each of the trajectory segments is filtered according to the moving direction of the trajectory segments to filter out the trajectory segments with different moving directions; the remaining trajectory segments after filtering are fused to generate a vehicle tracking result. Specifically, the movement direction of the vehicle in the trajectory is calculated according to the timestamp in each trajectory segment. If the movement direction of the vehicle in the trajectory segment is significantly different from the movement direction of the vehicle in the corresponding trajectory segment at the next moment, Then filter out the tracklet based on the motion direction. Specifically, for example, if the target ratio is 90%, if the ratio of the track segments with accurate results to all the track segments is less than 90%, then the moving direction of the vehicle in each track segment is determined; according to each track segment The moving direction of the trajectory segments filters each of the trajectory segments to filter out the trajectory segments with different moving directions; the remaining trajectory segments after filtering are fused to generate a vehicle tracking result.

In some examples of this embodiment, determining the moving direction of the vehicle in each of the trajectory segments includes: acquiring each timestamp included in each of the trajectory segments; according to the time in each of the trajectory segments The sequence of the stamps and the position of the vehicle corresponding to each of the time stamps determine the moving direction of the vehicle in each of the track segments. For example: car No. 10, in a track segment, the first moment is at position 1, the second moment is at position 2, and the second moment is the moment after the first moment, then the tracking position can be calculated to calculate the number of vehicle No. 10. Vehicle movement direction, and use it as the movement direction of the trajectory segment;

In some examples of this embodiment, filtering each of the trajectory segments according to the moving direction of each of the trajectory segments, so as to filter out the trajectory segments with different moving directions, includes: filtering a plurality of the video streams The vehicles contained in the vehicle are taken as target vehicles in turn, and the movement direction of the target vehicle is determined; the trajectory segment corresponding to the target vehicle is determined, and the movement direction of the vehicle in the determined trajectory segment and the movement of the target vehicle When the difference between the motion direction of the vehicle in any of the determined trajectory segments and the motion direction of the target vehicle reaches the target difference value, remove the trajectory segment that reaches the target difference value to filter out motion the trajectory segments in different directions. Specifically, for example, for the No. 10 car, in a trajectory segment, the first moment is at position 1, the second moment is at position 2, and the second moment is a moment after the first moment, then the tracking position can be calculated to calculate the 10th car. The vehicle movement direction of the vehicle No. 1 is taken as the movement direction of the trajectory segment; the vehicle No. 10 is in another trajectory segment, the third moment is at position 3, the fourth moment is at position 4, and the third moment is the second The moment after the moment, and the fourth moment is the moment after the third moment, at this time, the vehicle movement direction of the No. 10 car can be calculated and used as the movement direction of the trajectory segment. The movement directions are compared. If the movement directions of the two trajectory segments are significantly different, the trajectory segments from the first moment to the second moment are filtered out. If the movement directions of the two trajectory segments are slightly different, the two trajectory segments are retained;

In some examples, if the trajectory segment is short, it cannot form an effective direction of motion. Therefore, the short trajectory segment can be discarded. Specifically, statistics can be performed according to the timestamp of the trajectory segment. If the segment spanning time does not exceed the threshold, the track segment is considered to be short, and the track segment is discarded. The threshold corresponding to the track segment can be flexibly set according to the actual usage scenario. For example, short tracklets are filtered out according to the threshold. The threshold is set to 180 frames, and since a frame is sampled every 2 seconds, the threshold is 6 minutes.

In some examples of this embodiment, fusing the remaining track segments after filtering to generate a vehicle tracking result includes: merging the remaining track segments after filtering and having the same license plate recognition result , obtain at least two merged track segments; obtain the similarity between all the merged track segments, and perform track fusion according to the similarity among all the merged track segments to generate a vehicle tracking result. Specifically, the tracklets with the same license plate are merged to obtain at least two merged track segments, and then the similarity of each merged track segment is calculated respectively, wherein calculating the similarity of each merged track segment includes but is not limited to:

First, average the vehicle features of all frames contained in each merged trajectory segment to obtain the vehicle characteristics of the merged trajectory segment, that is, a 2048-dimensional vector; it is understandable that a trajectory segment spans many frames and contains at least one Vehicle target, each vehicle target has a ReID vehicle feature, add all the ReID features in a frame to obtain the frame feature, and average all the frame features to obtain the vehicle feature of the trajectory segment;

Then, the cosine similarity between the vectors of each combined trajectory segment is calculated to obtain a similarity matrix between each combined trajectory segment, and the similarity of each combined trajectory segment is obtained through the similarity matrix.

Performing trajectory fusion according to the similarity between all the merged trajectory segments to generate the vehicle tracking result includes: firstly performing k-means clustering between adjacent cameras in the similarity matrix obtained by the above method, and obtaining a similar camera after merging , and then perform K-means clustering among all cameras to obtain the fused multi-camera trajectory, and use the fused multi-camera trajectory as the vehicle tracking result.

The vehicle tracking method provided in this embodiment obtains the bounding box information of the vehicle in each of the video streams by performing vehicle detection processing on a plurality of video streams, and performs license plate recognition processing on the plurality of video streams to obtain each of the video streams. The license plate recognition result of the vehicle in the video stream; based on the bounding box information of the vehicle in each video stream and the license plate recognition result, obtain the trajectory segment corresponding to each of the video streams; according to each of the trajectory The license plate recognition result contained in the segment is fused to generate a vehicle tracking result, a trajectory segment is obtained based on the bounding box information and the license plate recognition result, and multiple trajectory segments are merged based on the trajectory segment to improve the vehicle. Accuracy and robustness of tracking.

In order to better understand the present invention, this embodiment provides a more specific example to illustrate the present invention. This example provides a vehicle tracking method. The vehicle tracking method is applied to a vehicle tracking system. The system uses scattered in industrial parks. The surveillance camera equipment automatically tracks each running vehicle.

Specifically, first prepare each dataset and each model:

Self-owned data set: Image and video data collected by multiple cameras arranged in the industrial park are used. The self-owned data set includes: 500 videos, and the image data scale is about 2000-10000 pieces. The scene of the vehicle tracking method here is an industrial park, where there are many trucks. Therefore, the self-owned data set is a variety of vehicle images (cars, trucks, trucks, etc.) captured by cameras arranged in the industrial park, and the scale is small. , 2000-10000, and then manually label the own data set by relevant personnel, including the license plate label.

Then, pre-train the initial vehicle detection model based on YOLOX on the open source MS-COCO target detection data set, and fine-tune it on its own data set to fine-tune the vehicle detector model. The purpose of fine-tuning here is to train a very good vehicle detection model. A vehicle detection model that is well adapted to the industrial park can accurately detect various vehicles entering and leaving the industrial park.

Pre-train the initial vehicle re-identification model based on the transformer on the open source VeRi data set, and fine-tune it on the own data set to obtain the vehicle ReID model (vehicle re-identification model), the purpose is based on the appearance of the vehicle. identified the same vehicle. For example, there are 4 cameras at an intersection that capture the same car from different angles, and the system needs to identify the same vehicle id in these 4 images. The purpose of fine-tuning here is to train a good vehicle ReID model that can adapt to the industrial park scene, and can accurately retrieve the same vehicle between different cameras.

The single-camera vehicle tracking model (Deep-SORT model) is fine-tuned on the own dataset to obtain the single-camera vehicle tracking trajectory. The purpose of fine-tuning here is to obtain a single-camera tracking model that is more suitable for the industrial park scene.

Train a ResNet50-based license plate recognition model on our own dataset.

As shown in FIG. 2 , FIG. 2 is a basic flow chart of a vehicle tracking method. The flow of the vehicle tracking method includes two branches, a solid line and a dashed line. The dotted line is the commonly used branch based on license plate recognition, and the solid line is the branch based on ReID features.

First, the video streams of multiple cameras distributed in the park are obtained, and after sampling one frame per second, they are input to the vehicle detection model and the license plate recognition model. A vehicle detector model that outputs BBox information for each vehicle. The license plate recognition model outputs the license plate recognition results for each vehicle. The output of the vehicle detection model actually contains three branches, one is to output the position of the predicted target frame, that is, the BBox bounding box information (LOC) of each vehicle, and the other is to output whether the target is the foreground (real target) or the background ( Invalid target) (Obj), a probability of outputting what kind of target the target is (in the vehicle detector, that is, the probability that the target is a vehicle, CLS), (the detection confidence score represents the detected target is a real vehicle The probability of , equal to OBJ*CLS, can be used to filter the detection results with low reliability, in some examples, remove the results with a reliability lower than 0.35)

Then, the vehicle target detection result obtained in the previous step is cut and sent to the vehicle ReID model, and the vehicle ReID model outputs the ReID feature of each vehicle, and the feature dimension is 2048.

Then, the obtained vehicle BBox information, ReID features and license plate recognition results are input into the single-camera vehicle tracking model based on deep-SORT, and single-camera tracking tracklets (track segments) are obtained. The tracklets contain vehicle ID, frame timestamp (timestamp). ), BBox, ReID features, and license plate recognition results.

Finally, the multi-camera tracking trajectory fusion is performed according to the trajectory segment. The multi-camera tracking trajectory fusion is divided into the following two cases:

First, if more than 90% of the tracklets contain accurate license plate recognition results, enter the dotted path and merge multi-camera tracklets directly based on the license plate, that is, merge tracklets with the same license plate (the remaining less than 10% do not contain accurate license plates) The tracklets of information are discarded to obtain multi-camera tracking trajectories.

Second, if the tracklets contain less license plate information, enter the track fusion as shown in Figure 3.

Track fusion, first calculate the moving direction of the vehicle in the track according to the timestamp. If the moving direction is very different, filter out the tracklet based on the moving direction. Depending on the threshold, shorter tracklets are also discarded. Specifically, for each tracklet, for each track, and each track contains vehicle ID, timestamp, BBOX, etc., for example, the No. 10 car, the current moment is at position 1, and the next moment is at position 2, you can Tracking position to calculate the direction of vehicle movement) is currently in the preparation stage of multi-camera trajectory fusion, and the target result is still the result of single-camera tracking. Filter the trajectories output by the single-camera tracking. It is understandable that a track already contains the timestamp of the frame, and statistics are made according to the timestamp. If the track spans less than 6 minutes, the track is considered to be short and discarded.

After filtering, the tracklets with the same license plate are merged to obtain a merged tracklet.

Then calculate the similarity of the remaining merged tracklets. Specifically: First, average the ReID features of all frames contained in each tracklet to obtain the ReID feature of the tracklet, that is, a 2048-dimensional vector. Second, calculate the cosine similarity between the tracklet vectors, and obtain the similarity matrix between the tracklets.

It can be understood that the similarity of each tracklet is calculated. Since the tracklet already contains all the ReID features, the average of all the ReID features in the tracklet will get the feature of the tracklet, and the expression form is also a 2048-dimensional vector. Then, the similarity matrix between tracklets is calculated. The cospin similarity between two different tracklets T ₁ and T ₂ is calculated by the following formula, where f(T ₁ ) represents the feature of the track T ₁ and f(T ₂ ) represents the feature of the track T ₂ .

From this, the similarity matrix M between all n trajectories can be obtained:

In the similarity matrix obtained by calculating the similarity, k-means clustering between adjacent cameras is firstly performed to obtain the combined trajectory of similar cameras. Then, perform K-means clustering among all cameras on the merged trajectory results to obtain the fused multi-camera trajectory.

Continuing the above example, specifically, according to the similarity matrix in the previous step, first perform the hierarchical clustering (Agglomerative clustering) between adjacent cameras, and use the complete link method to set the first distance threshold to a value in the range of 0.4-0.6 ( The distance threshold can be flexibly set by the relevant personnel according to the actual personnel, preferably, the first distance threshold is 0.5) to obtain a preliminary fusion trajectory. Hierarchical clustering is a bottom-up iterative clustering algorithm. First, each trajectory sample is regarded as a cluster, and then clusters with high similarity are merged according to certain rules until all samples form a cluster. Or when a certain condition is reached (if the distance between the two clusters is less than the first distance threshold of 0.5, the two clusters will be merged into one), the algorithm ends; then the second distance threshold is set to 0.85-0.95 (the The second distance threshold is flexibly set by the relevant personnel according to the actual personnel. Preferably, the second distance threshold is 0.9) and then perform cross-camera hierarchical clustering to obtain the fused multi-camera trajectory. At this time, the distance threshold is 0.9.

The vehicle tracking method provided in this embodiment cooperates with each other through the branches of the solid line and the dashed line. If the license plate recognition effect is ideal, the multi-camera tracking can be completed directly; in rainy weather or occlusion, the license plate recognition effect is not ideal, and the ReID feature is used to merge the multi-camera trajectories to improve the accuracy and robustness of the tracking system.

Based on the same concept, this embodiment provides a vehicle tracking device. As shown in FIG. 4 , the vehicle tracking device includes:

A detection module, the inspection module is used to perform vehicle detection processing on a plurality of video streams, obtain the bounding box information of vehicles in each of the video streams, and perform license plate recognition processing on the plurality of video streams to obtain each of the video streams. License plate recognition results of vehicles in the video stream;

a determination module, configured to acquire a trajectory segment corresponding to each of the video streams based on the bounding box information of the vehicle in each of the video streams and the license plate recognition;

A fusion module, configured to fuse a plurality of the trajectory segments according to the license plate recognition result included in each of the trajectory segments to generate a vehicle tracking result.

It should be understood that each module of the vehicle tracking device provided in this embodiment can implement the various steps of the above-mentioned vehicle tracking method in combination to achieve the same technical effect as the above-mentioned vehicle tracking method, which is not repeated here.

As shown in FIG. 5 , an embodiment of the present application provides an electronic device, including a processor 111 , a communication interface 112 , a memory 113 and a communication bus 114 , wherein the processor 111 , the communication interface 112 , and the memory 113 are completed through the communication bus 114 communication with each other,

a memory 113 for storing computer programs;

In an embodiment of the present application, the processor 111 is configured to implement the vehicle tracking method provided by any one of the foregoing method embodiments when executing the program stored in the memory 113, including:

Perform vehicle detection processing on a plurality of video streams to obtain bounding box information of vehicles in each of the video streams, and perform license plate recognition processing on a plurality of the video streams to obtain license plate recognition results of vehicles in each of the video streams;

Obtaining a trajectory segment corresponding to each of the video streams based on the bounding box information of the vehicle in each of the video streams and the license plate recognition result;

According to the license plate recognition result contained in each of the trajectory segments, a plurality of the trajectory segments are fused to generate a vehicle tracking result.

Embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the vehicle tracking method provided by any one of the foregoing method embodiments.

It should be noted that, in this document, relational terms such as "first" and "second" etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these Any such actual relationship or sequence exists between entities or operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above descriptions are only specific embodiments of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims (10)

1. A vehicle tracking method, wherein the vehicle tracking method comprises: Perform vehicle detection processing on a plurality of video streams to obtain bounding box information of vehicles in each of the video streams, and perform license plate recognition processing on a plurality of the video streams to obtain license plate recognition results of vehicles in each of the video streams; Obtaining a trajectory segment corresponding to each of the video streams based on the bounding box information of the vehicle in each of the video streams and the license plate recognition result; According to the license plate recognition result contained in each of the trajectory segments, a plurality of the trajectory segments are fused to generate a vehicle tracking result. 2. The method according to claim 1, characterized in that, according to the license plate recognition result contained in each of the trajectory segments, a plurality of the trajectory segments are fused to generate a vehicle tracking result, comprising: determining whether the vehicle identification results contained in each of the trajectory segments are accurate; When the ratio of the track segments with accurate results to all the track segments is not lower than the target ratio, the multiple track segments are fused directly based on the license plate recognition result to generate a vehicle tracking result. 3. The method according to claim 1 or 2, wherein, according to the license plate recognition result contained in each of the trajectory segments, a plurality of the trajectory segments are fused to generate a vehicle tracking result, comprising: : determining whether the vehicle identification results contained in each of the trajectory segments are accurate; When the ratio of the track segments with accurate results to all the track segments is lower than the target ratio, determining the moving direction of the vehicle in each track segment; Filtering each of the trajectory segments according to the movement direction of each of the trajectory segments, so as to filter out the trajectory segments with different movement directions; The track segments remaining after filtering are fused to generate vehicle tracking results. 4. The method according to claim 3, wherein determining the moving direction of the vehicle in each of the trajectory segments comprises: obtain each timestamp contained within each of said trajectory segments; According to the sequence of the time stamps in each of the track segments and the position of the vehicle corresponding to each of the time stamps, the moving direction of the vehicle in each of the track segments is determined. 5. The method according to claim 4, wherein filtering each of the trajectory segments according to the movement direction of each of the trajectory segments to filter out the trajectory segments with different movement directions, comprising: Taking the vehicles included in the plurality of video streams as target vehicles in sequence, and determining the moving direction of the target vehicle; determining the trajectory segment corresponding to the target vehicle, and comparing the motion direction of the vehicle in the determined trajectory segment with the motion direction of the target vehicle; When the difference between the moving direction of the vehicle in any of the determined trajectory segments and the moving direction of the target vehicle reaches the target difference value, remove the trajectory segment that reaches the target difference value, so as to filter out all the moving directions with different moving directions. track segment. 6 . The method according to claim 5 , wherein the remaining track segments after filtering are fused to generate a vehicle tracking result, comprising: 6 . Merging the track segments remaining after filtering and having the same license plate recognition result to obtain at least two merged track segments; Obtain the similarity between all the merged trajectory segments, and perform trajectory fusion according to the similarity among all the merged trajectory segments to generate a vehicle tracking result. 7. The method according to any one of claims 1-2, 4-6, characterized in that, based on the bounding box information of the vehicle in each of the video streams and the license plate recognition, obtaining each of the said Track segments corresponding to the video stream, including: Perform vehicle re-identification processing on the bounding box information to obtain vehicle characteristics corresponding to each vehicle; The bounding box information, the vehicle feature and the license plate recognition result of the vehicles in each of the video streams are respectively fused to obtain a trajectory segment corresponding to each of the video streams. 8. A vehicle tracking device, wherein the vehicle tracking device comprises: A detection module, the inspection module is used to perform vehicle detection processing on a plurality of video streams, obtain the bounding box information of vehicles in each of the video streams, and perform license plate recognition processing on the plurality of video streams to obtain each of the video streams. License plate recognition results of vehicles in the video stream; a determination module, configured to acquire a trajectory segment corresponding to each of the video streams based on the bounding box information of the vehicle in each of the video streams and the license plate recognition; A fusion module, configured to fuse a plurality of the trajectory segments according to the license plate recognition result included in each of the trajectory segments to generate a vehicle tracking result. 9. An electronic device, characterized in that it comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; memory for storing computer programs; The processor is configured to implement the steps of the vehicle tracking method according to any one of claims 1-7 when executing the program stored in the memory. 10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the vehicle tracking method according to any one of claims 1-7 are implemented.

Images