Study on Nighttime Pedestrian Trajectory-Tracking from the Perspective of Driving Blind Spots
<p>Nighttime driver’s blind spot pedestrian-tracking technology route.</p> "> Figure 2
<p>Algorithm implementation flowchart.</p> "> Figure 3
<p>C2faster structural diagram. (<b>a</b>) FasterNet block, (<b>b</b>) C2f-faster.</p> "> Figure 4
<p>BiFormer attention mechanism structure diagram.</p> "> Figure 5
<p>CARAFE upsampling structure diagram.</p> "> Figure 6
<p>Dynamic detection head DyHead structure diagram.</p> "> Figure 7
<p>Improved YOLOP network structure diagram.</p> "> Figure 8
<p>ShuffleNetV2 structure diagram.</p> "> Figure 9
<p>DIoU schematic diagram.</p> "> Figure 10
<p>Improved DeepSORT structure flowchart.</p> "> Figure 11
<p>The lane line-detection results at night are presented. In Scene 1, the road at night is unobstructed and the lane lines are clear. In Scene 2, the road at night has obstructions. In Scene 3, the lane lines on the road at night are unclear.</p> "> Figure 12
<p>FBCD-YOLOP Training Process Results Diagram.</p> "> Figure 13
<p>The results of the loss during the training and validation process of the tracking algorithm.</p> "> Figure 14
<p>Nighttime pedestrian-tracking results. (<b>a</b>) A frame from the first video sequence showing the initial detection and tracking of pedestrians by the proposed algorithm. (<b>b</b>) The corresponding frame from the first video sequence where the IDS-tracking process is shown; the proposed algorithm accurately tracks pedestrian ID3 through the crowd, while the YOLOP-DeepSort algorithm exhibits ID switches (highlighted by orange circles). (<b>c</b>) A frame from the second video sequence showing the proposed algorithm’s detection of pedestrians with no ID changes or false detections. (<b>d</b>) The corresponding frame from the second video sequence where the YOLOP-DeepSort algorithm mistakenly identifies a tree trunk and a wall crack as pedestrians (highlighted by red circles), demonstrating the superiority of the proposed algorithm in avoiding false detections.</p> ">
Abstract
:1. Introduction
2. Detection and Tracking Method
2.1. Improving the YOLOP Model
2.1.1. Feature Extraction C2f-Faster
2.1.2. BiFormer Attention Mechanism
2.1.3. CARAFE Upsampling
2.1.4. Dynamic Detection Head DyHead
2.2. Improved DeepSORT Tracking Module
2.2.1. ShuffleNetV2 Lightweight
2.2.2. Improved IoU Matching Mechanism
3. Experiment Results and Analysis
3.1. Data Introduction
3.2. Experimental Platform
3.3. Evaluation Indicators
- (1)
- Evaluation Metrics for Object-Detection
- (2)
- Evaluation Metrics for Tracking Algorithm
3.4. Object-Detection Experiment and Result Analysis
3.4.1. Lane Line-Detection
3.4.2. Nighttime Pedestrian-Detection
3.5. Target Re-Identification Experiment and Result Analysis
4. Conclusions
- (1)
- This study proposed a nighttime pedestrian behavior-recognition and -tracking scheme based on an optimized YOLOP. By using the FBCD-YOLOP algorithm to detect lane lines and pedestrians at night, posture information and lane line positions were accurately captured. The improved DeepSORT algorithm was then employed for precise pedestrian-tracking, enabling multi-target behavior-recognition in the driver’s blind spot environment at night. Experimental results demonstrated that the FBCD-YOLOP model enhanced lane line detection accuracy by 5.1 percentage points, with an IoU improvement of 0.8% and a detection speed increase of 25 FPS. For nighttime pedestrian-detection, the model achieved a precision of 89.6%, outperforming YOLOv5, TDL-YOLO, and YOLOP. Additionally, the improved DeepSORT algorithm achieved a MOTA of 86.3%, showing enhanced real-time performance and tracking stability, with a reduced parameter count of 24.8% of the original, making it suitable for embedded device deployment and automotive safety applications.
- (2)
- In the future, we will place greater emphasis on practical use in autonomous driving platforms while exploring more diverse modalities, such as infrared imaging and radar, to enhance the effectiveness of nighttime pedestrian-detection and -tracking, as well as pedestrian safety.
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Farooq, M.S.; Khalid, H.; Arooj, A.; Umer, T.; Asghar, A.B.; Rasheed, J.; Shubair, R.M.; Yahyaoui, A.; Farooq, M.S.; Khalid, H.; et al. A conceptual multi-layer framework for the detection of nighttime pedestrian in autonomous vehicles using deep reinforcement learning. Entropy 2023, 25, 135. [Google Scholar] [CrossRef] [PubMed]
- Cao, J.; Song, C.; Peng, S.; Song, S.; Zhang, X.; Shao, Y.; Xiao, F. Pedestrian detection algorithm for intelligent vehicles in complex scenarios. Sensors 2020, 20, 3646. [Google Scholar] [CrossRef] [PubMed]
- Georgescu, M.I.; Barbalau, A.; Ionescu, R.T.; Khan, F.S.; Popescu, M.; Shah, M. Anomaly detection in video via self-supervised and multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021. [Google Scholar]
- Li, F.; Li, X.; Liu, Q.; Li, Z. Occlusion handling and multi-scale pedestrian detection based on deep learning: A review. IEEE Access 2022, 10, 19937–19957. [Google Scholar] [CrossRef]
- Liu, S.; Wang, S.; Liu, X.; Lin, C.-T.; Lv, Z. Fuzzy detection aided real-time and robust visual tracking under complex environments. IEEE Trans. Fuzzy Syst. 2020, 29, 90–102. [Google Scholar] [CrossRef]
- Akshatha, K.R.; Karunakar, A.K.; Shenoy, S.B.; Pai, A.K.; Nagaraj, N.H.; Rohatgi, S.S. Human detection in aerial thermal images using faster R-CNN and SSD algorithms. Electronics 2022, 11, 1151. [Google Scholar] [CrossRef]
- Avola, D.; Cinque, L.; Diko, A.; Fagioli, A.; Foresti, G.L.; Mecca, A.; Pannone, D.; Piciarelli, C. MS-Faster R-CNN: Multi-stream backbone for improved Faster R-CNN object detection and aerial tracking from UAV images. Remote Sens. 2021, 13, 1670. [Google Scholar] [CrossRef]
- Liu, Y.; Zhou, S.; Liu, X.; Hao, C.; Fan, B.; Tian, J. Unbiased Faster R-CNN for Single-source Domain Generalized Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 17–21 June 2024. [Google Scholar]
- Ren, Z.; Wang, S.; Zhang, Y. Weakly supervised machine learning. CAAI Trans. Intell. Technol. 2023, 8, 549–580. [Google Scholar] [CrossRef]
- Chen, Z.; Guo, H.; Yang, J.; Jiao, H.; Feng, Z.; Chen, L.; Gao, T. Fast vehicle detection algorithm in traffic scene based on improved SSD. Measurement 2022, 201, 111655. [Google Scholar] [CrossRef]
- Niu, Y.; Cheng, W.; Shi, C.; Fan, S. YOLOv8-CGRNet: A lightweight object detection network leveraging context guidance and deep residual learning. Electronics 2023, 13, 43. [Google Scholar] [CrossRef]
- Liu, H.; Duan, X.; Lou, H.; Gu, J.; Chen, H.; Bi, L. Improved GBS-YOLOv5 algorithm based on YOLOv5 applied to UAV intelligent traffic. Sci. Rep. 2023, 13, 9577. [Google Scholar] [CrossRef]
- Kumar, S.; Singh, S.K.; Varshney, S.; Singh, S.; Kumar, P.; Kim, B.-G.; Ra, I.-H. Fusion of deep sort and Yolov5 for effective vehicle detection and tracking scheme in real-time traffic management sustainable system. Sustainability 2023, 15, 16869. [Google Scholar] [CrossRef]
- Li, Y.; Li, S.; Du, H.; Chen, L.; Zhang, D.; Li, Y. YOLO-ACN: Focusing on small target and occluded object detection. IEEE Access 2020, 8, 227288–227303. [Google Scholar] [CrossRef]
- Bansal, M.; Goyal, A.; Choudhary, A. A comparative analysis of K-nearest neighbor, genetic, support vector machine, decision tree, and long short term memory algorithms in machine learning. Decis. Anal. J. 2022, 3, 100071. [Google Scholar] [CrossRef]
- Luo, W.; Xing, J.; Milan, A.; Zhang, X.; Liu, W.; Zhao, X.; Kim, T.-K. Multiple object tracking: A literature review. Artif. Intell. 2021, 293, 103448. [Google Scholar] [CrossRef]
- Cao, Z.; Zhang, Y.; Tian, R.; Ma, R.; Hu, X.; Coleman, S.; Kerr, D. Object-aware SLAM based on efficient quadric initialization and joint data association. IEEE Robot. Autom. Lett. 2022, 7, 9802–9809. [Google Scholar] [CrossRef]
- Dendorfer, P.; Ošep, A.; Milan, A.; Schindler, K.; Cremers, D.; Reid, I.; Roth, S.; Leal-Taixé, L. Motchallenge: A benchmark for single-camera multiple target tracking. Int. J. Comput. Vis. 2021, 129, 845–881. [Google Scholar] [CrossRef]
- Chen, X.; Jia, Y.; Tong, X.; Li, Z. Research on pedestrian detection and deepsort tracking in front of intelligent vehicle based on deep learning. Sustainability 2022, 14, 9281. [Google Scholar] [CrossRef]
- Charef, N.; Ben Mnaouer, A.; Aloqaily, M.; Bouachir, O.; Guizani, M. Artificial intelligence implication on energy sustainability in Internet of Things: A survey. Inf. Process. Manag. 2023, 60, 103212. [Google Scholar] [CrossRef]
- Razzok, M.; Badri, A.; El Mourabit, I.; Ruichek, Y.; Sahel, A. Pedestrian detection and tracking system based on Deep-SORT, YOLOv5, and new data association metrics. Information 2023, 14, 218. [Google Scholar] [CrossRef]
- Rasheed, M.T.; Shi, D.; Khan, H. A comprehensive experiment-based review of low-light image enhancement methods and benchmarking low-light image quality assessment. Signal Process. 2023, 204, 108821. [Google Scholar] [CrossRef]
- Ngeni, F.; Mwakalonge, J.; Siuhi, S. Solving traffic data occlusion problems in computer vision algorithms using DeepSORT and quantum computing. J. Traffic Transp. Eng. 2024, 11, 1–15. [Google Scholar] [CrossRef]
- Masoud, K.; Maihami, V. A review on Kalman filter models. Arch. Comput. Methods Eng. 2023, 30, 727–747. [Google Scholar]
- Cossu, M.; Berta, R.; Forneris, L.; Fresta, M.; Lazzaroni, L.; Sauvaget, J.L.; Bellotti, F. YoloP-Based Pre-processing for Driving Scenario Detection. In International Conference on Applications in Electronics Pervading Industry, Environment and Society, Genoa, Italy, 28–28 September 2023; Springer Nature Switzerland: Cham, Switzerland, 2023. [Google Scholar]
- Li, A.; Zhang, Z.; Sun, S.; Feng, M.; Wu, C. MultiNet-GS: Structured Road Perception Model Based on Multi-Task Convolutional Neural Network. Electronics 2023, 12, 3994. [Google Scholar] [CrossRef]
- Lei, Y.; Pan, D.; Feng, Z.; Qian, J. Lightweight YOLOv5s human Ear recognition based on MobileNetV3 and ghostnet. Appl. Sci. 2023, 13, 6667. [Google Scholar] [CrossRef]
- Yang, F.; Huang, L.; Tan, X.; Yuan, Y. FasterNet-SSD: A small object detection method based on SSD model. Signal Image Video Process. 2024, 18, 173–180. [Google Scholar] [CrossRef]
- Zhu, L.; Wang, X.; Ke, Z.; Zhang, W.; Lau, R.W. Biformer: Vision transformer with bi-level routing attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023. [Google Scholar]
- Sun, Z. CBE-Net: A dedicated target detection algorithm for small vehicle. In Proceedings of the 2024 3rd International Symposium on Control Engineering and Robotics, Changsha, China, 24–26 May 2024. [Google Scholar]
- Wang, J.; Li, Y.; Wang, J.; Li, Y. An Underwater Dense Small Object Detection Model Based on YOLOv5-CFDSDSE. Electronics 2023, 12, 3231. [Google Scholar] [CrossRef]
- Du, Y.; Zhao, Z.; Song, Y.; Zhao, Y.; Su, F.; Gong, T.; Meng, H. Strongsort: Make deepsort great again. IEEE Trans. Multimed. 2023, 25, 8725–8737. [Google Scholar] [CrossRef]
- Gregory, F.W. Kalman filter. In Computer Vision: A Reference Guide; Springer International Publishing: Cham, Switzerland, 2021; pp. 721–723. [Google Scholar]
- Cengil, E.; Çınar, A.; Yıldırım, M. An efficient and fast lightweight-model with ShuffleNetv2 based on YOLOv5 for detection of hardhat-wearing. Rev. Comput. Eng. Stud 2022, 9, 116–123. [Google Scholar] [CrossRef]
- Atliha, V.; Sesok, D. Comparison of VGG and ResNet used as Encoders for Image Captioning. In Proceedings of the 2020 IEEE Open Conference of Electrical, Electronic and Information Sciences (eStream), Vilnius, Lithuania, 30 April 2020; IEEE: Piscataway, NJ, USA, 2020. [Google Scholar]
- You, L.; Chen, Y.; Xiao, C.; Sun, C.; Li, R. Multi-Object Vehicle Detection and Tracking Algorithm Based on Improved YOLOv8 and ByteTrack. Electronics 2024, 13, 3033. [Google Scholar] [CrossRef]
- Wang, X.; Hu, X.; Liu, P.; Tang, R. A Person Re-Identification Method Based on Multi-Branch Feature Fusion. Appl. Sci. 2023, 13, 11707. [Google Scholar] [CrossRef]
- Ortiz Castelló, V.; Salvador Igual, I.; del Tejo Catalá, O.; Perez-Cortes, J.C. High-profile vru detection on resource-constrained hardware using yolov3/v4 on bdd100k. J. Imaging 2020, 6, 142. [Google Scholar] [CrossRef] [PubMed]
Dataset | Total | Class | Training Set | Validation Set | Test Set |
---|---|---|---|---|---|
Market 1501 | 10,742 | 1 | 7584 | 2125 | 1033 |
BDD100K | 10,650 | 1 | 7500 | 2100 | 1050 |
Lighting Conditions | Total | Class | Training Set | Validation Set | Test Set |
---|---|---|---|---|---|
Complete Darkness | 7100 | 1 | 5028 | 1420 | 652 |
Low Light | 7192 | 1 | 5028 | 1450 | 714 |
Illuminated | 7100 | 1 | 5028 | 1355 | 717 |
Experimental Environment | |
---|---|
CPU | Intel(R)Core(TM)i7-12700H (Santa Clara, CA, USA) |
Memory | 16 GB |
Hard Drive | SSD 500 GB |
Graphics Card | RTX3060 12 GB |
OS | Win11 |
Python | 3.8 |
CUDA | 11.2 |
Pytorch | 1.12.1 |
Model | Acc% | IoU% | FPS |
---|---|---|---|
SCNN | 34.7 | 15.9 | 19 |
ENet-SAD | 37.6 | 16.0 | 50 |
YOLOP | 70.5 | 26.4 | 41 |
TDL-YOLO | 72.3 | 26.5 | 37 |
Ours | 75.6 | 27.2 | 66 |
Model | Precison/% | Recall/% | mAP/% | FPS | Params/106 |
---|---|---|---|---|---|
Faster R-CNN | 86.1 | 88.9 | 85.7 | 15 | 27.1 |
YOLOv3 | 80.1 | 81.3 | 79.5 | 63 | 51.5 |
YOLOv5s | 88.3 | 91.6 ↑ | 87.3 | 121 ↑ | 10.2 ↓ |
TDL-YOLO | 88.7 | 89.5 | 87.7 | 37 | 78.4 |
YOLOP | 85.8 | 87.3 | 84.5 | 41 | 47.4 |
Ours | 89.6 ↑ | 91.3 | 88.1 ↑ | 66 | 30.5 |
Lighting Environment | Precision (%) | Recall (%) | Miss Rate (%) | Threshold |
---|---|---|---|---|
Complete darkness | 85.2 | 87.3 | 14.5 | 0.6 |
Low light | 88.7 | 90.5 | 10.8 | 0.5 |
Illuminated | 89.6 | 91.3 | 7.2 | 0.4 |
Strategy 1 | Strategy 2 | Strategy 3 | Strategy 4 | Precision/% | mAP/% | Recall/% | FPS |
---|---|---|---|---|---|---|---|
85.8 | 84.5 | 87.3 | 46 | ||||
√ | 86.2 | 86.4 | 89.5 | 83 | |||
√ | 87.6 | 87.1 | 90.2 | 44 | |||
√ | √ | 87.9 | 86.4 | 89.9 | 47 | ||
√ | √ | √ | √ | 89.6 | 88.1 | 91.3 | 66 |
Comparison | t-Statistic | p-Value | Significant Difference (1 for Yes, 0 for No) |
---|---|---|---|
Strategy 1 vs. Strategy 2 | −2.93 | 0.0136 | 1 |
Strategy 1 vs. Strategy 3 | −12.62 | 0.00001 | 1 |
Strategy 1 vs. Strategy 4 | −13.37 | 0.00001 | 1 |
Strategy 1 vs. All | −23.18 | 0.000001 | 1 |
Strategy 2 vs. Strategy 3 | −8.89 | 0.0001 | 1 |
Strategy 2 vs. Strategy 4 | −9.43 | 0.0001 | 1 |
Strategy 2 vs. All | −16.89 | 0.000001 | 1 |
Strategy 3 vs. Strategy 4 | −2.34 | 0.0365 | 1 |
Strategy 3 vs. All | −7.65 | 0.0002 | 1 |
Strategy 4 vs. All | −8.88 | 0.0001 | 1 |
Model | Precision (Train)/% | Precision (Val)/% | Model Size/106 |
---|---|---|---|
DeepSORT | 85.1 | 75.6 | 43.8 |
DeepSORT-SNV2 | 87.9 | 78.2 | 2.7 |
Algorithm | MOTA/% | MOTP/% | IDS | FPS |
---|---|---|---|---|
YOLOP-DeepSort | 80.7 | 79.4 | 10 | 24 |
Modified YOLOP-DeepSort | 85.4 | 83.3 | 7 | 44 |
YOLOP-Modified DeepSort | 81.2 | 79.3 | 8 | 47 |
Ours(YOLOP-DeepSort) | 86.3 | 84.9 | 5 | 59 |
MOT Algorithm | MOTA/% | MOTP/% | IDS | FPS |
---|---|---|---|---|
Byte Track | 83.4 | 79.7 | 11 | 27 |
Strong SORT | 85.9 | 86.3 | 17 | 23 |
Ours | 86.3 | 84.9 | 5 | 59 |
Model | MOTA/% | MOTP/% | FPS |
---|---|---|---|
Faster R-CNN | 63.6 | 78.8 | 27 |
RT-DETR | 66.1 | 79.5 | 31 |
YOLOv3 | 70.2 | 80.2 | 35 |
YOLOv5s | 83.4 | 84.1 | 76 |
YOLOv7 | 85.9 | 84.3 | 72 |
YOLOP | 80.7 | 79.3 | 36 |
Ours | 86.3 | 84.9 | 59 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhao, W.; Ren, C.; Tan, A. Study on Nighttime Pedestrian Trajectory-Tracking from the Perspective of Driving Blind Spots. Electronics 2024, 13, 3460. https://doi.org/10.3390/electronics13173460
Zhao W, Ren C, Tan A. Study on Nighttime Pedestrian Trajectory-Tracking from the Perspective of Driving Blind Spots. Electronics. 2024; 13(17):3460. https://doi.org/10.3390/electronics13173460
Chicago/Turabian StyleZhao, Wei, Congcong Ren, and Ao Tan. 2024. "Study on Nighttime Pedestrian Trajectory-Tracking from the Perspective of Driving Blind Spots" Electronics 13, no. 17: 3460. https://doi.org/10.3390/electronics13173460