On-Board Smartphone-Based Road Hazard Detection with Cloud-Based Fusion
"> Figure 1
<p>Road hazard detection system framework.</p> "> Figure 2
<p>BeamNG simulation motion data generation framework.</p> "> Figure 3
<p>BeamNG simulation environment and one example of potholes.</p> "> Figure 4
<p>LSTM architecture for deep-learning-based road hazard detection.</p> "> Figure 5
<p>Cloud -based fusion approach.</p> "> Figure 6
<p>Vehicle motion data for road event hazards.</p> "> Figure 7
<p>Vehicle motion data for road defect hazards.</p> "> Figure 8
<p>Lateral acceleration motion data with and without filtering.</p> "> Figure 9
<p>LSTM training accuracy and loss for simulation data only—Test 1.</p> "> Figure 10
<p>LSTM training accuracy and loss for separated simulated and real data—Test 2.</p> "> Figure 11
<p>LSTM training accuracy and loss for simulation and real mixed data—Test 3.</p> "> Figure 12
<p>LSTM-based confusion matrix for Test 1 (simulation only) with Kalman-filtered data.</p> "> Figure 13
<p>LSTM-based confusion matrix for Test 2 (separate simulated and real data) with low-pass-filtered data.</p> "> Figure 14
<p>LSTM-based confusion matrix for Test 3 (mixed simulated and real data) with low-pass-filtered data.</p> "> Figure 15
<p>Road hazard representation on web UI.</p> "> Figure 16
<p>Threshold-based confusion matrix for Test 1 (simulation only).</p> "> Figure 17
<p>Threshold-based confusion matrix for Test 3 (real data only).</p> ">
Abstract
:1. Introduction
2. Materials and Methods
2.1. System Framework
2.2. Data Acquisition
2.3. Data Processing
2.4. Deep-Learning-Based Road Hazard Detection Model
2.5. Heterogeneous Training Methods
2.5.1. Test 1 (Simulation Only)
2.5.2. Test 2 (Simulation and Real, Separate)
2.5.3. Test 3 (Simulation and Real Mixed)
2.6. Cloud-Based Fusion
Algorithm 1 Clustering Algorithm |
|
2.7. Threshold-Based Road Hazard Detection Model
3. Experimental Results
3.1. Experimental Data Representation and Data Processing
3.1.1. Data Representation
3.1.2. Data Processing
3.2. Experimental Results and Analysis
3.2.1. LSTM Model Training Results
3.2.2. LSTM Model Testing Results
3.2.3. Cloud-Based Fusion Results
3.2.4. Threshold-Based Model Testing Results
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Available online: https://www.nhtsa.gov/press-releases/early-estimate-2021-traffic-fatalities (accessed on 10 January 2023).
- González, A.; O’brien, E.J.; Li, Y.-Y.; Cashell, K. The use of vehicle acceleration measurements to estimate road roughness. Veh. Syst. Dyn. 2008, 46, 483–499. [Google Scholar] [CrossRef]
- Chen, K.; Lu, M.; Fan, X.; Wei, M.; Wu, J. Road condition monitoring using on-board Three-axis Accelerometer and GPS Sensor. In Proceedings of the 2011 6th International ICST Conference on Communications and Networking in China (CHINACOM), Harbin, China, 17–19 August 2011; pp. 1032–1037. [Google Scholar] [CrossRef]
- Lei, T.; Mohamed, A.A.; Claudel, C. An IMU-based traffic and road condition monitoring system. HardwareX 2018, 4, e00045. [Google Scholar] [CrossRef]
- Li, Y.; Liu, C.; Shen, Y.; Cao, J.; Yu, S.; Du, Y. RoadID: A Dedicated Deep Convolutional Neural Network for Multipavement Distress Detection. J. Transp. Eng. Part B Pavements 2021, 147, 04021057. [Google Scholar] [CrossRef]
- Tsai, Y.C.; Kaul, V.; Mersereau, R.M. Critical assessment of pavement distress segmentation methods. J. Transp. Eng. 2010, 136, 11–19. [Google Scholar] [CrossRef]
- Llopis-Castello, D.; Paredes, R.; Parreno-Lara, M.; Garcia-Segura, T.; Pellicer, E. Automatic Classification and Quantification of Basic Distresses on Urban Flexible Pavement through Convolutional Neural Networks. J. Transp. Eng. Part B Pavements 2021, 147, 04021063. [Google Scholar] [CrossRef]
- Alipour, M.; Harris, D.K.; Miller, G.R. Robust pixel-level crack detection using deep fully convolutional neural networks. J. Comput. Civ. Eng. 2019, 33, 04019040. [Google Scholar] [CrossRef]
- Sattar, S.; Li, S.; Chapman, M. Road surface monitoring using smartphone sensors: A review. Sensors 2018, 18, 3845. [Google Scholar] [CrossRef] [PubMed]
- Varona, B.; Monteserin, A.; Teyseyre, A. A deep learning approach to automatic road surface monitoring and pothole detection. Pers. Ubiquitous Comput. 2020, 24, 519–534. [Google Scholar] [CrossRef]
- Chatterjee, A.; Tsai, Y.C. Training and testing of smartphone-based pavement condition estimation models using 3d pavement data. J. Comput. Civ. Eng. 2020, 34, 04020043. [Google Scholar] [CrossRef]
- Ramesh, A.; Nikam, D.; Balachandran, V.N.; Guo, L.; Wang, R.; Hu, L.; Comert, G.; Jia, Y. Cloud-Based Collaborative Road-Damage Monitoring with Deep Learning and Smartphones. Sustainability 2022, 14, 8682. [Google Scholar] [CrossRef]
- Ameddah, M.A.; Das, B.; Almhana, J. Cloud-Assisted Real-Time Road Condition Monitoring System for Vehicles. In Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, United Arab Emirates, 9–13 December 2018; pp. 1–6. [Google Scholar] [CrossRef]
- Yuan, Y.; Islam, M.S.; Yuan, Y.; Wang, S.; Baker, T.; Kolbe, L.M. EcRD: Edge-Cloud Computing Framework for Smart Road Damage Detection and Warning. IEEE Internet Things J. 2021, 8, 12734–12747. [Google Scholar] [CrossRef]
- Pham, V.; Pham, C.; Dang, T. Road damage detection and classification with detectron2 and faster r-cnn. In Proceedings of the 2020 IEEE International Conference on Big Data (Big Data), Atlanta, GA, USA, 10–13 December 2020; pp. 5592–5601. [Google Scholar]
- Mahjoub, S.; Chrifi-Alaoui, L.; Marhic, B.; Delahoche, L.; Masson, J.-B.; Derbel, N. Prediction of energy consumption based on LSTM Artificial Neural Network. In Proceedings of the 2022 19th International Multi-Conference on Systems, Signals and Devices (SSD), Sétif, Algeria, 6–10 May 2022; pp. 521–526. [Google Scholar] [CrossRef]
- Kapoor, A.; Rastogi, V.; Kashyap, N. Forecasting Daily Close Prices of Stock Indices using LSTM. In Proceedings of the 2020 2nd International Conference on Advances in Computing, Communication Control and Networking (ICACCCN), Greater Noida, India, 18–19 December 2020; pp. 10–14. [Google Scholar] [CrossRef]
- Ma, M.; Liu, C.; Wei, R.; Liang, B.; Dai, J. Predicting machine’s performance record using the stacked long short-term memory (LSTM) neural networks. J. Appl. Clin. Med. Phys. 2022, 23, e13558. [Google Scholar] [CrossRef] [PubMed]
- Poh, S.-C.; Tan, Y.-F.; Guo, X.; Cheong, S.-N.; Ooi, C.-P.; Tan, W.-H. LSTM and HMM Comparison for Home Activity Anomaly Detection. In Proceedings of the 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chengdu, China, 15–17 March 2019; pp. 1564–1568. [Google Scholar] [CrossRef]
- Maul, P.; Mueller, M.; Enkler, F.; Pigova, E.; Fischer, T.; Stamatogiannakis, L. BeamNG.tech Technical Paper. Available online: https://beamng.tech/blog/2021-06-21-beamng-tech-whitepaper/bng_technical_paper.pdf (accessed on 10 January 2023).
- Rio, A.; Alfian, M.; Sunardi, S. Noise Reduction in the Accelerometer and Gyroscope Sensor with the Kalman Filter Algorithm. J. Robot. Control (JRC) 2020, 2, 180–189. [Google Scholar] [CrossRef]
- Bondan, S.; Kitasuka, T.; Aritsugi, M. Vehicle Vibration Error Compensation on IMU-accelerometer Sensor Using Adaptive Filter and Low-pass Filter Approaches. J. Inf. Process. 2019, 27, 33–40. [Google Scholar]
- Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef] [PubMed]
- Jain, A.K.; Murty, M.N.; Flynn, P.J. Data clustering: A review. ACM Comput. Surv. 1999, 31, 264–323. [Google Scholar] [CrossRef]
- Na, S.; Xumin, L.; Yong, G. Research on k-means Clustering Algorithm: An Improved k-means Clustering Algorithm. In Proceedings of the 2010 Third International Symposium on Intelligent Information Technology and Security Informatics, Ji’an, China, 2–4 April 2010; pp. 63–67. [Google Scholar] [CrossRef]
- Nazeer, K.A.A.; Sebastian, M.P. Improving the Accuracy and Efficiency of the k-means Clustering Algorithm. In Proceedings of the World Congress on Engineering 2009 Vol I WCE 2009, London, UK, 1–3 July 2009. [Google Scholar]
Hazard Type | Simulation | Real-World | Total |
---|---|---|---|
No hazard | 1091 | 94 | 1185 |
Road defect hazard | 217 | 278 | 495 |
Road event hazard | 700 | 378 | 1078 |
Total | 2008 | 750 | 2758 |
Parameter Name | Parameter Used |
---|---|
Number of features | 3 |
Number of time steps | 80 |
Number of training epochs | 15 |
Optimizer | Adam |
Batch size | 512 |
Learning rate | 0.0025 |
Loss regularization | L2 loss 0.0015 |
Training Data | Testing Data | Filter Type | Training Accuracy | Testing Accuracy |
---|---|---|---|---|
Simulation | Simulation | Kalman | 95.5% | 97.8% |
Simulation | Simulation | Low-pass | 94.5% | 97.3% |
Training Data | Testing Data | Filter Type | Training Accuracy | Testing Accuracy |
---|---|---|---|---|
Simulation | Real | Kalman | 96.1% | 75.6% |
Simulation | Real | Low-pass | 95.2% | 79.7% |
Training Data | Testing Data | Filter Type | Training Accuracy | Testing Accuracy |
---|---|---|---|---|
Simulation + Real | Real | Kalman | 95.6% | 89.6% |
Simulation + Real | Real | Low-pass | 94.6% | 89.0% |
Hazard Type | Precision | Recall | F1 | Total Count |
---|---|---|---|---|
No damage | 0.99 | 0.97 | 0.98 | 418 |
Road Defect Hazard | 0.99 | 1 | 0.99 | 83 |
Road Event Hazard | 0.96 | 0.98 | 0.97 | 271 |
Hazard Type | Precision | Recall | F1 | Total Count |
---|---|---|---|---|
No damage | 0.6 | 0.43 | 0.5 | 131 |
Road Defect Hazard | 0.6 | 0.89 | 0.72 | 186 |
Road Event Hazard | 0.99 | 0.87 | 0.93 | 433 |
Hazard Type | Precision | Recall | F1 | Total Count |
---|---|---|---|---|
No damage | 0.64 | 0.65 | 0.64 | 65 |
Road Defect Hazard | 0.83 | 0.92 | 0.87 | 188 |
Road Event Hazard | 1 | 0.93 | 0.96 | 294 |
Cluster ID | Latitude | Longitude | Hazard Type | Total Count | True Count | Accuracy before Fusion | Accuracy after Fusion |
---|---|---|---|---|---|---|---|
0 | 34.8003 | −82.3274 | 11 | 25 | 23 | 92% | 100% |
1 | 34.7694 | −82.3954 | 12 | 25 | 25 | 100% | 100% |
2 | 34.7767 | −82.3075 | 12 | 25 | 24 | 96% | 100% |
2 | 34.7767 | −82.3075 | 11 | 25 | 23 | 92% | 100% |
3 | 34.7344 | −82.3744 | 11 | 25 | 20 | 80% | 100% |
4 | 34.7521 | −82.2983 | 12 | 25 | 23 | 92% | 100% |
5 | 34.7930 | −82.3013 | 11 | 25 | 24 | 96% | 100% |
6 | 34.8166 | −82.3215 | 12 | 25 | 25 | 100% | 100% |
7 | 34.7896 | −82.3245 | 12 | 25 | 25 | 100% | 100% |
8 | 34.7813 | −82.3109 | 11 | 25 | 24 | 96% | 100% |
Tests | Hazard Type | Accuracy | Precision | Recall | F1 | Total Count |
---|---|---|---|---|---|---|
Test 1 | No damage | 0.81 | 0.96 | 0.88 | 411 | |
Simulation | Road defect hazard | 82.25% | 0.65 | 0.79 | 0.71 | 84 |
data only | Road event hazard | 0.95 | 0.62 | 0.75 | 277 | |
Test 3 | No damage | 0.37 | 0.74 | 0.49 | 94 | |
Real-world | Road defect hazard | 62.93% | 0.81 | 0.5 | 0.62 | 278 |
data only | Road event hazard | 0.67 | 0.69 | 0.68 | 378 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bhosale, M.; Guo, L.; Comert, G.; Jia, Y. On-Board Smartphone-Based Road Hazard Detection with Cloud-Based Fusion. Vehicles 2023, 5, 565-582. https://doi.org/10.3390/vehicles5020031
Bhosale M, Guo L, Comert G, Jia Y. On-Board Smartphone-Based Road Hazard Detection with Cloud-Based Fusion. Vehicles. 2023; 5(2):565-582. https://doi.org/10.3390/vehicles5020031
Chicago/Turabian StyleBhosale, Mayuresh, Longxiang Guo, Gurcan Comert, and Yunyi Jia. 2023. "On-Board Smartphone-Based Road Hazard Detection with Cloud-Based Fusion" Vehicles 5, no. 2: 565-582. https://doi.org/10.3390/vehicles5020031