Application of Machine Learning Techniques for Predicting Potential Vehicle-to-Pedestrian Collisions in Virtual Reality Scenarios
<p>Methodology main scheme of the pedestrian behavior modeling procedure.</p> "> Figure 2
<p>A 3D reconstruction of collision scenarios. From left to right: SC1, SC2, and SC3 (Madrid, Spain).</p> "> Figure 3
<p>Gender distribution of the sample of users that performed the experimental VR session (by age group).</p> "> Figure 4
<p>A comparison of the adjective ratings, acceptability scores, and school grading scales, in relation to the average SUS score [<a href="#B34-applsci-12-11364" class="html-bibr">34</a>].</p> "> Figure 5
<p>Trajectory and speed plots, and positioning of events for a test simulation.</p> "> Figure 6
<p>Geometric definition of MAA for Av Machupichu (<b>a</b>) and Hermanos García Noblejas (<b>b</b>).</p> "> Figure 7
<p>VR scenario for estimating distances of different objects: cones, benches, trees, and vehicles.</p> "> Figure 8
<p>Clustered bar charts of predictors “Reaction” (<b>a</b>) and “Reaction zone” (<b>b</b>).</p> "> Figure 9
<p>Boxplot chart of “Average error of distances calculated in VR” filtered by Event.</p> "> Figure 10
<p>Grouped Scatter of “Event” by “Percentage of attention time”.</p> "> Figure 11
<p>Change in total sickness measure of each stage.</p> "> Figure 12
<p>SUS score frequency distribution across all participants.</p> "> Figure 13
<p>Individual decision tree model.</p> "> Figure 14
<p>Out-of-bag (OOB) error rate vs. number of trees.</p> "> Figure 15
<p>Individual tree of the Random Forest ensemble.</p> "> Figure 16
<p>Random Forest feature importance using MDI criterion.</p> "> Figure 17
<p>Random Forest feature importance using MDA criterion.</p> "> Figure 18
<p>Precision, recall, F1-score, and accuracy values for each classifier model.</p> "> Figure 19
<p>ROC curves and AUC value of each classifier.</p> ">
Abstract
:1. Introduction
2. State of the Art
Proposed Work
3. Materials and Methods
3.1. Methodology
3.2. VR Environment and Equipment
3.3. Development of VR Tests for Pedestrians: Procedure and Questionnaires
- 1.
- Signing of the participation agreement (personal data disclaimer)
- 2.
- Completion of SSQ questionnaire (SSQ1)
- 3.
- Stage 1 (ST1, 10 min). Familiarization with the VR technology using the Steam VR room. The user is asked to explore the room and recognize the boundaries of the play area, shown virtually through a blue mesh. ST1 lasts 10 min.
- 4.
- Completion of SSQ questionnaire (SSQ2)
- 5.
- Stage 2 (ST2, 30 min):
- 5.1.
- Familiarization with the VR environment in an ad hoc training scenario. The user is asked to walk around to become acquainted with the 3D environment and give feedback on the degree of immersion, clarity, and sound.
- 5.2.
- Estimation of distances in VR (cone, 8 m; benches, 10 m; tree, 12.5 m; car, 16 m). The user, from a fixed position in the play area, is asked to estimate the distance a series of objects are in a VR scenario.
- 5.3.
- Estimation of distances with real objects (same objects and distances as VR). The user, from the same fixed position, is asked to estimate the distance these real objects are from the same position, without wearing the HMD.
- 5.4.
- Speed calculation test. The subject must make three round trips on the pedestrian crossing of a street that is closed to traffic. Thus, the pedestrian’s average speed is measured for prospective analysis, and the subject becomes more comfortable walking in the virtual environment (and with the VR headset put on).
- 5.5.
- Crossing at Toreros Avenue (SC1). It allows putting the user in a context, simulating real crossing conditions.
- 6.
- Completion of SSQ (SSQ3) and SUS questionnaires.
- 7.
- Stage 3 (ST3, 15 min):
- 7.1.
- Crash tests. In SC2 and SC3, the pedestrian is requested to cross the road as in real life. When the user is crossing the pedestrian walkway, the application’s monitoring team triggers the hit-and-run event, launching a driver-piloted vehicle with no onboard AEB system, that skips the crossing priority.
- 7.2.
- Calculation of vehicle speed in VR. The pedestrian, standing on the sidewalk, must estimate the speed of the vehicle crossing the road. The traffic speeds are 30, 40, 60, and 80 km/h, respectively.
- 7.3.
- Calculation of the safety TTC. The user is positioned in front of an oncoming vehicle and is asked to react the moment the user considers it unsafe to remain in the position this pedestrian is in. Therefore, considering the gap acceptance and the relative speed at the moment of reaction, the TTC for which the pedestrian considers it safe to react can be worked out.
- 8.
- Completion of SSQ (SSQ4) and PQ questionnaires
3.4. Database Generation
3.5. Data Preprocessing
- The probability of being run over is significantly lower when the pedestrian stops and backs up than when accelerating, where the distribution of cases is 58.3% for accidents and 41.6% for avoidance. Failure to react is irrefutably implicated in a hit-and-run accident
- Reacting before reaching the lane in which the vehicle is traveling means an avoidance rate of 85.7%, while reacting once in that lane means being hit is 57.1%. In case of not changing the walking speed, the collision is ensured.
4. Results
4.1. Questionnaires Results
4.2. Model Fitting
5. Discussion
6. Conclusions
Author Contributions
Funding
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Observatorio Nacional de Seguridad Vial. Avance de Las Principales Cifras de La Siniestralidad Vial; Dirección General de Tráfico: Madrid, España, 2020.
- de Winter, J.; Leeuwen, P.M.; Happee, R. Advantages and Disadvantages of Driving Simulators: A Discussion. In Proceedings of the Proceedings of Measuring Behavior 2012, Utrecht, The Netherlands, 28 August 2012. [Google Scholar]
- Losada, Á.; Páez, F.J.; Luque, F.; Piovano, L.; Herrero Villamor, J.J. Santamaría, Asunción Improvement of the AEB Activation Algorithm Based on the Pedestrian Reaction. In Proceedings of the FISITA World Congress 2021—Technical Programme, FISITA, Prague, Czech Republic, 13–17 September 2021. [Google Scholar]
- Cordeil, M.; Dwyer, T.; Klein, K.; Laha, B.; Marriott, K.; Thomas, B.H. Immersive Collaborative Analysis of Network Connectivity: CAVE-Style or Head-Mounted Display? IEEE Trans. Visual. Comput. Graphics 2017, 23, 441–450. [Google Scholar] [CrossRef] [PubMed]
- Kim, H.; Kim, D.J.; Chung, W.H.; Park, K.-A.; Kim, J.D.K.; Kim, D.; Kim, K.; Jeon, H.J. Clinical Predictors of Cybersickness in Virtual Reality (VR) among Highly Stressed People. Sci. Rep. 2021, 11, 12139. [Google Scholar] [CrossRef] [PubMed]
- Kennedy, R.S.; Lane, N.E.; Berbaum, K.S.; Lilienthal, M.G. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. Int. J. Aviat. Psychol. 1993, 3, 203–220. [Google Scholar] [CrossRef]
- Sauro, J. Measuring Usability with the System Usability Scale (SUS). Available online: http://www.measuringu.com/sus.php (accessed on 7 August 2022).
- León-Domínguez, U.; Solís-Marcos, I.; Barrio-Álvarez, E.; Barroso y Martín, J.M.; León-Carrión, J. Safe Driving and Executive Functions in Healthy Middle-Aged Drivers. Appl. Neuropsychol. Adult 2017, 24, 395–403. [Google Scholar] [CrossRef] [PubMed]
- León-Domínguez, U.; Solís-Marcos, I.; López-Delgado, C.A.; Martín, J.M.B.; León-Carrión, J. A Frontal Neuropsychological Profile in Fitness to Drive. Accid. Anal. Prev. 2020, 148, 105807. [Google Scholar] [CrossRef] [PubMed]
- Ortiz, C.; Castro, J.J.; Alarcón, A.; Soler, M.; Anera, R.G. Quantifying Age-Related Differences in Visual-Discrimination Capacity: Drivers with and without Visual Impairment. Appl. Ergon. 2013, 44, 523–531. [Google Scholar] [CrossRef]
- Gruden, C.; Ištoka Otković, I.; Šraml, M. Safety Analysis of Young Pedestrian Behavior at Signalized Intersections: An Eye-Tracking Study. Sustainability 2021, 13, 4419. [Google Scholar] [CrossRef]
- Maruhn, P.; Schneider, S.; Bengler, K. Measuring Egocentric Distance Perception in Virtual Reality: Influence of Methodologies, Locomotion and Translation Gains. PLoS ONE 2019, 14, e0224651. [Google Scholar] [CrossRef] [Green Version]
- Petzoldt, T. On the Relationship between Pedestrian Gap Acceptance and Time to Arrival Estimates. Accid. Anal. Prev. 2014, 72, 127–133. [Google Scholar] [CrossRef]
- Iryo-Asano, M.; Hasegawa, Y.; Dias, C. Applicability of Virtual Reality Systems for Evaluating Pedestrians’ Perception and Behavior. Transp. Res. Procedia 2018, 34, 67–74. [Google Scholar] [CrossRef]
- Sankaran, M.; Vedagiri, P. Modeling Pedestrian Crossing Behavior and Safety at Signalized Intersections. Transp. Res. Rec. J. Transp. Res. Board 2018, 2672. [Google Scholar] [CrossRef]
- Gunn, C.; Mackus, M.; Griffin, C.; Munafò, M.R.; Adams, S. A Systematic Review of the Next-Day Effects of Heavy Alcohol Consumption on Cognitive Performance. Addiction 2018, 113, 2182–2193. [Google Scholar] [CrossRef] [PubMed]
- Stańczyk, T.; Jurecki, R.; Jakiewicz, M.; Walczak, S.; Janczur, R. Researches on the reaction of a pedestrian stepping into the road from the right side from behind and an obstacle realized on the track. J. Kones 2011, 18, 615–622. [Google Scholar]
- Chen, Z.; Ngai, D.C.K.; Yung, N.H.C. Pedestrian Behavior Prediction Based on Motion Patterns for Vehicle-to-Pedestrian Collision Avoidance. In Proceedings of the 2008 11th International IEEE Conference on Intelligent Transportation Systems, Beijing, China, 12–15 October 2008; pp. 316–321. [Google Scholar]
- Keller, C.G.; Gavrila, D.M. Will the Pedestrian Cross? A Study on Pedestrian Path Prediction. IEEE Trans. Intell. Transport. Syst. 2014, 15, 494–506. [Google Scholar] [CrossRef] [Green Version]
- Papathanasopoulou, V.; Spyropoulou, I.; Perakis, H.; Gikas, V.; Andrikopoulou, E. Classification of Pedestrian Behavior Using Real Trajectory Data. In Proceedings of the 2021 7th International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS), Heraklion, Greece, 16 June 2021; pp. 1–6. [Google Scholar]
- Volz, B.; Mielenz, H.; Agamennoni, G.; Siegwart, R. Feature Relevance Estimation for Learning Pedestrian Behavior at Crosswalks. In Proceedings of the 2015 IEEE 18th International Conference on Intelligent Transportation Systems, Gran Canaria, Spain, 15–18 September 2015; IEEE: Piscataway, NJ, USA, 2015; pp. 854–860. [Google Scholar]
- Schratter, M.; Bouton, M.; Kochenderfer, M.J.; Watzenig, D. Pedestrian Collision Avoidance System for Scenarios with Occlusions. In Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France, 9–12 June 2019; pp. 1054–1060. [Google Scholar]
- Lopez Rosado, A.; Chien, S.; Li, L.; Yi, Q.; Chen, Y.; Sherony, R. Certainty and Critical Speed for Decision Making in Tests of Pedestrian Automatic Emergency Braking Systems. IEEE Trans. Intell. Transport. Syst. 2017, 18, 1358–1370. [Google Scholar] [CrossRef] [Green Version]
- Wakim, C.F.; Capperon, S.; Oksman, J. A Markovian Model of Pedestrian Behavior. In Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583), The Hague, The Netherlands, 10–13 October 2004; pp. 4028–4033. [Google Scholar]
- Ouyang, W.; Zhou, H.; Li, H.; Li, Q.; Yan, J.; Wang, X. Jointly Learning Deep Features, Deformable Parts, Occlusion and Classification for Pedestrian Detection. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 40, 1874–1887. [Google Scholar] [CrossRef]
- Iftikhar, S.; Asim, M.; Zhang, Z.; El-Latif, A.A.A. Advance Generalization Technique through 3D CNN to Overcome the False Positives Pedestrian in Autonomous Vehicles. Telecommun. Syst. 2022, 80, 545–557. [Google Scholar] [CrossRef]
- Dai, X.; Hu, J.; Zhang, H.; Shitu, A.; Luo, C.; Osman, A.; Sfarra, S.; Duan, Y. Multi-Task Faster R-CNN for Nighttime Pedestrian Detection and Distance Estimation. Infrared Phys. Technol. 2021, 115, 103694. [Google Scholar] [CrossRef]
- Iftikhar, S.; Zhang, Z.; Asim, M.; Muthanna, A.; Koucheryavy, A.; Abd El-Latif, A.A. Deep Learning-Based Pedestrian Detection in Autonomous Vehicles: Substantial Issues and Challenges. Electronics 2022, 11, 3551. [Google Scholar] [CrossRef]
- Sherony, R.; Tian, R.; Chien, S.; Fu, L.; Chen, Y.; Takahashi, H. Pedestrian/Bicyclist Limb Motion Analysis from 110-Car TASI Video Data for Autonomous Emergency Braking Testing Surrogate Development. SAE Int. J. Trans. Saf. 2016, 4, 113–120. [Google Scholar] [CrossRef] [Green Version]
- Nie, B.; Li, Q.; Gan, S.; Xing, B.; Huang, Y.; Li, S.E. Safety Envelope of Pedestrians upon Motor Vehicle Conflicts Identified via Active Avoidance Behaviour. Sci. Rep. 2021, 11, 3996. [Google Scholar] [CrossRef] [PubMed]
- Tefft, B.C. Impact Speed and a Pedestrian’s Risk of Severe Injury or Death. Accid. Anal. Prev. 2013, 50, 871–878. [Google Scholar] [CrossRef] [PubMed]
- Sánchez de Madariaga, I.; García-Maroto, P. Women at UPM: Gender Statistics at Universidad Politécnica de Madrid; Fundación General de la Universidad Politécnica de Madrid: Madrid, España, 2014. [Google Scholar]
- Brooke, J. SUS—A Quick and Dirty Usability Scale. Usability Eval. Ind. 1996, 189, 4–7. [Google Scholar]
- Bangor, A.; Kortum, P.; Miller, J. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J. Usability Stud. 2009, 4, 114–123. [Google Scholar]
- Witmer, B.G.; Singer, M.J. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence 1998, 7, 225–240. [Google Scholar] [CrossRef]
- Inferential Tests on Correlations, Counts, and Means. In SPSS Data Analysis for Univariate, Bivariate, and Multivariate Statistics; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2018; pp. 201–202. ISBN 978-1-119-46577-5.
- Toğaçar, M.; Ergen, B.; Cömert, Z.; Özyurt, F. A Deep Feature Learning Model for Pneumonia Detection Applying a Combination of MRMR Feature Selection and Machine Learning Models. IRBM 2020, 41, 212–222. [Google Scholar] [CrossRef]
- Hastie, T.; Tibishirani, R.; Friedman, J. The Elements of Statistical Learning; Springer Series in Statistics (SSS); Springer: Berlin/Heidelberg, Germany, 2009; ISBN 978-0-387-84857-0. [Google Scholar]
- Cervantes, J.; Garcia-Lamont, F.; Rodríguez-Mazahua, L.; Lopez, A. A Comprehensive Survey on Support Vector Machine Classification: Applications, Challenges and Trends. Neurocomputing 2020, 408, 189–215. [Google Scholar] [CrossRef]
- Koch, P.; Bischl, B.; Flasch, O.; Bartz-Beielstein, T.; Weihs, C.; Konen, W. Tuning and Evolution of Support Vector Kernels. Evol. Intel. 2012, 5, 153–170. [Google Scholar] [CrossRef]
- Abu Alfeilat, H.A.; Hassanat, A.B.A.; Lasassmeh, O.; Tarawneh, A.S.; Alhasanat, M.B.; Eyal Salman, H.S.; Prasath, V.B.S. Effects of Distance Measure Choice on K-Nearest Neighbor Classifier Performance: A Review. Big Data 2019, 7, 221–248. [Google Scholar] [CrossRef] [Green Version]
- Quinlan, J.R. Induction of Decision Trees. Mach. Learn. 1986, 1, 81–106. [Google Scholar] [CrossRef] [Green Version]
- Rokach, L.; Maimon, O. Top-Down Induction of Decision Trees Classifiers—A Survey. IEEE Trans. Syst. Man Cybern. C 2005, 35, 476–487. [Google Scholar] [CrossRef] [Green Version]
- Breiman, L. Random Forests. Mach. Learn. 2001, 45, 5–32. [Google Scholar] [CrossRef] [Green Version]
- Mitchell, M.W. Bias of the Random Forest Out-of-Bag (OOB) Error for Certain Input Parameters. OJS 2011, 01, 205–211. [Google Scholar] [CrossRef]
- Loyola-Gonzalez, O. Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses from a Practical Point of View. IEEE Access 2019, 7, 154096–154113. [Google Scholar] [CrossRef]
- Karungaru, S.; Tsuji, R.; Terada, K. Driving Assistance: Pedestrians and Bicycles Accident Risk Estimation Using Onboard Front Camera. Int. J. Intell. Transp. Syst. Res. 2022. [Google Scholar] [CrossRef]
Scenario | Street Lane for VR Collision Point | Side from Which the Vehicle Appears | GPS Coordinates |
---|---|---|---|
Av. de los Toreros (SC1) | First | Left | 40°25′54.8″ N 3°39′41.8″ W |
Av. Machupichu (SC2) | Second | Left | 40°27′38.6″ N 3°37′57.7″ W |
Calle Hermanos García Noblejas (SC3) | Third | Left | 40°25′47.7″ N 3°37′56.3″ W |
SSQ Items | Nausea | Oculomotor | Disorientation |
---|---|---|---|
1. General discomfort | Օ | Օ | |
2. Fatigue | Օ | ||
3. Headache | Օ | ||
4. Eyestrain | Օ | ||
5. Difficulty focusing | Օ | Օ | |
6. Increased salivation | Օ | ||
7. Sweating | Օ | ||
8. Nausea | Օ | Օ | |
9. Difficulty concentrating | Օ | Օ | |
10. Fullness of head | Օ | ||
11. Blurred vision | Օ | Օ | |
12. Dizzy (eyes open) | Օ | ||
13. Dizzy (eyes closed) | Օ | ||
14. Vertigo | Օ | ||
15. Stomach awareness | Օ | ||
16. Burping | Օ | ||
Total | (1) | (2) | (3) |
SSQ Components | Computation | ||
1. General discomfort | (1) × 9.54 | ||
2. Fatigue | (2) × 7.58 | ||
3. Headache | (3) × 9.56 | ||
Total score (TS) | ((1) + (2) + (3)) × 3.74 |
Question | |
---|---|
1 | I think that I would like to use this product frequently. |
2 | I found the product unnecessarily complex |
3 | I thought the product was easy to use. |
4 | I think that I would need the support of a technical person to be able to use this product. |
5 | I found the various functions in the product were well integrated. |
6 | I thought there was too much inconsistency in this product. |
7 | I imagine that most people would learn to use this product very quickly. |
8 | I found the product very awkward to use. |
9 | I felt very confident using the product. |
10 | I needed to learn a lot of things before I could get going with this product. |
PAT | TTC | Average Error DR | Average Error DVR | Average Error VVR |
---|---|---|---|---|
<0.001 | 0.0388 | 0.2 | 0.062 | 0.031 |
PAT | TTC | Average Error DR | Average Error DVR | Average Error VVR | |
---|---|---|---|---|---|
PAT | 1 | 0.128 (sig = 0.343) | −0.106 (sig = 0.433) | −0.117 (sig = 0.388) | −0.141 (sig = 0.295) |
TTC | 0.128 (sig = 0.343) | 1 | −0.329 (sig = 0.012) | −0.355 (sig = 0.007) | 0.112 (sig = 0.409) |
Average error DR | −0.106 (sig = 0.433) | −0.329 (sig = 0.012) | 1 | 0.498 (sig = 0.001) * | 0.026 (sig = 0.848) |
Average error DVR | −0.117 (sig = 0.388) | −0.355 (sig = 0.007) | 0.498 (sig = 0.001) * | 1 | −0.077 (sig = 0.571) |
Average error VVR | −0.141 (sig = 0.295) | 0.112 (sig = 0.409) | −0.026 (sig = 0.848) | −0.077 (sig = 0.571) | 1 |
Reaction Type | Reaction Zone | Street Type | Event | |
---|---|---|---|---|
Reaction type | 1 | 60.034 (sig < 0.001) | 0.541 (sig = 0.763) | 41.464 (sig < 0.001) |
Reaction zone | 60.034 (sig < 0.001) | 1 | 7.738 (sig = 0.021) | 36.371 (sig < 0.001) |
Street type | 0.541 (sig = 0.763) | 7.738 (sig = 0.021) | 1 | 0.153 (sig = 0.696) |
Event | 41.464 (sig < 0.001) | 36.371 (sig < 0.001) | 0.153 (sig = 0.696) | 1 |
Model | Hyperparameters |
---|---|
SVM | C: [1; 10; 100; 1000]; Kernel function: [rbf; linear]; gamma (only for rbf): [1× 10−3; 1× 10−4] |
KNN | K: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]; metric (p): [1, Manhattan; 2, Minkowski] |
Individual decision tree * | Split criteria: [Cross-Entropy;Gini-index] |
Random Forest * | Split criteria: [Cross-Entropy;Gini-index], number trees: [determined through Out-of-the-bag error criteria] |
Model | Hyperparameters |
---|---|
SVM | C: 1000; Kernel function: rbf; gamma (only for rbf): 1 × 10−3 |
KNN | K: [1,31]; metric (p): 2 (Minkowski) |
Individual decision tree * | Split criteria: Gini-index |
Random Forest * | Split criteria: Gini-index, number trees: 24 (OOB rate stable around 0.12) |
SVM | KNN | Decision Tree | Random Forest | |
---|---|---|---|---|
Accuracy | 86.1% | 89% | 91.5% | 84.6% |
Precision | 86% | 91% | 90% | 84% |
Recall | 86% | 90% | 91% | 84% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Losada, Á.; Páez, F.J.; Luque, F.; Piovano, L. Application of Machine Learning Techniques for Predicting Potential Vehicle-to-Pedestrian Collisions in Virtual Reality Scenarios. Appl. Sci. 2022, 12, 11364. https://doi.org/10.3390/app122211364
Losada Á, Páez FJ, Luque F, Piovano L. Application of Machine Learning Techniques for Predicting Potential Vehicle-to-Pedestrian Collisions in Virtual Reality Scenarios. Applied Sciences. 2022; 12(22):11364. https://doi.org/10.3390/app122211364
Chicago/Turabian StyleLosada, Ángel, Francisco Javier Páez, Francisco Luque, and Luca Piovano. 2022. "Application of Machine Learning Techniques for Predicting Potential Vehicle-to-Pedestrian Collisions in Virtual Reality Scenarios" Applied Sciences 12, no. 22: 11364. https://doi.org/10.3390/app122211364