Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving
<p>A scene in which a driver’s emotional state data is being collected during real-world driving using the proposed data collection system. The driver is self-reporting their emotional state by touching the HMI application mounted on the vehicle center fascia. The screenshot on the right is the English translation of the GUI of the HMI application implemented in Korean.</p> "> Figure 2
<p>Figures of the dataset collection system hardware interface build in the vehicle. (<b>a</b>) Vehicle exterior; (<b>b</b>) Inside view of the vehicle center fascia; (<b>c</b>) Inside view of the vehicle passenger seat; (<b>d</b>) Vehicle trunk. Two cameras are installed to collect the image data of a driver’s face and posture (<span style="color: #70AD47">green</span>). A microphone is installed on the right side of the driver seat’s headrest to collect audio data in the cabin (<span style="color: #447AC4">blue</span>). Wristband-type wearable sensor is worn on the driver’s wrist to collect the driver’s bio-physiological data, and the collecting status can be monitored through a smartphone (<span style="color: #ED7C30">orange</span>). The CAN interface device supports the collection of vehicle CAN data (<span style="color: #FF0000">red</span>). The monitor installed on the center fascia is a touch screen for interaction with the driver (<span style="color: #E0BF00">yellow</span>). The computer installed in the trunk of the vehicle integrates the collected data (<span style="color: #FE2EF7">magenta</span>).</p> "> Figure 3
<p>Flow chart of the proposed data collection system during real-world driving.</p> "> Figure 4
<p>GUI of HMI application for self-reporting of driver emotional state. (<b>a</b>) GUI in default; (<b>b</b>) GUI in touch; (<b>c</b>) GUI example where “Angry|Disgusting” state is touched.</p> "> Figure 5
<p>Pie charts for self-reported emotion label proportion by driver. (<b>a</b>) Driver A; (<b>b</b>) Driver B; (<b>c</b>) Driver C; (<b>d</b>) Driver D; (<b>e</b>) Legend of the pie charts.</p> "> Figure 6
<p>Distribution of self-reported emotion labels in real-world driving. (<b>a</b>) Happy|Neutral; (<b>b</b>) Excited|Surprised; (<b>c</b>) Angry|Disgusting; (<b>d</b>) Sad|Fatigued.</p> "> Figure 7
<p>Distribution of vehicle speed by self-reported emotion labels in real-world driving. (<b>a</b>) Happy|Neutral; (<b>b</b>) Excited|Surprised; (<b>c</b>) Angry|Disgusting; (<b>d</b>) Sad|Fatigued.</p> "> Figure 8
<p>Example of the detection results of five face detectors. The bounding boxes (<span style="color: #FF0000">red</span>) are face detection results. (<b>a</b>) Haar; (<b>b</b>) Dlib; (<b>c</b>) OpenCV; (<b>d</b>) MMOD; (<b>e</b>) MTCNN.</p> "> Figure 9
<p>PR curve for face detectors capable of detecting the driver’s face. The thresholds are 0.5 and 0.75. (<b>a</b>) OpenCV, threshold is 0.5; (<b>b</b>) MMOD, threshold is 0.5; (<b>c</b>) MTCNN, threshold is 0.5; (<b>d</b>) OpenCV, threshold is 0.75; (<b>e</b>) MMOD, threshold is 0.75; (<b>f</b>) MTCNN, threshold is 0.75.</p> "> Figure 10
<p>Example image with IoU of 0.68. Area of union (<span style="color: #70AD47">green</span> and <span style="color: #FF0000">red</span>) is 7441, and area of overlap (<span style="color: #0000FF">blue</span>) is 5040.</p> "> Figure 11
<p>Deep learning-based personalized driver emotion recognition model.</p> ">
Abstract
:1. Introduction
- We proposed a data collection system that can collect the multimodal data of drivers during real-world driving tasks. The proposed system is capable of collecting real-world driving big data for driver emotion recognition while considering the minimization of behavioral disturbances.
- The proposed system comprises an HMI application through which drivers can report their emotional states. This application is designed to collect selected emotional states from the driver without cognitive disturbance during real-world driving by utilizing the haptic, acoustic response, and GUI, and eliminating the bias problem that may occur with the self-reporting by setting the interaction period.
- We deployed the proposed system on a vehicle and collected high-quality multimodal sensor data without any accidents during real-world driving experiments for over 122 h. To demonstrate the validity of our collected dataset, we provided various case studies such as statistical analysis, driver face detection, and personalized single and multimodal driver emotion recognition.
2. Related Works
3. Proposed Work
3.1. Video
3.2. Audio
3.3. Biophysiological
3.4. CAN
3.5. HMI
3.6. GUI
- Happy|Neutral;
- Excited|Surprised;
- Angry|Disgusting;
- Sad|Fatigued.
4. Experiments
4.1. Data Collection Experiment
Data | Sample Rate (Hz) | Format | Unit | |
---|---|---|---|---|
Video | RGB-front | 15 | .avi | - |
RGB-side | 15 | .avi | - | |
IR-front | 15 | .avi | - | |
IR-side | 15 | .avi | - | |
Audio | - | 44,100 | .wav | - |
Bio-physiological | Skin temperature | 4 | .csv | |
EDA | 4 | .csv | ||
PPG | 64 | .csv | ||
IBI | - | .csv | ||
HR | 1 | .csv | bpm | |
3-axis acceleration | 32 | .csv | ||
CAN | Accelerator pedal position | 100 | .csv | % |
Brake pedal position | 100 | .csv | % | |
Steering wheel angle | 100 | .csv | ||
Yaw rate | 100 | .csv | ||
Longitudinal acceleration | 100 | .csv | ||
Lateral acceleration | 100 | .csv | ||
Self-reported emotions | Emotional state | no less than | .csv | - |
4.2. Case Studies
4.2.1. Statistical Analysis
4.2.2. Driver Face Detection
4.2.3. Personalized Driver Emotion Recognition
- Single-modal of front image (): The single-modal recognition model of the front image uses front IR images for 2 s from 4 s to 2 s before the driver’s self-reporting. Because RGB images are vulnerable to changes in illuminance, IR images that can always capture a stable image are used as input. From 2 s before self-reporting, it shows uniform motion for self-reporting, so it is excluded from the input data. The input images are evenly time-divided into six equal parts and input to a face detector; the MMOD-based face detector outputs one cropped face image with the highest confidence value for each input. The cropped images are resized to the input shape of the feature extractor and sequentially fed into a feature extractor and a classifier based on CAPNet [41]. Because the classification form is different from that of CAPNet, only the number of units in the top layer of the classifier is modified to the number of representative driver emotional states. The last activation function is softmax and outputs the probability of each representative driver emotional state.
- Single-modal of side image (): The single-modal recognition model of the side image uses the side IR image captured 2 s before self-reporting. The reason for using the image from 2 s ago is the same as that for using the front image. The input image is fed into a feature extractor based on AlphaPose [42]. The feature extractor consists of layers up to just before outputting feature points in the form of histograms in AlphaPose. The classifier consists of a global max pooling layer and fully connected layers. The top layer of the classifier is the same as other classifiers to output the probability of each representative driver emotional states.
- Single-modal of biophysiological (): The single-modal recognition model of biophysiological data uses the PPG and EDA data for 10 s before the driver’s self-reporting. Since PPG and EDA have different sample rates, up-sampling using linear interpolation is applied to the EDA data to match the input shape. The biophysiological input is directly fed into the classifier without a feature extractor to output the probability of each representative driver emotional state. The classifier is composed of the fully connected and batch normalization layers.
- Single-modal of CAN (): The single-modal recognition model of CAN data uses all collected signals for 10 s before the driver’s self-reporting. The input data are down-sampled by a tenth before being fed into the feature extractor. The feature extractor is an encoder of long short-term memory-based autoencoder that extracts the feature vector for driving propensity. The classifier consists of fully connected layers and a dropout and outputs the probability of each representative driver emotional states by receiving the feature vector.
- Multimodal (M): The multimodal recognition model uses the input vectors of each classifier of single-modal as input vectors. The model is a deep learning-based ensemble model that outputs the probability of each representative driver emotional states by fusing all input vectors. The feature vectors of the front image, CAN, and side image are flattened using flatten and pooling layers. The flattened vectors are concatenated using the concatenate layer. The concatenated vector undergoes the normalization, fully connected layers, and softmax activation function to become the final output. The input modalities to fuse can be chosen, and the modals are denoted by a subscript, e.g., is the ensemble model that fuses the front image and biophysiological data. We evaluated three or more input modal combinations for multimodal models.
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Acknowledgments
Conflicts of Interest
Abbreviations
CAN | Controller area network |
HMI | Human–machine interaction |
GUI | Graphical user interface |
UX | User experience |
SAM | Self-assessment manikin |
RGB | Red green blue |
IR | Infrared |
E4 | E4 wristband |
EDA | Electrodermal activity |
PPG | Photoplethysmography |
IBI | Interbeat interval |
HR | Heart rate |
OBD | On-board diagnostics |
IoU | Intersection over union |
TP | True positive |
FP | False positive |
PR | Presicion–recall |
AP | Average precision |
FN | False negative |
Appendix A
Expression | Definition | Unit |
---|---|---|
Sample rate of the video data | Hz | |
Sample rate of the audio data | Hz | |
Sample rate of the self-reporting | Hz | |
Sample rate of the CAN data | Hz | |
Request time interval of HMI application | s | |
Re-request time interval of HMI application | s | |
Skip time interval of HMI application | s | |
K | Mileage for completing the train data collection | km |
Null hypothesis of the statistical hypothesis test | - | |
Alternative hypothesis of the statistical hypothesis test | - | |
Single-modal recognition model of the front image | - | |
Single-modal recognition model of the side image | - | |
Single-modal recognition model of the bio-phyological | - | |
Single-modal recognition model of the CAN | - | |
M | Multimodal recognition model | - |
N | Total number of representative emotions | - |
s | Second | - |
bpm | Beats per minute | - |
Gravitationnal acceleration | ||
FPS | Frame per second | - |
References
- Rouast, P.V.; Adam, M.T.; Chiong, R. Deep learning for human affect recognition: Insights and new developments. IEEE Trans. Affect. Comput. 2019, 12, 524–543. [Google Scholar] [CrossRef] [Green Version]
- Underwood, G.; Chapman, P.; Wright, S.; Crundall, D. Anger while driving. Transp. Res. Part F Traffic Psychol. Behav. 1999, 2, 55–68. [Google Scholar] [CrossRef]
- Jeon, M. Don’t cry while you’re driving: Sad driving is as bad as angry driving. Int. J. Hum.-Comput. Interact. 2016, 32, 777–790. [Google Scholar] [CrossRef]
- Hassib, M.; Braun, M.; Pfleging, B.; Alt, F. Detecting and influencing driver emotions using psycho-physiological sensors and ambient light. In Proceedings of the IFIP Conference on Human-Computer Interactionr, Paphos, Cyprus, 2–6 September 2019; pp. 721–742. [Google Scholar]
- Gao, H.; Yüce, A.; Thiran, J.P. Detecting emotional stress from facial expressions for driving safety. In Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France, 27–30 October 2014; pp. 5961–5965. [Google Scholar]
- Oh, G.; Ryu, J.; Jeong, E.; Yang, J.H.; Hwang, S.; Lee, S.; Lim, S. Drer: Deep learning-based driver’s real emotion recognizer. Sensors 2021, 21, 2166. [Google Scholar] [CrossRef]
- Healey, J.A.; Picard, R.W. Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 2005, 6, 156–166. [Google Scholar] [CrossRef] [Green Version]
- Angkititrakul, P.; Petracca, M.; Sathyanarayana, A.; Hansen, J.H. UTDrive: Driver behavior and speech interactive systems for in-vehicle environments. In Proceedings of the 2007 IEEE Intelligent Vehicles Symposium, Istanbul, Turkey, 13–15 June 2007; pp. 566–569. [Google Scholar]
- Singh, R.R.; Conjeti, S.; Banerjee, R. Biosignal based on-road stress monitoring for automotive drivers. In Proceedings of the 2012 National Conference on Communications (NCC), Kharagpur, India, 3–5 February 2012; pp. 1–5. [Google Scholar]
- Jones, C.; Jonsson, I.M. Using paralinguistic cues in speech to recognise emotions in older car drivers. In Affect and Emotion in Human-Computer Interaction; Springer: Berlin/Heidelberg, Germany, 2008; pp. 229–240. [Google Scholar]
- Ma, Z.; Mahmoud, M.; Robinson, P.; Dias, E.; Skrypchuk, L. Automatic detection of a driver’s complex mental states. In Proceedings of the International Conference on Computational Science and Its Applications, Trieste, Italy, 3–6 July 2017; pp. 678–691. [Google Scholar]
- Kato, T.; Kawanaka, H.; Bhuiyan, M.S.; Oguri, K. Classification of positive and negative emotion evoked by traffic jam based on electrocardiogram (ECG) and pulse wave. In Proceedings of the 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), Washington, DC, USA, 5–7 October 2011; pp. 1217–1222. [Google Scholar]
- Taib, R.; Tederry, J.; Itzstein, B. Quantifying driver frustration to improve road safety. In Proceedings of the CHI’14 Extended Abstracts on Human Factors in Computing Systems, Toronto, ON, Canada, 26 April–1 May 2014; pp. 1777–1782. [Google Scholar]
- Ihme, K.; Dömeland, C.; Freese, M.; Jipp, M. Frustration in the face of the driver: A simulator study on facial muscle activity during frustrated driving. Interact. Stud. 2018, 19, 487–498. [Google Scholar] [CrossRef]
- Zepf, S.; Hernandez, J.; Schmitt, A.; Minker, W.; Picard, R.W. Driver emotion recognition for intelligent vehicles: A survey. ACM Comput. Surv. 2020, 53, 1–30. [Google Scholar] [CrossRef]
- Ortega, J.D.; Kose, N.; Cañas, P.; Chao, M.A.; Unnervik, A.; Nieto, M.; Otaegui, O.; Salgado, L. Dmd: A large-scale multi-modal driver monitoring dataset for attention and alertness analysis. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; pp. 387–405. [Google Scholar]
- Jegham, I.; Khalifa, A.B.; Alouani, I.; Mahjoub, M.A. A novel public dataset for multimodal multiview and multispectral driver distraction analysis: 3MDAD. Signal Process. Image Commun. 2020, 88, 115960. [Google Scholar] [CrossRef]
- Martin, M.; Roitberg, A.; Haurilet, M.; Horne, M.; Reiß, S.; Voit, M.; Stiefelhagen, R. Drive&act: A multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019; pp. 2801–2810. [Google Scholar]
- Deo, N.; Trivedi, M.M. Looking at the driver/rider in autonomous vehicles to predict take-over readiness. IEEE Trans. Intell. Veh. 2019, 5, 41–52. [Google Scholar] [CrossRef] [Green Version]
- Song, T.; Zheng, W.; Song, P.; Cui, Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans. Affect. Comput. 2018, 11, 532–541. [Google Scholar] [CrossRef] [Green Version]
- Tao, W.; Li, C.; Song, R.; Cheng, J.; Liu, Y.; Wan, F.; Chen, X. EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans. Affect. Comput. 2020. [Google Scholar] [CrossRef]
- Li, S.; Deng, W. Deep facial expression recognition: A survey. IEEE Trans. Affect. Comput. 2020. [Google Scholar] [CrossRef] [Green Version]
- Kawaguchi, N.; Matsubara, S.; Takeda, K.; Itakura, F. Multimedia data collection of in-car speech communication. In Proceedings of the 7th European Conference on Speech Communication and Technology, Aalborg, Denmark, 3–7 September 2001. [Google Scholar]
- Bradley, M.M.; Lang, P.J. Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 1994, 25, 49–59. [Google Scholar] [CrossRef]
- McCarthy, C.; Pradhan, N.; Redpath, C.; Adler, A. Validation of the Empatica E4 wristband. In Proceedings of the 2016 IEEE EMBS International Student Conference (ISC), Ottawa, ON, Canada, 29–31 May 2016; pp. 1–4. [Google Scholar]
- Ragot, M.; Martin, N.; Em, S.; Pallamin, N.; Diverrez, J.M. Emotion recognition using physiological signals: Laboratory vs. wearable sensors. In Proceedings of the International Conference on Applied Human Factors and Ergonomics, Los Angeles, CA, USA, 17–21 July 2017; pp. 15–22. [Google Scholar]
- Shiffman, S.; Stone, A.A.; Hufford, M.R. Ecological momentary assessment. Annu. Rev. Clin. Psychol. 2008, 4, 1–32. [Google Scholar] [CrossRef]
- Jeon, M.; Walker, B.N. What to detect? Analyzing factor structures of affect in driving contexts for an emotion detection and regulation system. In Proceedings of the 55th Annual Meeting of the Human Factors and Ergonomics Society, Human Factors and Ergonomics Society, Las Vegas, NV, USA, 19–23 September 2011; Volume 55, pp. 1889–1893. [Google Scholar]
- Russell, J.A. A circumplex model of affect. J. Personal. Soc. Psychol. 1980, 39, 1161. [Google Scholar] [CrossRef]
- Schauss, A.G. Tranquilizing effect of color reduces aggressive behavior and potential violence. J. Orthomol. Psychiatry 1979, 8, 218–221. [Google Scholar]
- Kruskal, W.H.; Wallis, W.A. Use of ranks in one-criterion variance analysis. J. Am. Stat. Assoc. 1952, 47, 583–621. [Google Scholar] [CrossRef]
- Ostertagova, E.; Ostertag, O.; Kováč, J. Methodology and application of the Kruskal-Wallis test. In Applied Mechanics and Materials; Trans Tech Publications Ltd.: Bäch, Switzerland, 2014; Volume 611, pp. 115–120. [Google Scholar]
- Kruskal, W.H. Historical notes on the Wilcoxon unpaired two-sample test. J. Am. Stat. Assoc. 1957, 52, 356–360. [Google Scholar] [CrossRef]
- Hart, A. Mann-Whitney test is not just a test of medians: Differences in spread can be important. Bmj 2001, 323, 391–393. [Google Scholar] [CrossRef] [Green Version]
- Jain, V.; Learned-Miller, E. Fddb: A Benchmark for Face Detection in Unconstrained Settings; Technical Report UMCS-2010-009; University of Massachusetts: Amherst, MA, USA, 2010. [Google Scholar]
- Yang, S.; Luo, P.; Loy, C.C.; Tang, X. Wider face: A face detection benchmark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 5525–5533. [Google Scholar]
- Viola, P.; Jones, M. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, Kauai, HI, USA, 8–14 December 2001; Volume 1, p. I. [Google Scholar]
- Dalal, N.; Triggs, B. Histograms of oriented gradients for human detection. In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 20–25 June 2005; Volume 1, pp. 886–893. [Google Scholar]
- Viola, P.; Jones, M.J. Robust real-time face detection. Int. J. Comput. Vis. 2004, 57, 137–154. [Google Scholar] [CrossRef]
- King, D.E. Max-margin object detection. arXiv 2015, arXiv:1502.00046. [Google Scholar]
- Oh, G.; Jeong, E.; Lim, S. Causal affect prediction model using a past facial image sequence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 3550–3556. [Google Scholar]
- Fang, H.S.; Xie, S.; Tai, Y.W.; Lu, C. RMPE: Regional Multi-person Pose Estimation. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017. [Google Scholar]
Gender | Age (Year) | Driving Experience (Year) | Experiment Time (h) | Driving Mileage (km) | |
---|---|---|---|---|---|
Driver A | Male | 27 | more than 15 | 38 | 1375 |
Driver B | Male | 32 | between 11–15 | 43 | 1449 |
Driver C | Male | 26 | between 6–10 | 21 | 852 |
Driver D | Male | 28 | less than 5 | 20 | 770 |
Data | Statistical Hypothesis Test | Post-Hoc Test | |
---|---|---|---|
Reject | Number of Reject Pairs (Total Number of Pairs is 6) | ||
Bio-physiological | Skin temperature | Yes | 6 |
EDA | Yes | 5 | |
PPG | Yes | 3 | |
HR | Yes | 4 | |
CAN | Accelerator pedal position | Yes | 5 |
Brake pedal position | Yes | 6 | |
Steering wheel angle | Yes | 6 | |
Yaw rate | Yes | 3 | |
Longitudinal acceleration | Yes | 6 | |
Lateral acceleration | Yes | 5 |
Data | Statistical Hypothesis Test | Post-Hoc Test | |||||||
---|---|---|---|---|---|---|---|---|---|
Reject | Number of Reject Pairs (Total Number of Pairs is 6) | ||||||||
Driver A | Driver B | Driver C | Driver D | Driver A | Driver B | Driver C | Driver D | ||
Bio-physiological | Skin temperature | Yes | Yes | Yes | Yes | 5 | 6 | 6 | 6 |
EDA | Yes | Yes | Yes | Yes | 6 | 6 | 6 | 6 | |
PPG | No | Yes | No | Yes | - | 1 | - | 3 | |
HR | Yes | Yes | Yes | Yes | 4 | 5 | 5 | 2 | |
CAN | Accelerator pedal position | Yes | Yes | Yes | Yes | 5 | 6 | 6 | 6 |
Brake pedal position | Yes | Yes | Yes | Yes | 6 | 5 | 6 | 6 | |
Steering wheel angle | Yes | Yes | Yes | Yes | 6 | 6 | 6 | 6 | |
Yaw rate | Yes | Yes | Yes | Yes | 6 | 6 | 4 | 5 | |
Longitudinal acceleration | Yes | Yes | Yes | Yes | 6 | 5 | 3 | 6 | |
Lateral acceleration | Yes | Yes | Yes | Yes | 5 | 6 | 6 | 6 |
AP50 | AP75 | AP95 | Speed | GPU | |
---|---|---|---|---|---|
OpenCV | 68.4 | 51.4 | 0.0 | 400 FPS | Nvidia GTX 3080 |
MMOD | 83.8 | 18.1 | 0.0 | 260 FPS | Nvidia GTX 3080 |
MTCNN | 81.4 | 72.0 | 0.0 | 4 FPS | Nvidia GTX 3080 |
F1 | 0.696 | 0.698 | 0.619 | 0.355 | 0.613 | 0.430 | 0.469 | 0.469 | 0.228 |
Precision | 0.541 | 0.537 | 0.478 | 0.248 | 0.446 | 0.280 | 0.311 | 0.314 | 0.231 |
Recall | 0.975 | 0.998 | 0.879 | 0.630 | 0.982 | 0.923 | 0.950 | 0.927 | 0.225 |
F1 | 0.584 | 0.613 | 0.593 | 0.536 | 0.562 | 0.646 | 0.661 | 0.667 | 0.615 |
Precision | 0.419 | 0.442 | 0.475 | 0.492 | 0.420 | 0.539 | 0.522 | 0.500 | 0.468 |
Recall | 0.963 | 1.000 | 0.790 | 0.589 | 0.852 | 0.805 | 0.900 | 1.000 | 0.900 |
Average F1 | 0.496 | 0.444 | 0.447 | 0.561 | 0.456 | 0.500 | 0.607 | 0.557 | 0.483 | |
Excited | F1 | 0.359 | 0.301 | 0.362 | 0.653 | 0.344 | 0.487 | 0.444 | 0.465 | 0.417 |
| | Precision | 0.591 | 1.000 | 0.563 | 0.593 | 1.000 | 0.950 | 0.800 | 0.909 | 1.000 |
Surprised | Recall | 0.258 | 0.177 | 0.267 | 0.727 | 0.208 | 0.328 | 0.308 | 0.313 | 0.263 |
Angry | F1 | 0.293 | 0.196 | 0.147 | 0.263 | 0.216 | 0.280 | 0.571 | 0.400 | 0.200 |
| | Precision | 0.579 | 1.000 | 1.000 | 0.500 | 1.000 | 0.875 | 0.667 | 1.000 | 0.667 |
Disgusting | Recall | 0.196 | 0.109 | 0.080 | 0.179 | 0.121 | 0.167 | 0.500 | 0.250 | 0.118 |
Sad | F1 | 0.835 | 0.833 | 0.830 | 0.768 | 0.808 | 0.733 | 0.807 | 0.806 | 0.831 |
| | Precision | 1.000 | 1.000 | 1.000 | 0.977 | 0.995 | 1.000 | 0.926 | 1.000 | 1.000 |
Fatigued | Recall | 0.717 | 0.714 | 0.710 | 0.632 | 0.680 | 0.578 | 0.714 | 0.675 | 0.711 |
Average F1 | 0.488 | 0.472 | 0.481 | 0.450 | 0.491 | 0.468 | 0.491 | 0.501 | 0.511 | |
Excited | F1 | 0.450 | 0.403 | 0.333 | 0.286 | 0.511 | 0.417 | 0.537 | 0.511 | 0.583 |
| | Precision | 0.636 | 1.000 | 0.452 | 1.000 | 1.000 | 1.000 | 0.846 | 0.923 | 0.539 |
Surprised | Recall | 0.348 | 0.252 | 0.264 | 0.167 | 0.344 | 0.263 | 0.393 | 0.353 | 0.636 |
Angry | F1 | 0.270 | 0.270 | 0.373 | 0.204 | 0.321 | 0.194 | 0.227 | 0.273 | 0.233 |
| | Precision | 1.000 | 1.000 | 0.452 | 1.000 | 0.907 | 0.429 | 1.000 | 1.000 | 1.000 |
Disgusting | Recall | 0.156 | 0.156 | 0.264 | 0.114 | 0.195 | 0.125 | 0.128 | 0.158 | 0.132 |
Sad | F1 | 0.744 | 0.743 | 0.736 | 0.859 | 0.641 | 0.794 | 0.710 | 0.719 | 0.717 |
| | Precision | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 0.958 | 0.864 |
Fatigued | Recall | 0.593 | 0.592 | 0.582 | 0.753 | 0.472 | 0.658 | 0.550 | 0.575 | 0.613 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Oh, G.; Jeong, E.; Kim, R.C.; Yang, J.H.; Hwang, S.; Lee, S.; Lim, S. Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving. Sensors 2022, 22, 4402. https://doi.org/10.3390/s22124402
Oh G, Jeong E, Kim RC, Yang JH, Hwang S, Lee S, Lim S. Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving. Sensors. 2022; 22(12):4402. https://doi.org/10.3390/s22124402
Chicago/Turabian StyleOh, Geesung, Euiseok Jeong, Rak Chul Kim, Ji Hyun Yang, Sungwook Hwang, Sangho Lee, and Sejoon Lim. 2022. "Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving" Sensors 22, no. 12: 4402. https://doi.org/10.3390/s22124402