Sport-Related Human Activity Detection and Recognition Using a Smartwatch
<p>Illustration of triaxial acceleration of three kinds of sports: (<b>a</b>) the waveform of playing badminton, (<b>b</b>) the waveform of swimming, and (<b>c</b>) the waveform of running.</p> "> Figure 2
<p>The graph shows the correct way to wear the watch, as well as the form of the data used in this paper. The left side shows the proper way to wear the watch. The middle of the right side shows the waveform of sensor signals collected by the watch during badminton. On the top and bottom of the waveforms are real-time pictures of the wearer’s movements. In particular, the waveform in the green box corresponds to the serving movement, the waveform in the yellow box corresponds to the swinging movement, and the frames corresponding to these two motions are framed by the corresponding dotted colored line.</p> "> Figure 3
<p>The flowchart of non-periodic activity with complex motion states (NP_CMS) activity detection and recognition. (<b>a</b>) Interval generation: the bottom lines with different colors and lengths represent the candidate anchors that we preset. The bold lines show the candidate interval. This step picks out the candidate interval from the anchors by using cls and fine-tunes it with reg to better match the ground truth. (<b>b</b>) Interval-based activity recognition: the yellow part of the gray feature map represents the relative features of the second candidate interval. The relative features are extracted from the feature map and then pooled into a fixed size. Finally, the fixed-size features are entered into the fully connected layer and softmax layer to produce the final recognition result.</p> "> Figure 4
<p>The illustration of interval generation.</p> "> Figure 5
<p>The flowchart of weak periodicity and complex motion states (WP_CMS) activity detection and recognition. (<b>a</b>) The pipeline of periodic matching method. (<b>b</b>) The pipeline of recognition method.</p> "> Figure 6
<p>Algorithm of periodic matching based on classification.</p> "> Figure 7
<p>The architectures and configurations of the three networks used in this study. (<b>a</b>) Net_1 is used for interval generation in <a href="#sec3dot1dot1-sensors-19-05001" class="html-sec">Section 3.1.1</a>. (<b>b</b>) Net_2 is used for interval-based activity recognition in <a href="#sec3dot1dot2-sensors-19-05001" class="html-sec">Section 3.1.2</a>. (<b>c</b>) Net_3 is used for classification in <a href="#sec3dot2dot2-sensors-19-05001" class="html-sec">Section 3.2.2</a>. * indicates that the stride of the layer is 2; otherwise, it is 1. The curve with the arrow represents a “shortcut connection”.</p> "> Figure 8
<p>Visualization of the results. The black boxes are the real interval, the red box is the predicted interval of “serving”, and the blue box is the predicted interval of “swinging”. Their confidence levels are marked in the upper left corner.</p> "> Figure 9
<p>The duration distribution of the target motion states in badminton.</p> "> Figure 10
<p>The duration distribution of four strokes.</p> "> Figure 11
<p>Visualization of the detection and recognition results of our proposed algorithm. The green box indicates the detected breaststroke interval, the yellow box indicates the detected butterfly interval, the red box indicates the detected backstroke interval, and the blue box indicates the detected freestyle interval.</p> ">
Abstract
:1. Introduction
2. Datasets and Data Pre-Processing
2.1. Datasets
2.2. Data Pre-Processing and Input Adaptation
3. The Proposed Method for Complex Activity Detection and Recognition
3.1. NP_CMS Activity Monitoring
3.1.1. Interval Generation
- As mentioned above, a set of anchors of different sizes are set in the center of the receptive field corresponding to each feature. For the alignment, the length of the input signal must be an integer multiple of the stride of the CNN output with respect to the input signal. On this basis, we set the input signal to a multiple of 4 (the stride of Net_1, which is used for interval generation) by adding zero to the end of it. Let be the length of the original signal; the number of zeros added (denoted by ) can be obtained by the following calculation: . Therefore, our CNN can process the input of any length.
- The adjusted signal is divided into N equal parts (, where is the length of the adjusted signal). For each part of the signal, we set up a set of anchors with different lengths centered on the signal’s center. Therefore, we can get anchors, where is the number of anchors in each set. In this work, we set the length of anchors to [16, 24, 32, 40, 48, 56, 64, 72, 80, 96], so these anchors can match each ground truth.
- Sort all candidate intervals according to their scores and select the candidate interval with the highest score.
- Calculate the IOU between the selected interval and each of the other candidate intervals. Remove the candidate intervals with an IOU that is larger than the threshold.
- Repeat the above operations for the remaining candidate intervals until the last iteration, when the scores of all candidate intervals are less than the threshold.
3.1.2. Interval-Based Activity Recognition
3.1.3. The Loss of CNNs
3.2. WP_CMS Activity Monitoring
3.2.1. Classification-Based Periodic Matching
3.2.2. Classification with a Convolutional Neural Network
4. Experiments and Discussion
4.1. Network Setting
4.2. Experiments for NP_CMS Activity Detection and Recognition
- —True Positive, the number of detection results that match the real results;
- —False Positive, the number of detection results that do not match the real results;
- —False Negative, the number of real results that do not match the detection results;
- Precision—The proportion of correct detection results to all detection results, ;
- Recall—The proportion of correct detection results to all real results, ;
- mAP—The average accuracy under different recall rates, .
4.2.1. Evaluation of the Quality of Interval Generation
4.2.2. Comparison with Faster R-CNN
4.2.3. Comparison with Traditional Algorithms
4.3. Experiments for WP_CMS Activity Monitoring
5. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
Abbreviations
HAR | Human activity recognition |
HADR | Human activity detection and recognition |
CNN | Convolutional neural network |
NP_CMS | Non-periodic activity with complex motion states |
WP_CMS | Weak periodicity and complex motion states |
PAMAP2 | Physical Activity Monitoring for Aging People dataset |
DSA | Daily and Sports Activity dataset |
IOU | Intersection-over-union |
FC | Fully connected |
KNN | K-nearest neighbors |
NB | Naive Bayes |
RF | Random forest |
SVM | Support vector machine |
References
- Qi, W.; Su, H.; Yang, C.; Ferrigno, G.; De Momi, E.; Aliverti, A. A Fast and Robust Deep Convolutional Neural Networks for Complex Human Activity Recognition Using Smartphone. Sensors 2019, 19, 3731. [Google Scholar] [CrossRef] [PubMed]
- Gupta, P.; Dallas, T. Feature selection and activity recognition system using a single triaxial accelerometer. IEEE Trans. Biomed. Eng. 2014, 61, 1780–1786. [Google Scholar] [CrossRef] [PubMed]
- Lara, O.D.; Labrador, M.A. A survey on human activity recognition using wearable sensors. IEEE Commun. Surv. Tutor. 2013, 15, 1192–1209. [Google Scholar] [CrossRef]
- Ignatov, A. Real-time human activity recognition from accelerometer data using Convolutional Neural Networks. Appl. Soft Comput. 2018, 62, 915–922. [Google Scholar] [CrossRef]
- Jeong, C.Y.; Kim, M. An Energy-Efficient Method for Human Activity Recognition with Segment-Level Change Detection and Deep Learning. Sensors 2019, 19, 3688. [Google Scholar] [CrossRef] [PubMed]
- Ponce, H.; Miralles-Pechuán, L.; Martínez-Villaseñor, M. A flexible approach for human activity recognition using artificial hydrocarbon networks. Sensors 2016, 16, 1715. [Google Scholar] [CrossRef] [PubMed]
- Attal, F.; Mohammed, S.; Dedabrishvili, M.; Chamroukhi, F.; Oukhellou, L.; Amirat, Y. Physical human activity recognition using wearable sensors. Sensors 2015, 15, 31314–31338. [Google Scholar] [CrossRef] [PubMed]
- Siirtola, P.; Laurinen, P.; Röning, J.; Kinnunen, H. Efficient accelerometer-based swimming exercise tracking. In Proceedings of the 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), Paris, France, 11–15 April 2011; pp. 156–161. [Google Scholar]
- Jensen, U.; Blank, P.; Kugler, P.; Eskofier, B.M. Unobtrusive and energy-efficient swimming exercise tracking using on-node processing. IEEE Sens. J. 2016, 16, 3972–3980. [Google Scholar] [CrossRef]
- Brunner, G.; Melnyk, D.; Sigfússon, B.; Wattenhofer, R. Swimming style recognition and lap counting using a smartwatch and deep learning. In Proceedings of the 23rd International Symposium on Wearable Computers, London, UK, 9–13 September 2019; pp. 23–31. [Google Scholar]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2012; pp. 1097–1105. [Google Scholar]
- Nweke, H.F.; Teh, Y.W.; Al-Garadi, M.A.; Alo, U.R. Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Syst. Appl. 2018, 105, 233–261. [Google Scholar] [CrossRef]
- Chen, Y.; Xue, Y. A deep learning approach to human activity recognition based on single accelerometer. In Proceedings of the 2015 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Kowloon, China, 9–12 October 2015; pp. 1488–1492. [Google Scholar]
- Shrivastava, A.; Gupta, A.; Girshick, R. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 761–769. [Google Scholar]
- Zhou, X.; Yao, C.; Wen, H.; Wang, Y.; Zhou, S.; He, W.; Liang, J. EAST: An efficient and accurate scene text detector. In Proceedings of the Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2642–2651. [Google Scholar]
- Yang, W.; Jin, L.; Liu, M. DeepWriterID: An End-to-End Online Text-Independent Writer Identification System. IEEE Intell. Syst. 2016, 31, 45–53. [Google Scholar] [CrossRef]
- Xie, L.; Liu, Y.; Jin, L.; Xie, Z. DeRPN: Taking a further step toward more general object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January–1 February 2019; Volume 33, pp. 9046–9053. [Google Scholar]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2015; pp. 91–99. [Google Scholar]
- Zhuang, Z.; Xue, Y. TS-ICNN: Time Sequence-Based Interval Convolutional Neural Networks for Human Action Detection and Recognition. IEICE Trans. Inf. Syst. 2018, 101, 2534–2538. [Google Scholar] [CrossRef]
- Reiss, A.; Stricker, D. Introducing a new benchmarked dataset for activity monitoring. In Proceedings of the IEEE 2012 16th International Symposium on Wearable Computers, Newcastle, UK, 18–22 June 2012; pp. 108–109. [Google Scholar]
- Barshan, B.; Yüksek, M.C. Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units. Comput. J. 2014, 57, 1649–1667. [Google Scholar] [CrossRef]
- Van Laerhoven, K.; Borazio, M.; Burdinski, J.H. Wear is your mobile? Investigating phone carrying and use habits with a wearable device. Front. ICT 2015, 2, 10. [Google Scholar] [CrossRef]
- King, C.E.; Sarrafzadeh, M. A survey of smartwatches in remote health monitoring. J. Healthc. Inform. Res. 2018, 2, 1–24. [Google Scholar] [CrossRef] [PubMed]
- Mortazavi, B.; Nemati, E.; VanderWall, K.; Flores-Rodriguez, H.; Cai, J.; Lucier, J.; Naeim, A.; Sarrafzadeh, M. Can smartwatches replace smartphones for posture tracking? Sensors 2015, 15, 26783–26800. [Google Scholar] [CrossRef] [PubMed]
- Ha, S.; Choi, S. Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors. In Proceedings of the IEEE 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada, 24–29 July 2016; pp. 381–388. [Google Scholar]
- Dai, J.; Li, Y.; He, K.; Sun, J. R-fcn: Object detection via region-based fully convolutional networks. In Advances in Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2016; pp. 379–387. [Google Scholar]
- Bottou, L. Stochastic gradient learning in neural networks. Proc. Neuro-Nımes 1991, 91, 12. [Google Scholar]
- Bishop, C.M. Pattern Recognition and Machine Learning; Springer: Berlin/Heidelberg, Germany, 2006. [Google Scholar]
- Girshick, R. Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1440–1448. [Google Scholar]
- Peterson, L.E. K-nearest neighbor. Scholarpedia 2009, 4, 1883. [Google Scholar] [CrossRef]
- Jensen, F.V. An Introduction to Bayesian Networks; UCL Press: London, UK, 1996; Volume 210. [Google Scholar]
- Breiman, L. Random forests. Mach. Learn. 2001, 45, 5–32. [Google Scholar] [CrossRef]
- Burges, C.J. A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 1998, 2, 121–167. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
Fold-1 | Fold-2 | Fold-3 | Fold-4 | Average | |
---|---|---|---|---|---|
Recall rate | 97.78 | 96.28 | 90.94 | 97.73 | 95.68 |
Method | Fold-1 | Fold-2 | Fold-3 | Fold-4 | Average |
---|---|---|---|---|---|
67.72 | 67.13 | 76.47 | 68.38 | 69.93 | |
Our interval-based HADR | 87.87 | 86.63 | 85.48 | 89.07 | 87.26 |
Method | Recall of Serving | Recall of Swinging | mAP | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
F-1 | F-2 | F-3 | F-4 | Ave | F-1 | F-2 | F-3 | F-4 | Ave | F-1 | F-2 | F-3 | F-4 | Ave | |
ASSW | 36.6 | 83.3 | 90.0 | 88.0 | 74.5 | 83.1 | 92.0 | 94.6 | 92.4 | 90.5 | 10.3 | 25.3 | 30.2 | 29.3 | 23.8 |
FLSW | 36.6 | 72.0 | 50.0 | 72.0 | 57.6 | 67.6 | 62.1 | 75.7 | 72.3 | 69.5 | 14.3 | 25.7 | 23.7 | 30.9 | 23.7 |
Our interval-based HADR | 100 | 96.0 | 90.0 | 100 | 96.5 | 95.6 | 96.6 | 91.9 | 95.5 | 94.9 | 87.9 | 86.6 | 85.5 | 89.1 | 87.3 |
Method | Density (Distance between Adjacent Anchors) | Scale |
---|---|---|
ASSW | 12 or 16 or 20 | 30, 40, 50 |
FLSW | 24 | 60 |
Our interval-based HADR | 4 | 16, 24, 32, 40, 48, 56, 64, 72, 80, 96 |
Method | Time (ms) |
---|---|
ASSW | 48 |
FLSW | 11 |
Our interval-based HADR | 20 |
Classifier | Accuracy with Periodic Matching (%) | Accuracy without Periodic Matching (%) |
---|---|---|
CNN | 97.88 | 93.99 |
SVM | 97.96 | 94.17 |
Naive Bayes | 96.06 | 91.80 |
Random Forest | 96.97 | 92.80 |
KNN | 96.67 | 93.89 |
Classifier | Time (ms) |
---|---|
CNN | 37 |
SVM | 11 |
Naive Bayes | 12 |
Random Forest | 13 |
KNN | 15 |
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhuang, Z.; Xue, Y. Sport-Related Human Activity Detection and Recognition Using a Smartwatch. Sensors 2019, 19, 5001. https://doi.org/10.3390/s19225001
Zhuang Z, Xue Y. Sport-Related Human Activity Detection and Recognition Using a Smartwatch. Sensors. 2019; 19(22):5001. https://doi.org/10.3390/s19225001
Chicago/Turabian StyleZhuang, Zhendong, and Yang Xue. 2019. "Sport-Related Human Activity Detection and Recognition Using a Smartwatch" Sensors 19, no. 22: 5001. https://doi.org/10.3390/s19225001