[go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Spatiotemporal multi-scale bilateral motion network for gait recognition

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

The critical goal of gait recognition is to acquire the inter-frame walking habit representation from the gait sequences. The relations between frames, however, have not received adequate attention in comparison to the intra-frame features. In this paper, motivated by optical flow, the bilateral motion-oriented block is proposed to explore motion description at the feature level. It can allow the classic 2D convolutional structure to have the capability to directly portray gait movement patterns while preventing costly computations on the estimation of optical flow. Based on such features, we develop a set of multi-scale temporal representations that force the motion context to be richly described at various levels of temporal resolution. Furthermore, the dynamic information is sensitive to inaccurate segmentation on the edge, so a correction block is devised to eliminate the segmentation noise of silhouettes for getting more precise gait modality. Subsequently, the temporal feature set and the spatial features are combined to comprehensively characterize gait processes. Extensive experiments are conducted on CASIA-B and OU-MVLP datasets, and the results achieve an outstanding identification performance, which has demonstrated the effectiveness of the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Availability of data and materials

The data that support the findings of this study are available in OU-MVLP dataset and CASIA-B dataset at http://www.am.sanken.osaka-u.ac.jp/BiometricDB/GaitMVLP and http://www.cbsr.ia.ac.cn/english/Gait%20Databases, respectively.

References

  1. Zhu Z, Guo X, Yang T, Huang J, Deng J, Huang G, Du D, Lu J, Zhou J (2021) Gait recognition in the wild: a benchmark. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 14789–14799

  2. Sepas-Moghaddam A, Etemad A (2022) Deep gait recognition: a survey. IEEE Trans Pattern Anal Mach Intell

  3. Ben X, Gong C, Zhang P, Yan R, Wu Q, Meng W (2019) Coupled bilinear discriminant projection for cross-view gait recognition. IEEE Trans Circuits Syst Video Technol 30(3):734–747

    Article  Google Scholar 

  4. Du Y, Ai H, Lao S (2012) Evaluation of color spaces for person re-identification. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). IEEE, pp 1371–1374

  5. Hong P, Wu T, Wu A, Han X, Zheng W-S (2021) Fine-grained shape-appearance mutual learning for cloth-changing person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10513–10522

  6. Huang Y, Wu Q, Xu J, Zhong Y, Zhang Z (2021) Clothing status awareness for long-term person re-identification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 11895–11904

  7. Elharrouss O, Almaadeed N, Al-Maadeed S, Bouridane A (2021) Gait recognition for person re-identification. J Supercomput 77(4):3653–3672

    Article  Google Scholar 

  8. Sarkar S, Phillips PJ, Liu Z, Vega IR, Grother P, Bowyer KW (2005) The humanid gait challenge problem: Data sets, performance, and analysis. IEEE Trans Pattern Anal Mach Intell 27(2):162–177

    Article  Google Scholar 

  9. Han J, Bhanu B (2005) Individual recognition using gait energy image. IEEE Trans Pattern Anal Mach Intell 28(2):316–322

    Article  Google Scholar 

  10. Wang C, Zhang J, Wang L, Pu J, Yuan X (2011) Human identification using temporal information preserving gait template. IEEE Trans Pattern Anal Mach Intell 34(11):2164–2176

    Article  Google Scholar 

  11. Chao H, He Y, Zhang J, Feng J (2019) Gaitset: Regarding gait as a set for cross-view gait recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp 8126–8133

  12. Song X, Huang Y, Shan C, Wang J, Chen Y (2022) Distilled light gaitset: towards scalable gait recognition. Pattern Recognit Lett 157:27–34

    Article  Google Scholar 

  13. Wu Z, Huang Y, Wang L (2015) Learning representative deep features for image set analysis. IEEE Trans Multimed 17(11):1960–1968

    Article  Google Scholar 

  14. Lin B, Zhang S, Yu X (2021) Gait recognition via effective global-local feature representation and local temporal aggregation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 14648–14656

  15. Wolf T, Babaee M, Rigoll G (2016) Multi-view gait recognition using 3d convolutional neural networks. In: 2016 IEEE International Conference on Image Processing (ICIP). IEEE, pp. 4165–4169

  16. Choi S, Kim J, Kim W, Kim C (2019) Skeleton-based gait recognition via robust frame-level matching. IEEE Trans Inf Forensics Secur 14(10):2577–2592

    Article  Google Scholar 

  17. Li X, Makihara Y, Xu C, Yagi Y, Yu S, Ren M (2020) End-to-end model-based gait recognition. In: Proceedings of the Asian Conference on Computer Vision

  18. Ding X, Wang K, Wang C, Lan T, Liu L (2021) Sequential convolutional network for behavioral pattern extraction in gait recognition. Neurocomputing 463:411–421

    Article  Google Scholar 

  19. Tang J, Luo J, Tjahjadi T, Guo F (2016) Robust arbitrary-view gait recognition based on 3d partial similarity matching. IEEE Trans Image Process 26(1):7–22

    Article  MathSciNet  Google Scholar 

  20. Zhang Z, Troje NF (2005) View-independent person identification from human gait. Neurocomputing 69(1–3):250–256

    Article  Google Scholar 

  21. Muramatsu D, Makihara Y, Yagi Y (2015) View transformation model incorporating quality measures for cross-view gait recognition. IEEE Trans Cybern 46(7):1602–1615

    Article  Google Scholar 

  22. Ben X, Gong C, Zhang P, Jia X, Wu Q, Meng W (2019) Coupled patch alignment for matching cross-view gaits. IEEE Trans Image Process 28(6):3142–3157

    Article  MathSciNet  Google Scholar 

  23. Xing X, Wang K, Yan T, Lv Z (2016) Complete canonical correlation analysis with application to multi-view gait recognition. Pattern Recognit 50:107–117

    Article  Google Scholar 

  24. Hou S, Cao C, Liu X, Huang Y (2020) Gait lateral network: Learning discriminative and compact representations for gait recognition. In: European Conference on Computer Vision. Springer, pp 382–398

  25. Shiraga K, Makihara Y, Muramatsu D, Echigo T, Yagi Y (2016) Geinet: View-invariant gait recognition using a convolutional neural network. In: 2016 International Conference on Biometrics (ICB). IEEE, pp 1–8

  26. Song C, Huang Y, Huang Y, Jia N, Wang L (2019) Gaitnet: an end-to-end network for gait based human identification. Pattern Recognit 96:106988

    Article  Google Scholar 

  27. Bukhari M, Durrani MY, Gillani S, Yasmin S, Rho S, Yeo S-S (2022) Exploiting vulnerability of convolutional neural network-based gait recognition system. J Supercomput 2022:1–20

    Google Scholar 

  28. Wu Z, Huang Y, Wang L, Wang X, Tan T (2016) A comprehensive study on cross-view gait based human identification with deep CNNS. IEEE Trans Pattern Anal Mach Intell 39(2):209–226

    Article  Google Scholar 

  29. Xu C, Makihara Y, Li X, Yagi Y, Lu J (2021) Cross-view gait recognition using pairwise spatial transformer networks. IEEE Trans Circuits Syst Video Technol 2021:260–274

    Article  Google Scholar 

  30. Chen X, Luo X, Weng J, Luo W, Li H, Tian Q (2021) Multi-view gait image generation for cross-view gait recognition. IEEE Trans Image Process 30:3041–3055

    Article  Google Scholar 

  31. Zhang S, Wang Y, Li A (2021) Cross-view gait recognition with deep universal linear embeddings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 9095–9104

  32. Qin H, Chen Z, Guo Q, Wu QJ, Lu M (2021) Rpnet: Gait recognition with relationships between each body-parts. IEEE Trans Circuits Syst Video Technol 32(5):2990–3000

    Article  Google Scholar 

  33. Fan C, Peng Y, Cao C, Liu X, Hou S, Chi J, Huang Y, Li Q, He Z (2020) Gaitpart: Temporal part-based model for gait recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 14225–14233

  34. Yao L, Kusakunniran W, Wu Q, Xu J, Zhang J (2021) Collaborative feature learning for gait recognition under cloth changes. IEEE Trans Circuits Syst Video Technol

  35. Fan C, Liang J, Shen C, Hou S, Huang Y, Yu S (2023) Opengait: Revisiting gait recognition towards better practicality. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 9707–9716

  36. Sepas-Moghaddam A, Etemad A (2021) View-invariant gait recognition with attentive recurrent learning of partial representations. IEEE Trans Biom Behav Identity Sci 31:124–137

    Article  Google Scholar 

  37. Sepas-Moghaddam A, Ghorbani S, Troje NF, Etemad A (2020) Gait recognition using multi-scale partial representation transformation with capsules. In: 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, pp 8045–8052

  38. Zhang Y, Huang Y, Yu S, Wang L (2019) Cross-view gait recognition by discriminative feature learning. IEEE Trans Image Process 29:1001–1015

    Article  MathSciNet  Google Scholar 

  39. An W, Yu S, Makihara Y, Wu X, Xu C, Yu Y, Liao R, Yagi Y (2020) Performance evaluation of model-based gait on multi-view very large population database with pose sequences. IEEE Trans Biom Behav Identity Sci 2(4):421–430

    Article  Google Scholar 

  40. Cao Z, Simon T, Wei S-E, Sheikh Y (2017) Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7291–7299

  41. Güler RA, Neverova N, Kokkinos I (2018) Densepose: Dense human pose estimation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7297–7306

  42. Liao R, Yu S, An W, Huang Y (2020) A model-based gait recognition method with body pose and human prior knowledge. Pattern Recognit 98:107069

    Article  Google Scholar 

  43. Li N, Zhao X A strong and robust skeleton-based gait recognition method with gait periodicity priors. IEEE Trans Multimed 1–1 (2022 (Early Access))

  44. Lin B, Zhang S, Bao F (2020) Gait recognition with multiple-temporal-scale 3d convolutional neural network. In: Proceedings of the 28th ACM International Conference on Multimedia, pp 3054–3062

  45. Zhang Z, Tran L, Liu F, Liu X (2020) On learning disentangled representations for gait recognition. IEEE Trans Pattern Anal Mach Intell 2020:345–360

    Google Scholar 

  46. Fu Y, Wei Y, Zhou Y, Shi H, Huang G, Wang X, Yao Z, Huang T (2018) Horizontal pyramid matching for person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 33, pp 8295–8302

  47. Hermans A, Beyer L, Leibe B (2017) In defense of the triplet loss for person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, vol 1-10

  48. Lin T-Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2980–2988

  49. Bigün J, Granlund GH, Wiklund J (1991) Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Trans Pattern Anal Mach Intell 13(08):775–790

    Article  Google Scholar 

  50. Piergiovanni A, Ryoo MS (2019) Representation flow for action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 9945–9953

  51. Sun S, Kuang Z, Sheng L, Ouyang W, Zhang W (2018) Optical flow guided feature: A fast and robust motion representation for video action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1390–1399

  52. Kanopoulos N, Vasanthavada N, Baker RL (1988) Design of an image edge detection filter using the sobel operator. IEEE J Solid-State Circuits 23(2):358–367

    Article  Google Scholar 

  53. Juefei-Xu F, Naresh Boddeti V, Savvides M (2017) Local binary convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 19–28

  54. Kinoshita Y, Kiya H (2020) Fixed smooth convolutional layer for avoiding checkerboard artifacts in cnns. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 3712–3716

  55. Yu S, Tan D, Tan T (2006) A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition. In: 18th International Conference on Pattern Recognition (ICPR’06), vol 4. IEEE, pp 441–444

  56. Takemura N, Makihara Y, Muramatsu D, Echigo T, Yagi Y (2018) Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition. IPSJ Trans Comput Vis Appl 10(1):1–14

    Google Scholar 

  57. Han F, Li X, Zhao J, Shen F (2022) A unified perspective of classification-based loss and distance-based loss for cross-view gait recognition. Pattern Recognit 2022:108519

    Article  Google Scholar 

  58. Li H, Qiu Y, Zhao H, Zhan J, Chen R, Wei T, Huang Z (2022) Gaitslice: a gait recognition model based on spatio-temporal slice features. Pattern Recognit 124:108453

    Article  Google Scholar 

Download references

Funding

The work is supported by National Natural Science Foundation of China (61573114) and the program of China Scholarships Council (202006680049).

Author information

Authors and Affiliations

  1. School of electrical engineering, Anhui Polytechnic University, Wuhu, 241000, Anhui, China

    Xinnan Ding

  2. College of Intelligent Systems Science and Engineering, Harbin Engineering University, Harbin, 150001, Heilongjiang, China

    Xinnan Ding & Kejun Wang

  3. Department of Computer Science, Mathematics, Physics and Statistics, The University of British Columbia, Kelowna, BC, V1V1V7, Canada

    Shan Du

  4. School of Mathematics, Harbin Institute of Technology, Harbin, 150006, Heilongjiang, China

    Yu Zhang

  5. Key Laboratory of Intelligent Detection in Complex Environment of Aerospace, Land and Sea, Beijing Institute of Technology, Zhuhai, Zhuhai, 519085, Guangdong, China

    Kejun Wang

  6. School of Information, Beijing Institute of Technology, Zhuhai, Zhuhai, 519085, Guangdong, China

    Kejun Wang

Authors
  1. Xinnan Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shan Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kejun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

We confirm that all authors reviewed the manuscript. Xinnan Ding contributed to methodology, writing-Original draft preparation, software, validation, and formal analysis; Shan Du contributed to conceptualization, supervision, writing - review and editing, and formal analysis; Yu Zhang contributed to validation and investigation, and data curation; Kejun Wang contributed to supervision, writing - review and editing, and funding acquisition.

Corresponding author

Correspondence to Kejun Wang.

Ethics declarations

Ethical approval

Not applicable.

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ding, X., Du, S., Zhang, Y. et al. Spatiotemporal multi-scale bilateral motion network for gait recognition. J Supercomput 80, 3412–3440 (2024). https://doi.org/10.1007/s11227-023-05607-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-023-05607-3

Keywords

Search

Navigation