[go: up one dir, main page]
Skip to main content
SpringerLink
Log in

State-of-the-Art in 3D Face Reconstruction from a Single RGB Image

  • Conference paper
  • First Online:
Computational Science – ICCS 2021 (ICCS 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12746))

Included in the following conference series:

Abstract

Since diverse and complex emotions need to be expressed by different facial deformation and appearances, facial animation has become a serious and on-going challenge for computer animation industry. Face reconstruction techniques based on 3D morphable face model and deep learning provide one effective solution to reuse existing databases and create believable animation of new characters from images or videos in seconds, which greatly reduce heavy manual operations and a lot of time. In this paper, we review the databases and state-of-the-art methods of 3D face reconstruction from a single RGB image. First, we classify 3D reconstruction methods into three categories and review each of them. These three categories are: Shape-from-Shading (SFS), 3D Morphable Face Model (3DMM), and Deep Learning (DL) based 3D face reconstruction. Next, we introduce existing 2D and 3D facial databases. After that, we review 10 methods of deep learning-based 3D face reconstruction and evaluate four representative ones among them. Finally, we draw conclusions of this paper and discuss future research directions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Zhao, W.-Y., Chellappa, R.: Symmetric shape-from-shading using self-ratio image. Int. J. Comput. Vis. 45(1), 55–75 (2001)

    Article  Google Scholar 

  2. Zhang, R., Tsai, P., Cryer, J.-E., Shah, M.: Shape-from-shading: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 690–706 (1999)

    Google Scholar 

  3. Zhu, X., Lei, Z., Yan, J., Yi, D., Li, S.-Z.: High-fidelity pose and expression normalization for face recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 787–796 (2015)

    Google Scholar 

  4. Li, Y., Ma, L., Fan, H., Mitchell, K.: Feature-preserving detailed 3D face reconstruction from a single image. In: Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production, pp. 1–9 (2018)

    Google Scholar 

  5. Garrido, P., et al.: Reconstruction of personalized 3D face rigs from monocular video. ACM Trans. Graph. (TOG) 35(3), 1–15 (2016)

    Article  Google Scholar 

  6. Sengupta, S., Kanazawa, A., Castillo, C.-D., Jacobs, D.-W.: SfSNet: Learning shape, reflectance and illuminance of faces in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6296–6305 (2018)

    Google Scholar 

  7. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (1999)

    Google Scholar 

  8. Tran, L., Liu, X.: Nonlinear 3D face morphable model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7346–7355 (2018)

    Google Scholar 

  9. Zhu, X., Lei, Z., Liu, X.: Face alignment across large poses: a 3D solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 146–155 (2016)

    Google Scholar 

  10. Huber, P.: Real-time 3D morphable shape model fitting to monocular in-the-wild videos. University of Surrey (2017)

    Google Scholar 

  11. Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296–301. IEEE (2009)

    Google Scholar 

  12. Liu, P., Yu, H., Cang, S.: Adaptive neural network tracking control for underactuated systems with matched and mismatched disturbances. Nonlinear Dyn. 98(2), 1447–1464 (2019)

    Article  Google Scholar 

  13. Gerig, T., et al.: Morphable face models-an open framework. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 75–82. IEEE (2018)

    Google Scholar 

  14. Egger, B., Smith, W., Tewari, A., Wuhrer, A., Zollhoefer, M.: 3D morphable face models—past, present, and future. ACM Trans. Graph. (TOG) 39(5), 1–38 (2020)

    Article  Google Scholar 

  15. Zollhöfer, M., Thies, J., Garrido, P.: State of the art on monocular 3D face reconstruction, tracking, and applications. Comput. Graph. Forum 37(2), 523–550 (2018)

    Article  Google Scholar 

  16. Sun, L., Zhao, C., Yan, Z., Liu, P., Duckett, T., Stolkin, R.: A novel weakly-supervised approach for RGB-D-based nuclear waste object detection. IEEE Sens. J. (2018)

    Google Scholar 

  17. Chen, A., Chen, Z., Zhang, G., Mitchell, K., Yu, J.: Photo-realistic facial details synthesis from single image. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 9429–9439 (2019)

    Google Scholar 

  18. Kasinski, A., Florek, A., Schmidt, A.: The PUT face database. Image Process. Commun. 13(3–4), 59–64 (2008)

    Google Scholar 

  19. Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie. Image Vis. Comput. 28(5), 807–813 (2010)

    Article  Google Scholar 

  20. Milborrow, S., Morkel, J., Nicolls, F.: The MUCT landmarked face database. Pattern Recognit. Assoc. S. Afr. 201(0) (2010)

    Google Scholar 

  21. Grgic, M., Delac, K., Grgic, S.: SCface–surveillance cameras face database. Multimedia Tools Appl. 51(3), 863–879 (2011)

    Article  Google Scholar 

  22. Koestinger, M., Wohlhart, P., Roth, P., Bischof, H.: Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization. In: 2011 IEEE International Conference on Computer Vision Workshops, pp. 2144–2151. IEEE (2011)

    Google Scholar 

  23. Lui, Y.-M., Bolme, D., Phillips, P.-J., Beveridge, J.-R.: Preliminary studies on the good, the bad, and the ugly face recognition challenge problem. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 9–16. IEEE (2012)

    Google Scholar 

  24. Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: Conference on Computer Vision and Pattern Recognition. IEEE (2012)

    Google Scholar 

  25. Sagonas, C., Antonakos, E., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: database and results. Image Vis. Comput. 47, 3–18 (2016)

    Article  Google Scholar 

  26. Ng, H.-W., Winkler, S.: A data-driven approach to cleaning large face datasets. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 343–347. IEEE (2014)

    Google Scholar 

  27. Parkhi, O.-M., Vedaldi, A., Zisserman, A.: Deep face recognition (2015)

    Google Scholar 

  28. Liu, Z., Luo, P., Wang, X., Tang. X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)

    Google Scholar 

  29. Bansal, A., Nanduri, A., Castillo, C.-D., Ranjan, R., Chellappa, R.: UMDfaces: an annotated face dataset for training deep networks. In: 2017 IEEE International Joint Conference on Biometrics (IJCB), pp. 464–473. IEEE (2017)

    Google Scholar 

  30. Shlizerman, I.-K., Seitz, S.-M., Miller, D., Brossard, E.: The MegaFace benchmark: 1 million faces for recognition at scale. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4873–4882 (2016)

    Google Scholar 

  31. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1021–1030 (2017)

    Google Scholar 

  32. Huang, G.-B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments (2008)

    Google Scholar 

  33. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)

    Google Scholar 

  34. Panetta, K., et al.: A comprehensive database for benchmarking imaging systems. IEEE Trans. Pattern Anal. Mach. Intell. 42(3), 509–520 (2020)

    Google Scholar 

  35. Faltemier, T.-C., Bowyer, K.-W., Flynn, P.: Using a multi-instance enrollment representation to improve 3D face recognition. In: 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems, pp. 1–6. IEEE (2007)

    Google Scholar 

  36. Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.-J.: A 3D facial expression database for facial behavior research. In: 7th International Conference on Automatic Face and Gesture Recognition, pp. 211–216. IEEE (2006)

    Google Scholar 

  37. Yin, L., Chen, X, Sun, Y., Worm, T., Reale, M.: 3D dynamic facial expression database. In: 8th International Conference on Automatic Face & Gesture Recognition, pp. 1–6. IEEE (2008)

    Google Scholar 

  38. Heseltine, T., Pears, N.: Three-dimensional face recognition using combinations of surface feature map subspace components. Image Vis. Comput. 26(3), 382–396 (2008)

    Article  Google Scholar 

  39. Savran, A., et al.: Bosphorus database for 3D face analysis. In: Schouten, B., Juul, N.C., Drygajlo, A., Tistarelli, M. (eds.) BioID 2008. LNCS, vol. 5372, pp. 47–56. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89991-4_6

    Chapter  Google Scholar 

  40. Yin, B., Sun, Y., Wang, C., Ge, Y.: BJUT-3D large scale 3D face database and information processing. J. Comput. Res. Dev. 46(6), 1009 (2009)

    Google Scholar 

  41. Cosker, D., Krumhuber, E., Hilton, A.: A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: 2011 International Conference on Computer Vision, pp. 2296–2303. IEEE (2011)

    Google Scholar 

  42. Bagdanov, A.-D., Bimbo, A.-D.: The florence 2D/3D hybrid face dataset. In: Proceedings of ACM Workshop on Human Gesture and Behavior Understanding, pp. 79–80 (2011)

    Google Scholar 

  43. Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou., K.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Vis. Comput. Graph. 20(3), 413–425 (2013)

    Google Scholar 

  44. Zhang, X., Yin, L., Cohn, J.-F.: BP4D-spontaneous: a high-resolution spontaneous 3D dynamic facial expression database. Image Vis. Comput. 32(10), 1–6 (2014)

    Google Scholar 

  45. Zhang, Z., et al.: Multimodal spontaneous emotion corpus for human behavior analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3438–3446 (2016)

    Google Scholar 

  46. Le, H.-A., Kakadiaris, I.-A.: UHDB31: a dataset for better understanding face recognition across pose and illumination variation. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2555–2563 (2017)

    Google Scholar 

  47. Cheng, S., Kotsia, I., Pantic, M., Zafeiriou, S.: 4DFAB: a large scale 4D database for facial expression analysis and biometric applications. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5117–5126 (2018)

    Google Scholar 

  48. Ertugrul, I.-O., Cohn, J.-F., Jeni, L.-A., Zhang, A., Yin, L. Ji, Q.: Cross-domain au detection: domains, learning approaches, and measures. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 1–8. IEEE (2019)

    Google Scholar 

  49. Yang, H., Zhu, H., Wang, Y., Huang, M., Shen, Q.: FaceScape: a large-scale high quality 3D face dataset and detailed riggable 3D face prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 601–610 (2020)

    Google Scholar 

  50. Booth, J., Roussos, A., Ponniah, A., Dunaway, D., Zafeiriou, S.: Large scale 3D morphable models. Int. J. Comput. Vis. 126(2–4), 233–254 (2018)

    Article  MathSciNet  Google Scholar 

  51. Dai, H., Pears, N., Smith, W.-A., Duncan, C.: A 3D morphable model of craniofacial shape and texture variation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3085–3093 (2017)

    Google Scholar 

  52. Chang, F., Tran, A.-T., Hassner, T., Masi, I., Nevatia, R., Medioni, G.: ExpNet: landmark-free, deep, 3D facial expressions. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 122–129. IEEE (2018)

    Google Scholar 

  53. Booth, J., Roussos, A., Ververas, E., Antonakos, E., Ploumpis, S., Panagakis, Y.: 3D reconstruction of “in-the-wild” faces in images and videos. IEEE Trans. Pattern Anal. Mach. Intell. 40(11), 2638–2652 (2018)

    Article  Google Scholar 

  54. Shang, J., Shen, T., Li, S., Zhou, L., Zhen, M.: Self-supervised monocular 3D face reconstruction by occlusion-aware multi-view geometry consistency (2020)

    Google Scholar 

  55. Feng, Y., Wu, F., Shao, X., Wang, Y., Zhou, X.: Joint 3D face reconstruction and dense alignment with position map regression network. In: Proceedings of the European Conference on Computer Vision, pp. 534–551 (2018)

    Google Scholar 

  56. Zeng, Xi., Peng, X., Qiao, Y.: DF2Net: a dense-fine-finer network for detailed 3D face reconstruction. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2315–2324 (2019)

    Google Scholar 

  57. Zhang, J., Cai, H., Guo, Y., Peng, Z.: Landmark detection and 3D face reconstruction for caricature using a nonlinear parametric model (2020)

    Google Scholar 

  58. Browatzki, B., Wallraven, C.: 3FabRec: fast few-shot face alignment by reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6110–6120 (2020)

    Google Scholar 

  59. Wu, S., Rupprecht, C., Vedaldi, A.: Unsupervised learning of probably symmetric deformable 3D objects from images in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1–10 (2020)

    Google Scholar 

  60. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W., Frangi, A. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

Download references

Acknowledgements

This research is supported by the PDE-GIR project which has received funding from the European Union Horizon 2020 Research and Innovation Programme under the Marie Skodowska-Curie grant agreement No 778035.

Author information

Authors and Affiliations

  1. Bournemouth University, Bournemouth, BH12 5BB, UK

    Haibin Fu, Shaojun Bian, Ehtzaz Chaudhry, Lihua You & Jian Jun Zhang

  2. University of Cantabria, 39005, Cantabria, Spain

    Andres Iglesias

  3. Humain Ltd., Belfast, BT1 2LA, UK

    Shaojun Bian

Authors
  1. Haibin Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaojun Bian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ehtzaz Chaudhry
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andres Iglesias
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lihua You
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jian Jun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haibin Fu .

Editor information

Editors and Affiliations

  1. AGH University of Science and Technology, Krakow, Poland

    Maciej Paszynski

  2. Ludwig-Maximilians-Universität München, Munich, Germany

    Dieter Kranzlmüller

  3. University of Amsterdam, Amsterdam, The Netherlands

    Valeria V. Krzhizhanovskaya

  4. University of Tennessee at Knoxville, Knoxville, TN, USA

    Jack J. Dongarra

  5. University of Amsterdam, Amsterdam, The Netherlands

    Peter M.A. Sloot

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fu, H., Bian, S., Chaudhry, E., Iglesias, A., You, L., Zhang, J.J. (2021). State-of-the-Art in 3D Face Reconstruction from a Single RGB Image. In: Paszynski, M., Kranzlmüller, D., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2021. ICCS 2021. Lecture Notes in Computer Science(), vol 12746. Springer, Cham. https://doi.org/10.1007/978-3-030-77977-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77977-1_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77976-4

  • Online ISBN: 978-3-030-77977-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation