Abstract
Computer vision technology is advancing rare disease diagnosis to address unmet needs of the more than 300 million individuals affected globally; one in three rare diseases have a known facial phenotype. 3D face model reconstruction is a key driver of these advances. However, the utility of 3D reconstruction from images obtained from mobile phone cameras has been questionable due to relatively low quality 2D data and need external calibration methods (e.g. visual markers) to extract accurate measurements. Herein a novel implementation pipeline, leveraging deep learning technologies, that can successfully reconstruct 3D face models from multiple 2D images taken by mobile phone cameras for clinician usage is described. Specifically, Multi-view Stereo (MVS) has been introduced to this application for providing a cost-effective pipeline of 3D face dense reconstruction. As a state-of-the-art MVS method, deep-learning based MVS has shown its strong generalization capability of using the low quality 2D face images to reconstruct 3D face models without camera calibration. The results demonstrate conceptual proof of a analytic pipeline to satisfy the clinician’s needs.
Supported by The Centre for Research Excellence in Neurocognitive Disorders, Neuroscience Research Australia.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baynam, G., et al.: 3-dimensional facial analysis-facing precision public health. Front. Pub. Health 5, 1–6 (2017). https://doi.org/10.3389/fpubh.2017.00031
Beeler, T., Bickel, B., Beardsley, P., Sumner, B., Gross, M.: High-quality single-shot capture of facial geometry. In: ACM SIGGRAPH 2010 Papers, pp. 1–9 (2010)
Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194 (1999)
Chang, J.R., Chen, Y.S.: Pyramid stereo matching network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5410–5418 (2018)
Cheng, S., et al.: Deep stereo using adaptive thin volume representation with uncertainty awareness. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2524–2534 (2020)
Collins, R.T.: A space-sweep approach to true multi-image matching. In: Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 358–363. IEEE (1996)
Duggal, S., Wang, S., Ma, W.C., Hu, R., Urtasun, R.: DeepPruner: learning efficient stereo matching via differentiable PatchMatch. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4384–4393 (2019)
Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 873–881 (2015)
Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., Debevec, P.: Multiview face capture using polarized spherical gradient illumination. In: Proceedings of the 2011 SIGGRAPH Asia Conference, pp. 1–10 (2011)
Gu, X., Fan, Z., Zhu, S., Dai, Z., Tan, F., Tan, P.: Cascade cost volume for high-resolution multi-view stereo and stereo matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2495–2504 (2020)
Haendel, M., et al.: How many rare diseases are there? Nat. Rev. Drug Discov. 19(2), 77–78 (2020). https://doi.org/10.1038/d41573-019-00180-y
Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge, New York (2003)
Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., Aanæs, H.: Large scale multi-view stereopsis evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 406–413 (2014)
Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the 4th Eurographics Symposium on Geometry Processing, vol. 7 (2006)
Kendall, A., et al.: End-to-end learning of geometry and context for deep stereo regression. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 66–75 (2017)
Knapitsch, A., Park, J., Zhou, Q.Y., Koltun, V.: Tanks and temples: benchmarking large-scale scene reconstruction. ACM Trans. Graph. (ToG) 36(4), 1–13 (2017)
Kung, S., et al.: Monitoring of therapy for mucopolysaccharidosis type i using dysmorphometric facial phenotypic signatures. In: Zschocke, J., Baumgartner, M., Morava, E., Patterson, M., Rahman, S., Peters, V. (eds.) JIMD Reports, Volume 22. JR, vol. 22, pp. 99–106. Springer, Heidelberg (2015). https://doi.org/10.1007/8904_2015_417
Morales, A., Piella, G., Sukno, F.M.: Survey on 3D face reconstruction from uncalibrated images. Comput. Sci. Rev. 40, 100400 (2021). https://doi.org/10.1016/j.cosrev.2021.100400
Ng, P.C., Henikoff, S.: SIFT: predicting amino acid changes that affect protein function. Nucleic Acids Res. 31(13), 3812–3814 (2003)
Palmer, R.L., Helmholz, P., Baynam, G.: Cliniface: phenotypic visualisation and analysis using non-rigid registration of 3D facial images. Int. Arch. Photogram. Remote Sens. Spat. Inf. Sci. - ISPRS Arch. 43(B2), 301–308 (2020). https://doi.org/10.5194/isprs-archives-XLIII-B2-2020-301-2020
Poulton, C., Thomas, Y.: Scanning a new landscape, 22–24 March 2020
Rai, M.C.E.L., Werghi, N., Al Muhairi, H., Alsafar, H.: Using facial images for the diagnosis of genetic syndromes: a survey. In: 2015 International Conference on Communications, Signal Processing, and their Applications, ICCSPA 2015, pp. 1–6. IEEE (2015)
Rubinstein, Y.R., et al.: The case for open science: rare diseases. JAMIA Open 3(3), 472–486 (2020). https://doi.org/10.1093/jamiaopen/ooaa030
Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104–4113 (2016)
Schops, T., et al.: A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3260–3269 (2017)
Wang, F., Galliani, S., Vogel, C., Speciale, P., Pollefeys, M.: PatchmatchNet: learned multi-view patchmatch stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14194–14203 (2021)
Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 191139 (1980)
Xiao, Y., Zhu, H., Yang, H., Diao, Z., Lu, X., Cao, X.: Detailed facial geometry recovery from multi-view images by learning an implicit function. arXiv preprint arXiv:2201.01016 (2022)
Xu, Q., Tao, W.: Multi-scale geometric consistency guided multi-view stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5483–5492 (2019)
Yang, J., Mao, W., Alvarez, J.M., Liu, M.: Cost volume pyramid based depth inference for multi-view stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4877–4886 (2020)
Yao, Y., Luo, Z., Li, S., Fang, T., Quan, L.: MVSNet: depth inference for unstructured multi-view stereo. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 785–801. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01237-3_47
Yao, Y., Luo, Z., Li, S., Shen, T., Fang, T., Quan, L.: Recurrent MVSNet for high-resolution multi-view stereo depth inference. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5525–5534 (2019)
Zeng, D., Zhao, Q., Long, S., Li, J.: Examplar coherent 3D face reconstruction from forensic Mugshot database. Image Vis. Comput. 58, 193–203 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, Y., Li, L., An, S., Helmholz, P., Palmer, R., Baynam, G. (2022). 3D Face Reconstruction with Mobile Phone Cameras for Rare Disease Diagnosis. In: Aziz, H., Corrêa, D., French, T. (eds) AI 2022: Advances in Artificial Intelligence. AI 2022. Lecture Notes in Computer Science(), vol 13728. Springer, Cham. https://doi.org/10.1007/978-3-031-22695-3_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-22695-3_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22694-6
Online ISBN: 978-3-031-22695-3
eBook Packages: Computer ScienceComputer Science (R0)