[go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

Deep graph learning for spatially-varying indoor lighting prediction

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Lighting prediction from a single image is becoming increasingly important in many vision and augmented reality (AR) applications in which shading and shadow consistency between virtual and real objects should be guaranteed. However, this is a notoriously ill-posed problem, especially for indoor scenarios, because of the complexity of indoor luminaires and the limited information involved in 2D images. In this paper, we propose a graph learning-based framework for indoor lighting estimation. The core is a new lighting model (DSGLight) based on depth-augmented spherical Gaussians (SGs) and a graph convolutional network (GCN) that infers the new lighting representation from a single low dynamic range (LDR) image of limited field-of-view. Our lighting model builds 128 evenly distributed SGs over the indoor panorama, where each SG encodes the lighting and the depth around that node. The proposed GCN then learns the mapping from the input image to DSGLight. Compared with existing lighting models, our DSGLight encodes both direct lighting and indirect environmental lighting more faithfully and compactly. It also makes network training and inference more stable. The estimated depth distribution enables temporally stable shading and shadows under spatially-varying lighting. Through thorough experiments, we show that our method obviously outperforms existing methods both qualitatively and quantitatively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Einabadi F, Guillemaut J, Hilton A. Deep neural models for illumination estimation and relighting: a survey. Comput Graphics Forum, 2021, 40: 315–331

    Article  Google Scholar 

  2. Liu Y, Gevers T, Li X Q. Estimation of sunlight direction using 3D object models. IEEE Trans Image Process, 2015, 24: 932–942

    Article  MathSciNet  Google Scholar 

  3. Hold-Geoffroy Y, Sunkavalli K, Hadap S, et al. Deep outdoor illumination estimation. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2373–2382

  4. Zhang J, Sunkavalli K, Hold-Geoffroy Y, et al. All-weather deep outdoor lighting estimation. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 10150–10158

  5. Hold-Geoffroy Y, Athawale A, Lalonde J. Deep sky modeling for single image outdoor lighting estimation. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 6920–6928

  6. Yu P, Guo J, Huang F, et al. Hierarchical disentangled representation learning for outdoor illumination estimation and editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 15313–15322

  7. Barron J, Malik J. Intrinsic scene properties from a single RGB-D image. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2013. 38

  8. Maier R, Kim K, Cremers D, et al. Intrinsic3D: high-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting. In: Proceedings of IEEE International Conference on Computer Vision, Venice, 2017. 3133–3141

  9. Weber H, Prévost D, Lalonde J. Learning to estimate indoor lighting from 3D objects. In: Proceedings of International Conference on 3D Vision, Verona, 2018. 199–207

  10. Gardner M, Hold-Geoffroy Y, Sunkavalli K, et al. Deep parametric indoor lighting estimation. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision, Seoul, 2019. 7174–7182

  11. Gardner M A, Sunkavalli K, Yumer E, et al. Learning to predict indoor illumination from a single image. ACM Trans Graph, 2017, 36: 1–14

    Article  Google Scholar 

  12. Garon M, Sunkavalli K, Hadap S, et al. Fast spatially-varying indoor lighting estimation. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 6901–6910

  13. Song S, Funkhouser T A. Neural illumination: lighting prediction for indoor environments. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 2019. 6918–6926

  14. Srinivasan P P, Mildenhall B, Tancik M, et al. Lighthouse: predicting lighting volumes for spatially-coherent illumination. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, 2020. 8077–8086

  15. Zhan F, Zhang C, Yu Y, et al. Emlight: lighting estimation via spherical distribution approximation. In: Proceedings of the AAAI Conference on Artificial Intelligence, 2021

  16. Zhan F, Zhang C, Hu W, et al. Sparse needlets for lighting estimation with spherical transport loss. In: Proceedings of the IEEE International Conference on Computer Vision, 2021

  17. Cheng D, Shi J, Chen Y, et al. Learning scene illumination by pairwise photos from rear and front mobile cameras. Comput Graphics Forum, 2018, 37: 213–221

    Article  Google Scholar 

  18. Green R. Spherical harmonic lighting: The gritty details. 2003. https://api.semanticscholar.org/CorpusID:116856600

  19. Liu B, Xu K, Martin R R. Static scene illumination estimation from videos with applications. J Comput Sci Technol, 2017, 32: 430–442

    Article  Google Scholar 

  20. Tsai Y T, Shih Z C. All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation. ACM Trans Graph, 2006, 25: 967–976

    Article  Google Scholar 

  21. Wang J, Ren P, Gong M, et al. All-frequency rendering of dynamic, spatially-varying reflectance. ACM Trans Graph, 2009, 28: 133

    Article  Google Scholar 

  22. Xu K, Sun W L, Dong Z, et al. Anisotropic spherical Gaussians. ACM Trans Graph, 2013, 32: 1–11

    Google Scholar 

  23. Wu C, Wilburn B, Matsushita Y, et al. High-quality shape from multi-view stereo and shading under general illumination. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2011. 969–976

  24. Karsch K, Hedau V, Forsyth D, et al. Rendering synthetic objects into legacy photographs. ACM Trans Graph, 2011, 30: 1–12

    Article  Google Scholar 

  25. Khan E A, Reinhard E, Fleming R W, et al. Image-based material editing. ACM Trans Graph, 2006, 25: 654–663

    Article  Google Scholar 

  26. Wei X, Chen G, Dong Y, et al. Object-based illumination estimation with rendering-aware neural networks. In: Proceedings of the 16th European Conference on Computer Vision, Glasgow, 2020. 380–396

  27. Atwood J, Towsley D. Diffusion-convolutional neural networks. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, 2015. 2001–2009

  28. Micheli A. Neural network for graphs: a contextual constructive approach. IEEE Trans Neural Netw, 2009, 20: 498–511

    Article  Google Scholar 

  29. Niepert M, Ahmed M, Kutzkov K. Learning convolutional neural networks for graphs. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning, 2016. 2014–2023

  30. Defferrard M, Bresson X, Vandergheynst P. Convolutional neural networks on graphs with fast localized spectral filtering. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, Red Hook, 2016. 3844–3852

  31. Hamilton W L, Ying R, Leskovec J. Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017. 1025–1035

  32. Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks. In: Proceedings of the 5th International Conference on Learning Representations, Toulon, 2017

  33. Shuman D I, Narang S K, Frossard P, et al. The emerging field of signal processing on graphs: extending high-dimensional data analysis to networks and other irregular domains. IEEE Signal Process Mag, 2013, 30: 83–98

    Article  Google Scholar 

  34. Bastings J, Titov I, Aziz W, et al. Graph convolutional encoders for syntax-aware neural machine translation. 2017. ArXiv:1704.04675

  35. Marcheggiani D, Titov I. Encoding sentences with graph convolutional networks for semantic role labeling. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017

  36. Yan S, Xiong Y, Lin D. Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence and 30th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence, 2018. 7444–7452

  37. Li Y, Ouyang W, Zhou B, et al. Factorizable net: an efficient subgraph-based framework for scene graph generation. In: Proceedings of European Conference on Computer Vision, 2018. 346–363

  38. Qi X, Liao R, Jia J, et al. 3D graph neural networks for RGBD semantic segmentation. In: Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV), 2017. 5209–5218

  39. Yang J, Lu J, Lee S, et al. Graph R-CNN for scene graph generation. 2018. ArXiv:1808.00191

  40. Xiao J, Ehinger K A, Oliva A, et al. Recognizing scene viewpoint using panoramic place representation. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, 2012. 2695–2702

  41. Eilertsen G, Kronander J, Denes G, et al. HDR image reconstruction from a single exposure using deep CNNs. ACM Trans Graph, 2017, 36: 1–15

    Article  Google Scholar 

  42. Nimier-David M, Vicini D, Zeltner T, et al. Mitsuba 2: a retargetable forward and inverse renderer. ACM Tran Graph, 2019, 38: 1–17

    Article  Google Scholar 

  43. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations, San Diego, 2015

  44. Li G, Müller M, Thabet A K, et al. DeepGCNs: can GCNs go as deep as CNNs? In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision, Seoul, 2019. 9266–9275

  45. Li Z, Shafiei M, Ramamoorthi R, et al. Inverse rendering for complex indoor scenes: shape, spatially-varying lighting and SVBRDF from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 2475–2484

  46. Abadi M, Agarwal A, Barham P, et al. TensorFlow: large-scale machine learning on heterogeneous systems, 2015. arXiv:1603.04467

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant Nos. 62032011, 61972194).

Author information

Authors and Affiliations

  1. State Key Lab for Novel Software Technology, Nanjing University, Nanjing, 210023, China

    Jiayang Bai, Jie Guo, Chenchen Wang, Zhenyu Chen, Zhen He, Shan Yang, Piaopiao Yu, Yan Zhang & Yanwen Guo

Authors
  1. Jiayang Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenchen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhenyu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhen He
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Piaopiao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Yan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yanwen Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jie Guo or Yanwen Guo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bai, J., Guo, J., Wang, C. et al. Deep graph learning for spatially-varying indoor lighting prediction. Sci. China Inf. Sci. 66, 132106 (2023). https://doi.org/10.1007/s11432-022-3576-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-022-3576-9

Keywords

Search

Navigation