[go: up one dir, main page]
Skip to main content

Advertisement

Springer Nature Link
Log in

Attention-based network for passive non-light-of-sight reconstruction in complex scenes

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Passive non-line-of-sight (NLOS) reconstruction has received considerable success in diverse fields. However, the existing reconstruction methods ignore that complex scenes attenuate object-related information and view object-related information and noise in measured images as equivalent, yielding low-quality recovery. We propose an attention-based encoder–decoder (AED) network to tackle this problem. Specifically, we introduce an attention in the attention (A2B) module that can prune the attention layers to help the network focus on the object-related information in the measured images. In addition, we establish several datasets in complex scenes, including varying ambient light conditions and parameter settings of reconstruction systems, as well as complex hidden objects, to verify the generalization of our method. Experiments on our constructed datasets demonstrate that our methods achieve better recovery performance than existing methods, with more robustness to complex scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data and Code Availability Statement

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

Notes

  1. Object-related information refers to properties like colour, texture, and shape of the hidden object in a measured image.

References

  1. Liu, X., Bauer, S., Velten, A.: Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems. Nat. Commun. 11(1), 1645 (2020)

    Article  Google Scholar 

  2. Isogawa, M., Yuan, Y., O’Toole, M., Kitani, K.M.: Optical non-line-of-sight physics-based 3d human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7013–7022 (2020)

  3. Metzler, C.A., Heide, F., Rangarajan, P., Balaji, M.M., Viswanath, A., Veeraraghavan, A., Baraniuk, R.G.: Deep-inverse correlography: towards real-time high-resolution non-line-of-sight imaging. Optica 7(1), 63–71 (2020)

    Article  Google Scholar 

  4. Willomitzer, F., Li, F., Balaji, M.M., Rangarajan, P., Cossairt, O.: High resolution non-line-of-sight imaging with superheterodyne remote digital holography. In: Computational Optical Sensing and Imaging, pp. 2–2. Optica Publishing Group, Columbia (2019)

  5. Saunders, C., Murray-Bruce, J., Goyal, V.K.: Computational periscopy with an ordinary digital camera. Nature 565(7740), 472–475 (2019)

    Article  Google Scholar 

  6. Saunders, C., Goyal, V.K.: Fast computational periscopy in challenging ambient light conditions through optimized preconditioning. In: 2021 IEEE International Conference on Computational Photography (ICCP), pp. 1–9, IEEE (2021)

  7. Yedidia, A.B., Baradad, M., Thrampoulidis, C., Freeman, W.T., Wornell, G.W.: Using unknown occluders to recover hidden scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12231–12239 (2019)

  8. Seidel, S.W., Ma, Y., Murray-Bruce, J., Saunders, C., Freeman, W.T., Christopher, C.Y., Goyal, V.K.: Corner occluder computational periscopy: estimating a hidden scene from a single photograph. In: 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1–9. IEEE (2019)

  9. Tanaka, K., Mukaigawa, Y., Kadambi, A.: Enhancing passive non-line-of-sight imaging using polarization cues (2019). arXiv preprint arXiv:1911.12906

  10. Seidel, S.W., Murray-Bruce, J., Ma, Y., Yu, C., Freeman, W.T., Goyal, V.K.: Two-dimensional non-line-of-sight scene estimation from a single edge occluder. IEEE Trans. Comput. Imaging 7, 58–72 (2020)

    Article  MathSciNet  Google Scholar 

  11. Chen, W., Daneau, S., Mannan, F., Heide, F.: Steady-state non-line-of-sight imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6790–6799 (2019)

  12. Bouman, K.L., Ye, V., Yedidia, A.B., Durand, F., Wornell, G.W., Torralba, A., Freeman, W.T.: Turning corners into cameras: Principles and methods. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2270–2278 (2017)

  13. Klein, J., Peters, C., Martín, J., Laurenzis, M., Hullin, M.B.: Tracking objects outside the line of sight using 2d intensity images. Sci. Rep. 6(1), 1–9 (2016)

    Article  Google Scholar 

  14. Mu, F., Mo, S., Peng, J., Liu, X., Nam, J.H., Raghavan, S., Velten, A., Li, Y.: Physics to the rescue: deep non-line-of-sight reconstruction for high-speed imaging. IEEE Trans. Pattern Anal. Mach. Intell. (2022). https://doi.org/10.1109/TPAMI.2022.3203383

    Article  Google Scholar 

  15. Wu, H., Liu, S., Meng, X., Yang, X., Yin, Y.: Non-line-of-sight imaging based on an untrained deep decoder network. Opt. Lett. 47(19), 5056–5059 (2022). https://doi.org/10.1364/OL.471319

    Article  Google Scholar 

  16. Zhou, C., Wang, C.-Y., Liu, Z.: Non-line-of-sight imaging off a phong surface through deep learning (2020). arXiv preprint arXiv:2005.00007

  17. Tingyi, Y., Mu, Q., Liu, H., Han, S.: Non-line-of-sight imaging through deep learning. Acta Optica Sinica 39(7), 0711002 (2019)

    Article  Google Scholar 

  18. Sun, L., Shi, J., Wu, X., Sun, Y., Zeng, G.: Photon-limited imaging through scattering medium based on deep learning. Opt. Express 27(23), 33120–33134 (2019)

    Article  Google Scholar 

  19. Geng, R., Hu, Y., Lu, Z., Yu, C., Li, H., Zhang, H., Chen, Y.: Passive non-line-of-sight imaging using optimal transport. IEEE Trans. Image Process. 31, 110–124 (2021)

    Article  Google Scholar 

  20. He, J., Wu, S., Wei, R., Zhang, Y.: Non-line-of-sight imaging and tracking of moving objects based on deep learning. Opt. Express 30(10), 16758–16772 (2022). https://doi.org/10.1364/OE.455803

    Article  Google Scholar 

  21. Sun, Y., Wu, X., Shi, J., Zeng, G.: Scattering-assisted computational imaging. In: Photonics, vol. 9, p. 512. MDPI (2022)

  22. Chen, X., Li, M., Chen, T., Zhan, S.: Long-range non-line-of-sight imaging based on projected images from multiple light fields. Photonics (2023). https://doi.org/10.3390/photonics10010025

    Article  Google Scholar 

  23. Zhu, S., Sua, Y.M., Bu, T., Huang, Y.-P.: Compressive non-line-of-sight imaging with deep learning. Phys. Rev. Appl. 19(3), 034090 (2023)

    Article  Google Scholar 

  24. Su, X., Hong, Y., Ye, J., Xu, F., Yuan, X.: Multi-scale iterative model-guided unfolding network for nlos reconstruction. In: Computer Graphics Forum, vol. 42 (2023)

  25. Peng, J., Xiong, Z., Tan, H., Huang, X., Li, Z.-P., Xu, F.: Boosting photon-efficient image reconstruction with a unified deep neural network. IEEE Trans. Pattern Anal. Mach. Intell. 45(4), 4180–4197 (2022)

    Google Scholar 

  26. Zhu, D., Cai, W.: Fast non-line-of-sight imaging with two-step deep remapping. ACS Photon. 9(6), 2046–2055 (2022)

    Article  Google Scholar 

  27. Huang, C., He, J., Wei, R., Weng, Y., Wang, W., Wang, C., Zhang, Y.: 45.2: high-performance non-line-of-sight imaging based on deep learning. In: SID Symposium Digest of Technical Papers, vol. 54, pp. 321–322. Wiley Online Library (2023)

  28. Sun, Y., Shi, J., Sun, L., Fan, J., Zeng, G.: Image reconstruction through dynamic scattering media based on deep learning. Opt. Express 27(11), 16032–16046 (2019). https://doi.org/10.1364/OE.27.016032

    Article  Google Scholar 

  29. Wang, Z., Huang, H., Li, H., Chen, Z., Han, J., Pu, J.: Non-line-of-sight imaging and location determination using deep learning. Opt. Lasers Eng. 169, 107701 (2023). https://doi.org/10.1016/j.optlaseng.2023.107701

    Article  Google Scholar 

  30. Chen, H., Gu, J., Zhang, Z.: Attention in attention network for image super-resolution (2021). arXiv preprint arXiv:2104.09497

  31. Zhao, H., Kong, X., He, J., Qiao, Y., Dong, C.: Efficient image super-resolution using pixel attention. In: European Conference on Computer Vision, pp. 56–72. Springer, Berlin (2020)

  32. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989). https://doi.org/10.1162/neco.1989.1.4.541

    Article  Google Scholar 

  33. Kumar, A., Deni Raj, E.: Silhouettes for human posture recognition. IEEE Dataport. [Online; accessed 06-08-2020] (2020). https://doi.org/10.21227/9c9b-3j44

  34. Aistudio: Hand gesture recognition dataset (2020). https://aistudio.baidu.com/aistudio/datasetdetail/51629. Online. Accessed 27 Aug. 2020

  35. tecperson, K.: Sign Language MNIST: Drop-In Replacement for MNIST for Hand Gesture Recognition Tasks (2017). https://www.kaggle.com/datamunge/sign-language-mnist. Online; Accessed 20 Oct. 2017

  36. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017). arXiv preprint arXiv:1708.07747

  37. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. University of Toronto (2009)

  38. Wang, Y., Zhang, Y., Huang, M., Chen, Z., Jia, Y., Weng, Y., Xiao, L., Xiang, X.: Accurate but fragile passive non-line-of-sight recognition. Commun. Phys. 4, 1–9 (2021)

    Article  Google Scholar 

  39. Kingma, D.P., Ba, J.: ADAM: a method for stochastic optimization (2014). arXiv preprint arXiv:1412.6980

  40. Ma, J., Yarats, D.: On the adequacy of untuned warmup for adaptive optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 8828–8836 (2021)

Download references

Acknowledgements

This research was funded by Hunan Provincial Innovation Foundation for Postgraduate of FUNDER grant number CX20220646. Huang’s research was partially supported by NSFC Project (11971410) and China’s National Key R &D Programs (2020YFA0713500).

Author information

Author notes

  1. Yaqin Zhang and Meiyu Huang have contributed equally to this work.

Authors and Affiliations

  1. Qian Xuesen Laboratory of Space Technology, China Academy of Space Technology, Beijing, 100094, People’s Republic of China

    Yaqin Zhang, Meiyu Huang, Yangyang Wang & Xueshuang Xiang

  2. School of Mathematics and Computational Science, Xiangtan University, Xiangtan, 411105, People’s Republic of China

    Yaqin Zhang & Yunqing Huang

  3. College of Mathematics and Physics, Beijing University of Chemical Technology, Beijing, 100029, People’s Republic of China

    Zhao Chen

Authors
  1. Yaqin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meiyu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yangyang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yunqing Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xueshuang Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yunqing Huang or Xueshuang Xiang.

Ethics declarations

Conflicts of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 964 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Huang, M., Wang, Y. et al. Attention-based network for passive non-light-of-sight reconstruction in complex scenes. Vis Comput 40, 8073–8083 (2024). https://doi.org/10.1007/s00371-023-03223-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03223-z

Keywords