[go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

A joint convolution auto-encoder network for infrared and visible image fusion

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

    We’re sorry, something doesn't seem to be working properly.

    Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Background: Leaning redundant and complementary relationships is a critical step in the human visual system. Inspired by the infrared cognition ability of crotalinae animals, we design a joint convolution auto-encoder (JCAE) network for infrared and visible image fusion. Methods: Our key insight is to feed infrared and visible pair images into the network simultaneously and separate an encoder stream into two private branches and one common branch, the private branch works for complementary features learning and the common branch does for redundant features learning. We also build two fusion rules to integrate redundant and complementary features into their fused feature which are then fed into the decoder layer to produce the final fused image. We detail the structure, fusion rule and explain its multi-task loss function. Results: Our JCAE network achieves good results in terms of both visual quality and objective evaluation metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Bertinetto L, Valmadre J, Henriques JF, Vedaldi A, Torr PHS (2016) Fully-convolutional siamese networks for object tracking. arXiv:1606.09549

  2. Chen M (2016) Image fusion of visual and infrared image based on nsct and compressed sensing. J Image Graph 21(1):0039–0044

    MathSciNet  Google Scholar 

  3. Deshmukh M, Bhosale U (2010) Image fusion and image quality assessment of fused images. Int J Image Process 4(5):484–508

    Google Scholar 

  4. Dong C, Chen C, Tang X (2016) Accelerating the super-resolution convolutional neural network. In: European conference on computer vision, pp 391–407

  5. Epstein B, Meir R, Michaeli T (2017) Joint auto-encoders: a flexible multi-task learning framework. In: ArXiv e-prints

  6. Feng Y, Chen F, Ji Y-M, Wu F, Sun J (2021) Efficient cross-modality graph reasoning for rgb-infrared person re-identification. IEEE Signal Process Lett 28:1425–1429

    Article  Google Scholar 

  7. Han Y, Cai Y, Cao Y, Xu X (2013) A new image fusion performance metric based on visual information fidelity. Inf Fusion 14(2):127–135

    Article  Google Scholar 

  8. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  9. Li Q, Li J, Li Z (2009) Image fusion based on nonsubsampled contourlet transform and evaluation. Appl Res Comput 26(3):1138–1134

    MathSciNet  Google Scholar 

  10. Li S, Kang X, Hu J (2013) Image fusion with guided filtering. IEEE Trans Image Process 22(7):2864–2875

    Article  Google Scholar 

  11. Li H, Wu X (2018) Densefuse: a fusion approach to infrared and visible images. IEEE transactions on image processing: a publication of the IEEE Signal Processing Society

  12. Liu Z, Blasch E, Xue Z, Zhao J, Laganiere R, Wu W (2011) Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Trans Pattern Anal Mach Intell 34 (1):94–109

    Article  Google Scholar 

  13. Liu Y, Liu S, Wang Z (2015) A general framework for image fusion based on multi-scale transform and sparse representation. Inf Fusion 24:147–164

    Article  Google Scholar 

  14. Liu Y, Chen X, Ward RK, Wang ZJ (2016) Image fusion with convolutional sparse representation. IEEE Signal Process Lett 23(12):1882–1886

    Article  Google Scholar 

  15. Liu Y, Chen X, Peng H, Wang Z (2017) Multi-focus image fusion with a deep convolutional neural network. Inf Fusion 36:191–207

    Article  Google Scholar 

  16. Luo X, Zhang Z, Wu X (2016) A novel algorithm of remote sensing image fusion based on shift-invariant shearlet transform and regional selection. AEU-Int J Electron Commun 70(2):186–197

    Article  Google Scholar 

  17. Luo X, Zhang Z, Zhang B, Wu X-J (2017) Image fusion with contextual statistical similarity and nonsubsampled shearlet transform. IEEE Sensors J 17(6):1760–1771

    Article  Google Scholar 

  18. Luo X, Gao Y, Wang A, Zhang Z, Wu X-J (2021) Ifsepr: a general framework for image fusion based on separate representation learning. IEEE Trans Multimed:1–1

  19. Luo X, Wang A, Zhang Z, Xiang X, Wu X-J (2021) Latraivf: an infrared and visible image fusion method based on latent regression and adversarial training. IEEE Trans Instrum Meas 70:1–16

    Google Scholar 

  20. Ma J, Chen C, Li C, Huang J (2016) Infrared and visible image fusion via gradient transfer and total variation minimization. Inf Fusion 31(C):100–109

    Article  Google Scholar 

  21. Ma J, Ma Y, Li C (2018) Infrared and visible image fusion methods and applications: a survey. Inf Fusion 45:153–178

    Article  Google Scholar 

  22. Ma J, Yu W, Liang P, Li C, Jiang J (2019) Fusiongan: a generative adversarial network for infrared and visible image fusion. Inf Fusion 48:11–26

    Article  Google Scholar 

  23. Pang H, Zhu M, Guo L (2012) Multifocus color image fusion using quaternion wavelet transform. In: International congress on image and signal processing, pp 543–546

  24. Prabhakar KR, Srikar VS, Babu RV (2017) Deepfuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: IEEE international conference on computer vision, pp 4724–4732

  25. Qi C, Ye W, Miao B, Qi G (2008) Fusion algorithm of infrared and visible light images based on nsct transform. Syst Eng Electron 30(4):593–596

    Google Scholar 

  26. Shelhamer E, Long J, Darrell T (2014) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 39(4):1–1

    Google Scholar 

  27. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. In: Computer science

  28. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Bengio Y, LeCun Y (eds) 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  29. Toet A, Hogervorst MA (2012) Progress in color night vision. Opt Eng 51(1):1–20

    Article  Google Scholar 

  30. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84

    Article  Google Scholar 

  31. Wang W, Chang F (2011) A multi-focus image fusion method based on laplacian pyramid. J Comput 6(12):2559–2566

    Article  Google Scholar 

  32. Wu F, Jing X-Y, Dong X, Hu R, Yue D, Wang L, Ji Y-M, Wang R, Chen G (2020) Intraspectrum discrimination and interspectrum correlation analysis deep network for multispectral face recognition. IEEE Trans Cybern 50 (3):1009–1022

    Article  Google Scholar 

  33. Wu F, Jing X-Y, Feng Y, Ji Y-M, Wang R (2021) Spectrum-aware discriminative deep feature learning for multi-spectral face recognition. Pattern Recognit 111(2):107632

    Article  Google Scholar 

  34. Xu H, Ma J, Jiang J, Guo X, Ling H (2020) U2fusion: a unified unsupervised image fusion network. IEEE Trans Pattern Anal Mach Intell

  35. Yu Z, Yu L, Sun P, Yan H, Zhao X, Zhang L (2020) Ifcnn: a general image fusion framework based on convolutional neural network. Inf Fusion 54:99–118

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Suzhou University of Science and Technology, Suzhou, China

    Zhancheng Zhang

  2. Jiangnan University, Wuxi, China

    Yuanhao Gao, Mengyu Xiong, Xiaoqing Luo & Xiao-Jun Wu

Authors
  1. Zhancheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuanhao Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mengyu Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoqing Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiao-Jun Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoqing Luo.

Ethics declarations

This study was funded by the National Natural Science Foundation of P. R. China (grant number 61772237) and the Six Talent Peaks Project in Jiangsu Province (grant numbe XYDXX-030).

Ethics approval and consent to participate

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Gao, Y., Xiong, M. et al. A joint convolution auto-encoder network for infrared and visible image fusion. Multimed Tools Appl 82, 29017–29035 (2023). https://doi.org/10.1007/s11042-023-14758-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14758-7

Keywords