[go: up one dir, main page]
Skip to main content
SpringerLink
Log in

A multi-color and multistage collaborative network guided by refined transmission prior for underwater image enhancement

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Due to the attenuation and scattering properties of light in underwater scenes, underwater images are generally subject to color deviations and low contrast, which is not conducive to the follow-up algorithms. To alleviate these two problems, we propose a multi-color and multistage collaborative network guided by refined transmission, called MMCGT, to accomplish the enhancement tasks. Specifically, we first design an accurate method of parameter estimation to derive transmission priors that are more suitable for underwater imaging, such as min–max conversion, low-pass filter-based estimation and saturation detection. Then, we propose a multistage and multi-color space collaborative network to decompose the underwater image enhancement task into more straightforward and controllable subtasks, including colorful feature extraction, color deviation detection, and image position information retention. Finally, we apply the derived transmission prior to the transmission-guided block of the network and effectively combine the well-designed physical-inconsistency loss with Charbonnier loss and VGG loss to guide the MMCGT to compensate for the quality-degraded regions better. Extensive experiments show that MMCGT achieves better evaluation results under the dual guidance of physics and deep learning than the competing methods in visual quality and quantitative metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The underwater image datasets used in the paper are all publicly accessible. The following are links to relevant datasets. The training dataset and the Color-Check7 dataset: https://github.com/Li-Chongyi/Ucolor. The Test-R90 dataset and the Test-C60 dataset: https://li-chongyi.github.io/proj_benchmark.html. The Test-S1000 dataset: https://github.com/saeed-anwar/UWCNN. The SQUID dataset: http://csms.haifa.ac.il/profiles/tTreibitz/datasets/ambient_forwardlooking/index.html. The Video-Test dataset: https://github.com/dlut-dimt/Realworld-Underwater-Image-Enhancement-RUIE-Benchmark.

References

  1. Ancuti, C.O., Ancuti, C., De Vleeschouwer, C., Bekaert, P.: Color balance and fusion for underwater image enhancement. IEEE Trans. Image Process. 27, 379–393 (2017)

    Article  MathSciNet  Google Scholar 

  2. Berman, D., Levy, D., Avidan, S., Treibitz, T.: Underwater single image color restoration using haze-lines and a new quantitative dataset. IEEE Trans. Pattern Anal. Mach. Intell. 43, 2822–2837 (2020)

    Google Scholar 

  3. Bouguer, P.: Essai d’optique, sur la gradation de la lumiere. Claude Jombert (1729)

  4. Charbonnier, P., Blanc-Feraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of 1st International Conference on Image Processing pp. 168–172. IEEE (1994)

  5. Chiang, J.Y., Chen, Y.C.: Underwater image enhancement by wavelength compensation and dehazing. IEEE Trans. Image Process. 21, 1756–1769 (2011)

    Article  MathSciNet  Google Scholar 

  6. Daw, A., Karpatne, A., Watkins, W.D., Read, J.S., Kumar, V.: Physics-guided neural networks (pgnn): an application in lake temperature modeling. In: Knowledge-Guided Machine Learning. pp. 353–372. Chapman and Hall/CRC (2017)

  7. Drews, P.L., Nascimento, E.R., Botelho, S.S., Campos, M.F.M.: Underwater depth estimation and image restoration based on single images. IEEE Comput. Graphics Appl. 36, 24–35 (2016)

    Article  Google Scholar 

  8. Fu, Z., Wang, W., Huang, Y., Ding, X., Ma, K.K.: Uncertainty inspired underwater image enhancement. arXiv preprint arXiv:2207.09689 (2022)

  9. Galdran, A., Pardo, D., Picón, A., Alvarez-Gila, A.: Automatic red-channel underwater image restoration. J. Vis. Commun. Image Represent. 26, 132–145 (2015)

    Article  Google Scholar 

  10. Golts, A., Freedman, D., Elad, M.: Unsupervised single image dehazing using dark channel prior loss. IEEE Trans. Image Process. 29, 2692–2701 (2019)

    Article  Google Scholar 

  11. Guo, Y., Li, H., Zhuang, P.: Underwater image enhancement using a multiscale dense generative adversarial network. IEEE J. Ocean. Eng. 45, 862–870 (2019)

    Article  Google Scholar 

  12. He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2341–2353 (2010)

    Google Scholar 

  13. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1397–1409 (2012)

    Article  Google Scholar 

  14. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7132–7141. (2018)

  15. Iqbal, K., Odetayo, M., James, A., Salam, R.A., Talib, A.Z.H.: Enhancing the low quality images using unsupervised colour correction method. In: 2010 IEEE International Conference on Systems, pp. 1703–1709. Man and Cybernetics, IEEE (2010)

  16. Islam, M.J., Xia, Y., Sattar, J.: Fast underwater image enhancement for improved visual perception. IEEE Robot. Autom. Lett. 5, 3227–3234 (2020)

    Article  Google Scholar 

  17. Jerlov, N.G.: Marine Optics. Elsevier, Amsterdam (1976)

    Google Scholar 

  18. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision. pp. 694–711. Springer (2016)

  19. Lai, Y., Xu, H., Lin, C., Luo, T., Wang, L.: A two-stage and two-branch generative adversarial network-based underwater image enhancement. Vis. Comput. 1–15 (2022)

  20. Li, C., Anwar, S., Hou, J., Cong, R., Guo, C., Ren, W.: Underwater image enhancement via medium transmission-guided multi-color space embedding. IEEE Trans. Image Process. 30, 4985–5000 (2021)

    Article  Google Scholar 

  21. Li, C., Anwar, S., Porikli, F.: Underwater scene prior inspired deep underwater image and video enhancement. Pattern Recogn. 98, 107038 (2020)

    Article  Google Scholar 

  22. Li, C., Guo, C., Ren, W., Cong, R., Hou, J., Kwong, S., Tao, D.: An underwater image enhancement benchmark dataset and beyond. IEEE Trans. Image Process. 29, 4376–4389 (2019)

    Article  Google Scholar 

  23. Li, C., Guo, J., Guo, C.: Emerging from water: underwater image color correction based on weakly supervised color transfer. IEEE Signal Process. Lett. 25, 323–327 (2018)

    Article  Google Scholar 

  24. Li, J., Skinner, K.A., Eustice, R.M., Johnson-Roberson, M.: Watergan: unsupervised generative network to enable real-time color correction of monocular underwater images. IEEE Robot. Autom. Lett. 3, 387–394 (2017)

    Google Scholar 

  25. Li, Z., Wei, Z., Wen, C., Zheng, J.: Detail-enhanced multi-scale exposure fusion. IEEE Trans. Image Process. 26, 1243–1252 (2017)

    Article  MathSciNet  Google Scholar 

  26. Lin, R., Liu, J., Liu, R., Fan, X.: Global structure-guided learning framework for underwater image enhancement. Vis. Comput. 1–16 (2021)

  27. Liu, H., Yang, J., Wu, Z., Zhang, Q., Deng, Y.: A fast single image dehazing method based on dark channel prior and retinex theory. Acta Automat. Sin. 41, 1264–1273 (2015)

    Google Scholar 

  28. Liu, R., Fan, X., Zhu, M., Hou, M., Luo, Z.: Real-world underwater enhancement: challenges, benchmarks, and solutions under natural light. IEEE Trans. Circuits Syst. Video Technol. 30, 4861–4875 (2020)

    Article  Google Scholar 

  29. Loshchilov, I., Hutter, F.: Sgdr: stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)

  30. Mhala, N.C., Pais, A.R.: A secure visual secret sharing (vss) scheme with cnn-based image enhancement for underwater images. Vis. Comput. 37, 2097–2111 (2021)

    Article  Google Scholar 

  31. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind’’ image quality analyzer. IEEE Signal Process. Lett. 20, 209–212 (2012)

    Article  Google Scholar 

  32. Monika, R., Samiappan, D., Kumar, R.: Underwater image compression using energy based adaptive block compressive sensing for iout applications. Vis. Comput. 37, 1499–1515 (2021)

    Article  Google Scholar 

  33. Naik, A., Swarnakar, A., Mittal, K.: Shallow-uwnet: compressed model for underwater image enhancement (student abstract). In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 15853–15854. (2021)

  34. Naik, S.K., Murthy, C.: Hue-preserving color image enhancement without gamut problem. IEEE Trans. Image Process. 12, 1591–1598 (2003)

    Article  Google Scholar 

  35. Panetta, K., Gao, C., Agaian, S.: Human-visual-system-inspired underwater image quality measures. IEEE J. Oceanic Eng. 41, 541–551 (2015)

    Article  Google Scholar 

  36. Pang, Y., Wu, C., Wu, H., Yu, X.: Over-sampling strategy-based class-imbalanced salient object detection and its application in underwater scene. Vis. Comput. 1–16 (2022)

  37. Peng, L., Zhu, C., Bian, L.: U-shape transformer for underwater image enhancement. IEEE Trans. Image Process. (2023)

  38. Peng, Y.T., Cao, K., Cosman, P.C.: Generalization of the dark channel prior for single image restoration. IEEE Trans. Image Process. 27, 2856–2868 (2018)

    Article  MathSciNet  Google Scholar 

  39. Peng, Y.T., Cosman, P.C.: Single image restoration using scene ambient light differential. In: 2016 IEEE International Conference on Image Processing (ICIP). pp. 1953–1957. IEEE (2016)

  40. Qiao, N., Di, L.: Underwater image enhancement combining low-dimensional and global features. Vis. Comput. 1–11 (2022)

  41. Sharma, G., Wu, W., Dalal, E.N.: The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain). Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur, vol. 30, pp. 21–30. (2005)

  42. Tang, Y., Kawasaki, H., Iwaguchi, T.: Underwater image enhancement by transformer-based diffusion model with non-uniform sampling for skip strategy. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 5419–5427. (2023)

  43. Tian, C., Xu, Y., Zuo, W., Du, B., Lin, C.W., Zhang, D.: Designing and training of a dual cnn for image denoising. Knowl.-Based Syst. 226, 106949 (2021)

    Article  Google Scholar 

  44. Tian, C., Xu, Y., Zuo, W., Lin, C.W., Zhang, D.: Asymmetric cnn for image superresolution. IEEE Trans. Syst. Man Cybern. Syst. 52, 3718–3730 (2021)

    Article  Google Scholar 

  45. Tian, C., Xu, Y., Zuo, W., Zhang, B., Fei, L., Lin, C.W.: Coarse-to-fine cnn for image super-resolution. IEEE Trans. Multimed. 23, 1489–1502 (2020)

    Article  Google Scholar 

  46. Tian, C., Zhuge, R., Wu, Z., Xu, Y., Zuo, W., Chen, C., Lin, C.W.: Lightweight image super-resolution with enhanced cnn. Knowl.-Based Syst. 205, 106235 (2020)

    Article  Google Scholar 

  47. Wang, Y., Guo, J., Gao, H., Yue, H.: Uiec\({}^{\hat{\,}}\) 2-net: Cnn-based underwater image enhancement using two color space. Signal Process. Image Commun. 96, 116250 (2021)

    Article  Google Scholar 

  48. Yang, M., Sowmya, A.: An underwater color image quality evaluation metric. IEEE Trans. Image Process. 24, 6062–6071 (2015)

    Article  MathSciNet  Google Scholar 

  49. Yuan, W., Fu, C., Liu, R., Fan, X.: Ssob: searching a scene-oriented architecture for underwater object detection. Vis. Comput. 1–10 (2022)

  50. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14821–14831. (2021)

  51. Zhang, J., Cao, Y., Fang, S., Kang, Y., Wen Chen, C.: Fast haze removal for nighttime image using maximum reflectance prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7418–7426. (2017)

  52. Zhang, K., Zhu, D., Li, J., Gao, X., Gao, F., Lu, J.: Learning stacking regression for no-reference super-resolution image quality assessment. Signal Process. 178, 107771 (2021)

    Article  Google Scholar 

  53. Zhang, S., Yu, D., Zhou, Y., Wu, Y., Ma, Y.: Enhanced visual perception for underwater images based on multistage generative adversarial network. Vis. Comput. 1–13 (2022)

Download references

Funding

This author funded by Natural science research project of Guizhou Provincial Department of Education (QianJiaoJi[2022]029, QianJiaoHeKY[2021]022), Key Disciplines of Guizhou Province-Computer Science and Technology (ZDXK [2018]007).

Author information

Authors and Affiliations

  1. State Key Laboratory of Public Big Data, Institute for Artificial Intelligence, College of Computer Science and Technology, Guizhou University, Guiyang, 550025, Guizhou, China

    Ting Ouyang, Yongjun Zhang, Haoliang Zhao, Yitong Yang & Yujie Xu

  2. School of Mathematics and Big Datam, GuiZhou Education University, Guiyang, 550001, Guizhou, China

    Zhongwei Cui

Authors
  1. Ting Ouyang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongjun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haoliang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhongwei Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yitong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yujie Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yongjun Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interest or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

According to the definition of the ambient light(Eq. (9)) and the underwater image(Eq. (8)) in the above text, we can easily get the value range of \(t\left( \lambda \right) _{\text {rough}}^{d\left( x \right) }\) as follows:

$$\begin{aligned} \begin{aligned} 0 \le \max \begin{pmatrix} \displaystyle \max _{y\in \Omega \left( x \right) }\frac{I_{r}\left( y \right) -B_{r}\left( y \right) }{1-B_{r}\left( y \right) }, \\ \displaystyle \max _{y\in \Omega \left( x \right) }\frac{B_{g}\left( y \right) -I_{g}\left( y \right) }{B_{g}\left( y \right) }, \\ \displaystyle \max _{y\in \Omega \left( x \right) }\frac{B_{b}\left( y \right) -I_{b}\left( y \right) }{B_{b}\left( y \right) }. \end{pmatrix} \le 1 \end{aligned} \end{aligned}$$
(A.1)

for each scenic spot \(x \in \left[ 0,1 \right] \), they have the following relationship with the ambient light:

$$\begin{aligned} \begin{aligned} B_{\lambda }\left( x \right) -I_{\lambda }\left( x \right) =E_{\lambda }\left( x \right) \cdot t_{\lambda }\left( x \right) ^{d\left( x \right) }\cdot \left( 1-\rho _{\lambda }\left( x \right) \right) \end{aligned} \end{aligned}$$
(A.2)

where \(E_{\lambda }\left( x \right) \) is the total incident light of the scenic point x, and its value is always \(\ge 0\). \(t_{\lambda }\left( x \right) ^{d\left( x \right) }\) is the transmission that we need to derive. According to the Beer-Lambert law [3], its equivalent formula is \(t_{\lambda }\left( x \right) ^{d\left( x \right) }=e^{-\beta d\left( x \right) }\), where \(d\left( x \right) \) is the distance from the camera to the radiant object and \(\beta \) is the light attenuation coefficient, so \(t_{\lambda }\left( x \right) ^{d\left( x \right) } \ge 0\). \(\rho _{\lambda }\left( x \right) \) is the reflectivity of a scene point, and its value falls in \(\left[ 0,1 \right] \). Therefore, we can draw the following conclusion:

$$\begin{aligned} \begin{aligned} B_{\lambda }\left( x \right) \ge I_{\lambda }\left( x \right) \end{aligned} \end{aligned}$$
(A.3)

In order to facilitate understanding, we convert Eq. (A.1) into Eq. (A.4):

$$\begin{aligned} \begin{aligned}&\max \begin{pmatrix} \displaystyle \max _{y\in \Omega \left( x \right) }\frac{I_{r}\left( y \right) -B_{r}\left( y \right) }{1-B_{r}\left( y \right) }, \\ \displaystyle \max _{y\in \Omega \left( x \right) }\frac{B_{g}\left( y \right) -I_{g}\left( y \right) }{B_{g}\left( y \right) }, \\ \displaystyle \max _{y\in \Omega \left( x \right) }\frac{B_{b}\left( y \right) -I_{b}\left( y \right) }{B_{b}\left( y \right) }.\end{pmatrix}\\&\quad = \max \begin{pmatrix} \displaystyle \max _{y\in \Omega \left( x \right) }1-\frac{1-I_{r}\left( y \right) }{1-B_{r}\left( y \right) }, \\ \displaystyle \max _{y\in \Omega \left( x \right) }1-\frac{I_{g}\left( y \right) }{B_{g}\left( y \right) }, \\ \displaystyle \max _{y\in \Omega \left( x \right) }1-\frac{I_{b}\left( y \right) }{B_{b}\left( y \right) }. \end{pmatrix} \end{aligned} \end{aligned}$$
(A.4)

Obviously, for g channel and b channel, their values are always in \(\left[ 0,1 \right] \). But for the r channel, we need to discuss it further. For any scenic spot \(x^{*}\), we mark its corresponding ambient light point as \(x_{0}\). It is clear that:

$$\begin{aligned} \begin{aligned} \max _{y\in \Omega \left( x^{*} \right) }\left( I_{r}\left( y \right) -B_{r}\left( y \right) \right) \le \max _{y\in \Omega \left( x_{0} \right) }\left( I_{r}\left( y \right) -B_{r}\left( y \right) \right) \end{aligned} \end{aligned}$$
(A.5)

Based on the fact that \(x_{0}\in \Omega \left( x_{0} \right) \), Eq. (A.5) can be further written as:

$$\begin{aligned} \begin{aligned} \max _{y\in \Omega \left( x^{*} \right) }\left( I_{r}\left( y \right) -B_{r}\left( y \right) \right) \le I_{r}\left( x_{0} \right) -B_{r}\left( x_{0} \right) \le 1-B_{r}\left( x_{0} \right) \end{aligned} \nonumber \\ \end{aligned}$$
(A.6)

Due to the serious attenuation of the light in the red band underwater, there is little residual energy to reach underwater scene, so \(0< 1-B_{r}\left( x \right) \le 1\), and we can deduce that:

$$\begin{aligned} \begin{aligned} \frac{\max _{y\in \Omega \left( x^{*} \right) }\left( I_{r}\left( y \right) -B_{r}\left( y \right) \right) }{1-B_{r}\left( x_{0} \right) }\le 1 \end{aligned} \end{aligned}$$
(A.7)

In addition, due to \(x^{*}\) and \(x_{0}\) are one-to-one correspondences, Eq. (A.8) is equivalent to Eq. (A.7):

$$\begin{aligned} \begin{aligned} \max _{y\in \Omega \left( x^{*} \right) }\frac{I_{r}\left( y \right) -B_{r}\left( y \right) }{1-B_{r}\left( y \right) }\le 1 \end{aligned} \end{aligned}$$
(A.8)

To sum up, when the maximum value of Eq. (A.1) falls in r channel, the value range of Eq. (A.1) is as follows:

$$\begin{aligned} \begin{aligned}&0 \le \max \begin{pmatrix} \displaystyle \max _{y\in \Omega \left( x \right) }\frac{B_{g}\left( y \right) -I_{g}\left( y \right) }{B_{g}\left( y \right) }, \\ \displaystyle \max _{y\in \Omega \left( x \right) }\frac{B_{b}\left( y \right) -I_{b}\left( y \right) }{B_{b}\left( y \right) }. \end{pmatrix} \\&\quad \le \max _{y\in \Omega \left( x^{*} \right) }\frac{I_{r}\left( y \right) -B_{r}\left( y \right) }{1-B_{r}\left( y \right) } \le 1 \end{aligned} \end{aligned}$$
(A.9)

According to the above discussion, the convergence of Eq. (8) is verified.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ouyang, T., Zhang, Y., Zhao, H. et al. A multi-color and multistage collaborative network guided by refined transmission prior for underwater image enhancement. Vis Comput (2024). https://doi.org/10.1007/s00371-023-03215-z

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-023-03215-z

Keywords

Search

Navigation