[go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

BCMask: a finer leaf instance segmentation with bilayer convolution mask

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Whether in natural scenes or laboratory environments, leaf instance segmentation is still a challenging task in high-throughput plant phenotypic research. Because compared with normal instance objects, leaves have more complex boundaries and severe inter-leaf occlusions. In this paper, we present an effective two-stage method called Bilayer Convolution Mask (BCMask) for high-quality leaf instance segmentation. BCMask consists of three main modules: (1) Bottom-up Path Augmentation (BPA) module is added after Feature Pyramid Network (FPN) in Faster R-CNN. BPA shortens the information path between lower layers and high-level layers, and helps accurate semantical features in lower layers to enhance the entire feature hierarchy; (2) Bilayer Occlusion Module. This module consists of two convolutional layers with a residual structure, which decouples the occluding leaves and the partially occluded target leaf during the mask regression; (3) Mask Refining Module. This module uses an iterative refinement method with adaptive selection to classify pixels, which effectively alleviates the problem of inaccurate leaf boundary segmentation. To validate BCMask, this paper takes the chrysanthemum seedling leaf dataset for experiment, which is collected in the natural environment with complex boundaries and severe occlusions. Two remarkable public datasets CVPPA and Komatsuna under laboratory environments are also added as supplements to validate the robustness of BCMask. The proposed method achieves the 60.42% average precision (AP) score outperforming state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig.12
Fig.13
Fig.14

Similar content being viewed by others

References

  1. Zhao, C., et al.: Crop phenomics: current status and perspectives. Front. Plant Sci. 10, 714 (2019)

    Article  Google Scholar 

  2. Wang, Z., Cui, J., Zhu, Y.: Plant recognition based on Jaccard distance and BOW. Multimedia Syst. 26, 495–508 (2020)

    Article  Google Scholar 

  3. Kim, D., Kim, J.: Procedural modeling and visualization of multiple leaves. Multimedia Syst. 23, 435–449 (2017)

    Article  Google Scholar 

  4. Mccormick, R.F., Truong, S.K., Mullet, J.E.: 3D sorghum reconstructions from depth images identify QTL regulating shoot architecture. Plant Physiol. (2016). https://doi.org/10.1104/pp.16.00948

    Article  Google Scholar 

  5. Scharr, H., et al.: Leaf segmentation in plant phenotyping: a collation study. Mach. Vis. Appl. 27, 585–606 (2016)

    Article  Google Scholar 

  6. Wang, Z., Wang, K., Yang, F., Pan, S., Han, Y.: Image segmentation of overlapping leaves based on Chan-Vese model and Sobel operator. Inf. Process. Agric. 5, 1–10 (2018)

    Google Scholar 

  7. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: ICCV 2017, Venice, Italy, October 22–29, 2017, pp. 2980–2988 (2017)

  8. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39, 1137–1149 (2017)

    Article  Google Scholar 

  9. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: ECCV 2014, Zurich, Switzerland, September 6–12, 2014, pp. 740–755 (2014)

  10. Cordts, M., et al.: the cityscapes dataset for semantic urban scene understanding. In: CVPR 2016, Las Vegas, NV, USA, June 26–July 1, pp. 3213–3223 (2016)

  11. Neuhold, G., Ollmann, T., Bulò, S.R., Kontschieder, P.: The mapillary vistas dataset for semantic understanding of street scenes. In ICCV 2017, Venice, Italy, October 22–29, 2017, pp. 5000–5009 (2017)

  12. Triki, A., Bouaziz, B., Gaikwad, J., Mahdi, W.: Deep leaf: mask R-CNN based leaf detection and segmentation from digitized herbarium specimen images. Pattern Recogn. Lett. 150, 76–83 (2021)

    Article  Google Scholar 

  13. Yang, X., et al.: Instance segmentation and classification method for plant leaf images based on ISC-MRCNN and APS-DCCNN. IEEE Access 8, 151555–151573 (2020)

    Article  Google Scholar 

  14. Zeiler, M. D., Fergus, R.: Visualizing and understanding convolutional networks. In: ECCV 2014, Zurich, Switzerland, September 6–12, 2014, pp. 818–833 (2014)

  15. He, Y., He, N., Zhang, R., Yan, K., Yu, H.: Multi-scale feature balance enhancement network for pedestrian detection. Multimedia Syst. 28, 1135–1145 (2022)

    Article  Google Scholar 

  16. Wang, H., Song, Y., Huo, L., Chen, L., He, Q.: Multiscale object detection based on channel and data enhancement at construction sites. Multimedia Syst. (2022). https://doi.org/10.1007/s00530-022-00983-x(2022)

    Article  Google Scholar 

  17. Ke, L., Tai, Y.-W., Tang, C.-K.: Deep occlusion-aware instance segmentation with overlapping bilayers. In: CVPR 2021, June 19–25, pp. 4018–4027 (2021)

  18. Scharr, H., Pridmore, T., Tsaftaris, S.A.: Computer vision problems in plant phenotyping, CVPPP 2017: introduction to the CVPPP 2017 workshop papers. In: 2017 ICCV workshop, pp. 2020–2021 (2017)

  19. Uchiyama, H., et al.: An easy-to-setup 3D phenotyping platform for KOMATSUNA dataset. In: 2017 ICCV workshop, pp. 2038–2045 (2017)

  20. Prasetyo, E., Adityo, R.D., Suciati, N., Fatichah, C.: Mango leaf image segmentation on HSV and YCbCr color spaces using Otsu thresholding. In: ICST 2017, pp. 99–103 (2017)

  21. Pape, J.-M., Klukas, C.: 3-D histogram-based segmentation and leaf detection for rosette plants. In: ECCV 2014 Workshop, pp. 61–74 (2014)

  22. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM. 60, 84–90 (2017)

    Article  Google Scholar 

  23. Huang, Z., Huang, L., Gong, Y., Huang, C., Wang, X.: Mask scoring R-CNN. In: CVPR 2019, Long Beach, CA, USA, June 16–20, 2019, pp. 6402–6411 (2019)

  24. De Brabandere, B., Neven, D., Van Gool, L.: Semantic instance segmentation with a discriminative loss function. arXiv preprint. arXiv:1708.02551 (2017)

  25. Wang, X., Zhang, R., Kong, T., Li, L., Shen, C.: SOLOv2: dynamic and fast instance segmentation. In: Advances in neural information processing systems (NIPS), pp. 17721–17732 (2020)

  26. Lin, T.-Y., et al. Feature pyramid networks for object detection. In: CVPR 2017, Honolulu, HI, USA, July 21–26, 2017, pp. 936–944 (2017)

  27. Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path aggregation network for instance segmentation. In: CVPR 2018, Salt Lake City, UT, USA, June 18–22, 2018, pp. 8759–8768(2018)

  28. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR 2016, Las Vegas, NV, USA, June 26–July 1, pp. 770–778 (2016)

  29. Li, X., Liu, Z., Luo, P., Loy, C. C., Tang, X.: Not all pixels are equal: difficulty-aware semantic segmentation via deep layer cascade. In: CVPR 2017, Honolulu, HI, USA, July 21–26, 2017, pp. 6459–6468 (2017)

  30. Tang, C., et al.: Look closer to segment better: boundary patch refinement for instance segmentation. In: CVPR 2021, pp. 13921–13930 (2021)

  31. Zhang, G. et al.: RefineMask: towards high-quality instance segmentation with fine-grained features. In: CVPR 2021, pp. 6857–6865 (2021)

  32. He, K., Zhang, X., Ren, S. & Sun, J.: delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: ICCV 2015, Santiago, Chile, December 13–16, 2015, pp. 1026–1034 (2015)

  33. Priya G, Piotr D, et al.: Accurate, Large Minibatch SGD: training ImageNet in 1 hour. arXiv preprint. arXiv: 1706.02677 (2017)

  34. Ketkar, N.: Stochastic gradient descent. A stochastic approximation method. IEEE Trans. Syst. Man Cybern. 1, 338–344 (1971)

    Google Scholar 

  35. Kirillov, A., Wu, Y., He, K., Girshick, R.: PointRend: image segmentation as rendering. In: CVPR 2020, pp. 9796–9805 (2020)

  36. Chen, H., Sun, K., Tian, Z., Shen, C., Huang, Y., Yan, Y.: BlendMask: top-down meets bottom-up for instance segmentation. In: CVPR 2020, pp. 8570–8578 (2020)

  37. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: CVPR 2018, Salt Lake City, UT, USA, June 18–22, 2018, pp. 8759–8768 (2018), pp. 7794–7803(2018)

  38. Cao, Y., Xu, J., Lin, S., Wei, F., Hu, H.: GCNet: non-local networks meet squeeze-excitation networks and beyond. In: ICCV workshop, pp. 1971–1980 (2019)

Download references

Funding

Project supported by National Key R&D Program of China (2019YFE0125500-04), National Natural Science Foundation of China (61806097, 32101617).

Author information

Authors and Affiliations

  1. College of Artificial Intelligence, Nanjing Agricultural University, Nanjing, 210095, China

    Xingjian Gu, Yongjie Zhu & Shougang Ren

  2. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China

    Xiangbo Shu

  3. National Engineering and Technology Center for Information Agriculture, Nanjing, 210095, China

    Shougang Ren

Authors
  1. Xingjian Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongjie Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shougang Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiangbo Shu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Xingjian Gu and Yongjie Zhu carried out the experiments using the BCmask, BCNet, BlendMask, collected Chrysanthemum seeding leaves datasets and do ablation experiments. Shougang Ren is responsible for the planning of the whole project and agrees to serve as the author responsible for contact and ensures communication. Xiangbo Shu gave some constructive idea, some experimental analysis and provided computing resource. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Shougang Ren.

Ethics declarations

Conflicts of interest

The authors declare that there is no conflict of interest with any individual/organization for the present work.

Research involving human participants and/or animals

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, X., Zhu, Y., Ren, S. et al. BCMask: a finer leaf instance segmentation with bilayer convolution mask. Multimedia Systems 29, 1145–1159 (2023). https://doi.org/10.1007/s00530-022-01044-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-01044-z

Keywords