Automatic Counting of in situ Rice Seedlings from UAV Images Based on a Deep Fully Convolutional Neural Network
"> Figure 1
<p>An example of the rice seedling annotations in the rice seedling counting (RSC) dataset: (<b>A</b>) raw unmanned aerial vehicle (UAV) image, (<b>B</b>) manually-annotated cropped image, and (<b>C</b>) exact positions of the rice seedlings.</p> "> Figure 2
<p>The proposed Basic Network method for counting rice seedlings.</p> "> Figure 3
<p>The proposed Combined Network method for counting rice seedlings.</p> "> Figure 4
<p>Example of a ground truth density map: manually-annotated cropped image (<b>left</b>) and corresponding ground truth rice seedling density map (<b>right</b>). Density map regression treats the two-dimensional density map as the regression target. Our proposed Basic Network regresses the global count computed from the global density map.</p> "> Figure 5
<p>Example of the area of rice seedling annotations: (<b>A</b>) cropped UAV image, (<b>B</b>) manually-annotated rice seedling area image, and (<b>C</b>) its corresponding ground truth binary map. Binary map regression treats the two-dimensional binary map as the regression target in our proposed Segmentation Network.</p> "> Figure 6
<p>Predicted count obtained from the Basic Network model versus the ground truth count for each of the 40 images in the RSC dataset.</p> "> Figure 7
<p>Manual counting versus automatic counting using our Basic Network model. The 1:1 line is dashed red. <span class="html-italic">R</span><sup>2</sup> and RMSE represent the coefficient of determination and the root mean square error, respectively.</p> "> Figure 8
<p>Performance of the Combined Network model under different intensity value threshold ranges. The X axis is the lower limit of the intensity value threshold range, the Y axis is the upper limit of the intensity value threshold range, and the Z axis is the accuracy of our proposed Combined Network for counting rice seedlings.</p> "> Figure 9
<p>Segmentation map (<b>B</b>) predicted by the Segmentation Network on a cropped UAV image (<b>A</b>) and the corresponding binary map (<b>C</b>) calculated by Equation (1). The threshold range is fixed at [90, 265].</p> "> Figure 10
<p>Predicted count obtained from the Combined Network and Basic Network models versus the ground truth count for each of the 40 images in the RSC dataset.</p> "> Figure 11
<p>Manual counting versus automatic counting with our Combined Network model. The 1:1 line is dashed red. <span class="html-italic">R</span><sup>2</sup> and RMSE represent the coefficient of determination and the root mean square error, respectively.</p> "> Figure 12
<p>Distribution of count errors for the Basic Network and Combined Network.</p> "> Figure 13
<p>Two examples of erroneous predictions obtained from the Segmentation Network: (<b>A</b>) cropped UAV images, where the upper image contains roads, shadows, and rice seedling areas, while the lower image contains weeds and rice seedling areas; (<b>B</b>) the corresponding ground truth binary maps; (<b>C</b>) segmentation maps predicted by the Segmentation Network; (<b>D</b>) the corresponding binary maps generated by Equation (1) based on the empirically determined intensity value threshold range.</p> "> Figure 14
<p>Examples of predicted rice seedling distribution maps extracted from the final density map: (<b>A</b>) cropped UAV images where rice seedlings are unevenly (upper image) and evenly (lower image) distributed; (<b>B</b>) ground truth rice seedling distribution map where the points represent rice seedlings; (<b>C</b>) predicted rice seedling distribution map extracted from the final density map using the k-NN algorithm.</p> ">
Abstract
:1. Introduction
2. Materials
2.1. Experimental Site and Imaging Devices
2.2. Rice Seedling Counting Dataset
3. Methods
3.1. Combined Network for Counting Rice Seedlings
3.1.1. Basic Network Architecture
3.1.2. Combined Network Architecture
3.2. Ground Truth
3.3. Learning and Implementation
4. Experimental Evaluation
4.1. Evaluation Metrics
4.2. Evaluating the Efficacy of the Proposed Basic Network
4.3. Evaluating the Efficacy of the Proposed Combined Network
5. Discussion
5.1. Performance of the Segmentation Network
5.2. Analysis of Predicted Distribution Maps
5.3. Comparison with Other Techniques
6. Conclusions and Future Work
Author Contributions
Funding
Conflicts of Interest
References
- Zhang, Q. Inaugural article: Strategies for developing green super rice. Proc. Natl. Acad. Sci. USA 2007, 104, 16402–16409. [Google Scholar] [CrossRef] [PubMed]
- Lu, H.; Cao, Z.; Xiao, Y.; Fang, Z.; Zhu, Y.; Xian, K. Fine-grained maize tassel trait characterization with multi-view representations. Comput. Electron. Agric. 2015, 118, 143–158. [Google Scholar] [CrossRef]
- Chen, J.; Gao, H.; Zheng, X.M.; Jin, M.; Weng, J.F.; Ma, J.; Ren, Y.; Zhou, K.; Wang, Q.; Wang, J.; et al. An evolutionarily conserved gene, FUWA, plays a role in determining panicle architecture, grain shape and grain weight in rice. Plant J. 2015, 83, 427–438. [Google Scholar] [CrossRef] [PubMed]
- Li, W.; Niu, Z.; Chen, H.; Li, D.; Wu, M.; Zhao, W. Remote estimation of canopy height and aboveground biomass of maize using high-resolution stereo images from a low-cost unmanned aerial vehicle system. Ecol. Indic. 2016, 67, 637–648. [Google Scholar] [CrossRef]
- Verger, A.; Vigneau, N.; Chéron, C.; Gilliot, J.M.; Comar, A.; Baret, F. Green area index from an unmanned aerial system over wheat and rapeseed crops. Remote Sens. Environ. 2014, 152, 654–664. [Google Scholar] [CrossRef]
- Gao, J.; Liao, W.; Nuyttens, D.; Lootens, P.; Vangeyte, J.; Pižurica, A.; He, Y.; Pieters, J.G. Fusion of pixel and object-based features for weed mapping using unmanned aerial vehicle imagery. Int. J. Appl. Earth Obs. Geoinf. 2018, 67, 43–53. [Google Scholar] [CrossRef]
- Wan, L.; Li, Y.; Cen, H.; Zhu, J.; Yin, W.; Wu, W.; Zhu, H.; Sun, D.; Zhou, W.; He, Y. Combining UAV-Based Vegetation Indices and Image Classification to Estimate Flower Number in Oilseed Rape. Remote Sens. 2018, 10, 1484. [Google Scholar] [CrossRef]
- Berni, J.; Zarco-Tejada, P.J.; Suarez, L.; Fereres, E. Thermal and Narrowband Multispectral Remote Sensing for Vegetation Monitoring from an Unmanned Aerial Vehicle. Inst. Electr. Electron. Eng. 2009, 47, 722–738. [Google Scholar] [CrossRef]
- Uto, K.; Seki, H.; Saito, G.; Kosugi, Y. Characterization of Rice Paddies by a UAV-Mounted Miniature Hyperspectral Sensor System. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2013, 6, 851–860. [Google Scholar] [CrossRef]
- Püschel, H.; Sauerbier, M.; Eisenbeiss, H. A 3D Model of Castle Landenberg (CH) from Combined Photogrammetric Processing of Terrestrial and UAV-based Images. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2008, 37, 93–98. [Google Scholar]
- Moranduzzo, T.; Mekhalfi, M.L.; Melgani, F. LBP-based multiclass classification method for UAV imagery. In Proceedings of the 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Milan, Italy, 26–31 July 2015; pp. 2362–2365. [Google Scholar]
- Lin, A.Y.-M.; Novo, A.; Har-Noy, S.; Ricklin, N.D.; Stamatiou, K. Combining GeoEye-1 Satellite Remote Sensing, UAV Aerial Imaging, and Geophysical Surveys in Anomaly Detection Applied to Archaeology. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2011, 4, 870–876. [Google Scholar] [CrossRef]
- Moranduzzo, T.; Melgani, F. A SIFT-SVM method for detecting cars in UAV images. In Proceedings of the 2012 IEEE International Geoscience and Remote Sensing Symposium, Munich, Germany, 22–27 July 2012; pp. 6868–6871. [Google Scholar]
- Moranduzzo, T.; Melgani, F. Automatic Car Counting Method for Unmanned Aerial Vehicle Images. IEEE Trans. Geosci. Remote Sens. 2014, 52, 1635–1647. [Google Scholar] [CrossRef]
- Moranduzzo, T.; Melgani, F. Detecting Cars in UAV Images with a Catalog-Based Approach. IEEE Trans. Geosci. Remote Sens. 2014, 52, 6356–6367. [Google Scholar] [CrossRef]
- Moranduzzo, T.; Melgani, F.; Bazi, Y.; Alajlan, N. A fast object detector based on high-order gradients and Gaussian process regression for UAV images. Remote Sens. 2015, 36, 37–41. [Google Scholar]
- Fernandez-Gallego, J.A.; Kefauver, S.C.; Gutiérrez, N.A.; Nieto-Taladriz, M.T.; Araus, J.L. Wheat ear counting in-field conditions: High throughput and low-cost approach using RGB images. Plant Methods 2018, 14, 22–34. [Google Scholar] [CrossRef] [PubMed]
- Zhou, C.; Liang, D.; Yang, X.; Yang, H.; Yue, J.; Yang, G. Wheat Ears Counting in Field Conditions Based on Multi-Feature Optimization and TWSVM. Front. Plant Sci. 2018, 9, 1024–1040. [Google Scholar] [CrossRef] [PubMed]
- Guo, W.; Zheng, B.; Potgieter, A.B.; Diot, J.; Watanabe, K.; Noshita, K.; Jordan, D.R.; Wang, X.; Watson, J.; Ninomiya, S.; et al. Aerial Imagery Analysis–Quantifying Appearance and Number of Sorghum Heads for Applications in Breeding and Agronomy. Front. Plant Sci. 2018, 9, 1544–1553. [Google Scholar] [CrossRef] [PubMed]
- Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 24–27 June 2014; pp. 580–587. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 37, 1904–1916. [Google Scholar] [CrossRef] [PubMed]
- Girshick, R. Fast R-CNN. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 13–16 December 2015; pp. 1440–1448. [Google Scholar]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 2015, 39, 91–99. [Google Scholar] [CrossRef] [PubMed]
- Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 779–788. [Google Scholar]
- Redmon, J.; Farhadi, A. YOLO9000: Better, Faster, Stronger. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 22–25 July 2017; pp. 6517–6525. [Google Scholar]
- Boominathan, L.; Kruthiventi, S.S.; Babu, R.V. Crowdnet: A deep convolutional network for dense crowd counting. In Proceedings of the 2016 ACM on Multimedia Conference (ACMMM), Amsterdam, The Netherlands, 15–19 October 2016; pp. 640–644. [Google Scholar]
- Zhang, C.; Li, H.; Wang, X.; Yang, X. Cross-scene crowd counting via deep convolutional neural networks. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 8–10 June 2015; pp. 2980–2988. [Google Scholar]
- Sam, D.B.; Surya, S.; Babu, R.V. Switching convolutional neural network for crowd counting. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 22–25 July 2017; pp. 7263–7271. [Google Scholar]
- Lempitsky, V.; Zisserman, A. Learning to count objects in images. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada, 6–11 December 2010; pp. 1324–1332. [Google Scholar]
- Xie, W.; Noble, J.A.; Zisserman, A. Microscopy cell counting and detection with fully convolutional regression networks. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. 2018, 6, 283–292. [Google Scholar] [CrossRef]
- Onoro-Rubio, D.; López-Sastre, R.J. Towards perspective-free object counting with deep learning. In Proceedings of the 13th European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 8–16 October 2016; pp. 615–629. [Google Scholar]
- Arteta, C.; Lempitsky, V.; Noble, J.A.; Zisserman, A. Interactive object counting. In Proceedings of the 13th European Conference on Computer Vision (ECCV), Zurich, Switzerland, 6–12 September 2014; pp. 504–518. [Google Scholar]
- Arteta, C.; Lempitsky, V.; Zisserman, A. Counting in the wild. In Proceedings of the 13th European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 8–16 October 2016; pp. 615–629. [Google Scholar]
- Rahnemoonfar, M.; Sheppard, C. Deep count: Fruit counting based on deep simulated learning. Sensors 2017, 17, 905. [Google Scholar] [CrossRef] [PubMed]
- Lu, H.; Cao, Z.; Xiao, Y.; Zhuang, B.; Shen, C. TasselNet: Counting maize tassels in the wild via local counts regression network. Plant Methods 2017, 13, 79–96. [Google Scholar] [CrossRef] [PubMed]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 2015 International Conference on Learning Representations (ICLR), San Diego, CA, USA, 7–9 May 2015; pp. 1556–1570. [Google Scholar]
- Pan, J.; Sayrol, E.; Giro-i-Nieto, X.; McGuinness, K.; O’Connor, N.E. Shallow and deep convolutional networks for saliency prediction. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 598–606. [Google Scholar]
- Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 8–10 June 2015; pp. 3431–3440. [Google Scholar]
- Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Savannah, GA, USA, 2–4 November 2016; pp. 265–283. [Google Scholar]
- Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 8–10 June 2015; pp. 1–9. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
- ENVI Crop Science 1.1. Crop Counting and Metrics Tutorial. 2018. Available online: http://www.harrisgeospatial.com/portals/0/pdfs/envi/TutorialCropCountingMetrics.pdf (accessed on 9 March 2019).
ACC | MAE | |
---|---|---|
Fold-1 | 78.74% | 1838.965 |
Fold-2 | 83.51% | 1500.078 |
Fold-3 | 74.05% | 2161.408 |
Fold-4 | 91.47% | 923.5362 |
Average | 81.94% | 1605.997 |
ACC | MAE | |||
---|---|---|---|---|
Basic Network | Combined Network | Basic Network | Combined Network | |
Fold-1 | 78.74% | 91.91% | 1838.965 | 772.3573 |
Fold-2 | 83.51% | 94.07% | 1500.078 | 623.1984 |
Fold-3 | 74.05% | 92.76% | 2161.408 | 754.4838 |
Fold-4 | 91.47% | 94.68% | 923.5362 | 638.9999 |
Average | 81.94% | 93.35% | 1605.997 | 697.2598 |
Methods | MAE | Acc | Times (s) | |
---|---|---|---|---|
Count Crops tool (without pre-processing) | Channel R | 1371.675 | 87.29% | 40.93 |
Channel G | 2362.125 | 77.54% | 41.26 | |
Channel B | 1595.35 | 84.63% | 41.04 | |
Count Crops tool (pre-processing) | Channel R | 1001.325 | 90.71% | 27.57 |
Channel G | 1901.925 | 81.76% | 27.49 | |
Channel B | 1204.375 | 88.16% | 27.73 | |
Our proposed Combined Network method | 697.2598 | 93.35% | 0.93 |
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wu, J.; Yang, G.; Yang, X.; Xu, B.; Han, L.; Zhu, Y. Automatic Counting of in situ Rice Seedlings from UAV Images Based on a Deep Fully Convolutional Neural Network. Remote Sens. 2019, 11, 691. https://doi.org/10.3390/rs11060691
Wu J, Yang G, Yang X, Xu B, Han L, Zhu Y. Automatic Counting of in situ Rice Seedlings from UAV Images Based on a Deep Fully Convolutional Neural Network. Remote Sensing. 2019; 11(6):691. https://doi.org/10.3390/rs11060691
Chicago/Turabian StyleWu, Jintao, Guijun Yang, Xiaodong Yang, Bo Xu, Liang Han, and Yaohui Zhu. 2019. "Automatic Counting of in situ Rice Seedlings from UAV Images Based on a Deep Fully Convolutional Neural Network" Remote Sensing 11, no. 6: 691. https://doi.org/10.3390/rs11060691