Manhole Cover Classification Based on Super-Resolution Reconstruction of Unmanned Aerial Vehicle Aerial Imagery
<p>Flow chart for locating and classifying manhole covers.</p> "> Figure 2
<p>YOLOv8 structure diagram.</p> "> Figure 3
<p>Manhole cover positioning data set display.</p> "> Figure 4
<p>Picture overlap rate display. (<b>a</b>) cutout schematic, (<b>b</b>) avoiding manhole cover cutout schematic.</p> "> Figure 5
<p>Fuzzy display of manhole cover.</p> "> Figure 6
<p>SRGAN network structure.</p> "> Figure 7
<p>Enhanced display of text super-resolution reconstruction dataset.</p> "> Figure 8
<p>SRGAN network training process.</p> "> Figure 9
<p>VGG16_BN network architecture diagram.</p> "> Figure 10
<p>Category display of manhole cover classification.</p> "> Figure 11
<p>K-value cross-validation dataset division.</p> "> Figure 12
<p>Aerial image of manhole cover positioning.</p> "> Figure 13
<p>Super-resolution reconstruction of aerial image.</p> "> Figure 14
<p>Evaluation index diagram.</p> "> Figure 15
<p>Display of the results of manhole cover localization. (<b>a</b>) sunny day shot of manhole covers (<b>b</b>,<b>e</b>) cloudy day shot with different types of manhole covers (<b>c</b>,<b>d</b>) mutilated manhole covers (<b>f</b>) stained cover manhole covers.</p> "> Figure 16
<p>Text positioning results display. (<b>a</b>,<b>b</b>) square manhole covers (<b>c</b>) stained covered manhole covers (<b>d</b>,<b>e</b>) sunny shooting manhole covers (<b>f</b>) mutilated manhole covers.</p> "> Figure 17
<p>Super-resolution reconstruction results.</p> "> Figure 18
<p>Hyperparameter training loss results.</p> "> Figure 19
<p>The result of super-resolution image text recognition.</p> "> Figure 20
<p>Character recognition network training result diagram. (<b>a</b>) low-brightness manhole cover (<b>b</b>–<b>d</b>) high-brightness manhole cover.</p> ">
Abstract
:1. Introduction
2. Methods
2.1. Manhole Cover Positioning
2.1.1. YOLOv8 Network Architecture
2.1.2. YOLOv8 Network Data Set Production
2.2. Text Localization
Text Positioning Data Set Production
2.3. Super-Resolution Network Reconstruction Manhole Cover Text
2.3.1. SRGAN Network
2.3.2. The Production and Enhancement of SRGAN Network Dataset
2.3.3. SRGAN Network Training
2.4. Image Classification Realizes Manhole Cover Classification
2.4.1. VGG16_BN Network Architecture
2.4.2. Text Recognition Data Set Production
3. Experiments
3.1. Software and Hardware
3.2. Dataset Acquisition
3.2.1. Manhole Cover Positioning Data Set Acquisition
3.2.2. Text Location Data Set Acquisition
3.2.3. Manhole Cover Image Super-Resolution Data Set Acquisition
3.2.4. VGG16_BN Network Data Set Production
3.3. Evaluation Parameters and Indicators
3.4. Setting of Training Parameters
4. Results
4.1. Analysis of Manhole Cover Positioning Results
4.2. Analysis of Text Positioning Results
4.3. Super-Resolution Reconstruction of Morehole Cover Text
4.4. Analysis of Manhole Cover Classification Results
5. Conclusions
- The experimental results showed that it was an effective method to locate the manhole cover and text before text recognition.
- The experiment proved that the accuracy of manhole cover classification could be improved by using the method of image super-resolution reconstruction to clarify the cap text and reconstruct the aerial image with missing text details due to the long aerial distance.
- The method of using VGG16_BN to classify manhole covers is an effective method. It can accurately identify the type of manhole cover, and the recognition accuracy is as high as 97.62%.
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Boller, D.; de Vitry, M.M.; Wegner, J.D.; Leitão, J.P. Automated localization of urban drainage infrastructure from public-access street-level images. Urban Water J. 2019, 16, 480–493. [Google Scholar] [CrossRef]
- Chahinian, N.; Delenne, C.; Commandré, B.; Derras, M.; Deruelle, L.; Bailly, J.-S. Automatic mapping of urban wastewater networks based on manhole cover locations. Comput. Environ. Urban Syst. 2019, 78, 101370. [Google Scholar] [CrossRef]
- Yamaguchi, T.; Mizutani, T. Detection and localization of manhole and joint covers in radar images by support vector machine and Hough transform. Autom. Constr. 2021, 126, 103651. [Google Scholar] [CrossRef]
- Luo, X.; Wu, Y.; Zhao, L. YOLOD: A Target Detection Method for UAV Aerial Imagery. Remote Sens. 2022, 14, 3240. [Google Scholar] [CrossRef]
- Eslami, E.; Yun, H.-B. Attention-Based Multi-Scale Convolutional Neural Network (A+MCNN) for Multi-Class Classification in Road Images. Sensors 2021, 21, 5137. [Google Scholar] [CrossRef]
- Hu, Q.; Li, L.; Duan, J.; Gao, M.; Liu, G.; Wang, Z.; Huang, D. Object Detection Algorithm of UAV Aerial Photography Image Based on Anchor-Free Algorithms. Electronics 2023, 12, 1339. [Google Scholar] [CrossRef]
- Ying, X.; Wang, Q.; Li, X.; Yu, M.; Jiang, H.; Gao, J.; Liu, Z.; Yu, R. Multi-Attention Object Detection Model in Remote Sensing Images Based on Multi-Scale. IEEE Access 2019, 7, 94508–94519. [Google Scholar] [CrossRef]
- Liu, W.; Cheng, D.; Yin, P.; Yang, M.; Li, E.; Xie, M.; Zhang, L. Small Manhole Cover Detection in Remote Sensing Imagery with Deep Convolutional Neural Networks. ISPRS Int. J. Geo Inf. 2019, 8, 49. [Google Scholar] [CrossRef]
- Yu, Y.; Guan, H.; Ji, Z. Automated Detection of Urban Road Manhole Covers Using Mobile Laser Scanning Data. IEEE Trans. Intell. Transp. Syst. 2015, 16, 3258–3269. [Google Scholar] [CrossRef]
- Yu, Y.; Guan, H.; Li, D.; Jin, C.; Wang, C.; Li, J. Road Manhole Cover Delineation Using Mobile Laser Scanning Point Cloud Data. IEEE Geosci. Remote Sens. Lett. 2020, 17, 152–156. [Google Scholar] [CrossRef]
- Wei, Z.; Yang, M.; Wang, L.; Ma, H.; Chen, X.; Zhong, R. Customized Mobile LiDAR System for Manhole Cover Detection and Identification. Sensors 2019, 19, 2422. [Google Scholar] [CrossRef]
- Pang, D.; Guan, Z.; Luo, T.; Su, W.; Dou, R. Real-time detection of road manhole covers with a deep learning model. Sci. Rep. 2023, 13, 16479. [Google Scholar] [CrossRef]
- Mattheuwsen, L.; Vergauwen, M. Manhole Cover Detection on Rasterized Mobile Mapping Point Cloud Data Using Transfer Learned Fully Convolutional Neural Networks. Remote Sens. 2020, 12, 3820. [Google Scholar] [CrossRef]
- Wang, Y.; Sun, G.; Guo, S. Target Detection Method for Low-Resolution Remote Sensing Image Based on ESRGAN and ReDet. Photonics 2021, 8, 431. [Google Scholar] [CrossRef]
- Gong, Y.; Liao, P.; Zhang, X.; Zhang, L.; Chen, G.; Zhu, K.; Tan, X.; Lv, Z. Enlighten-GAN for Super Resolution Reconstruction in Mid-Resolution Remote Sensing Images. Remote Sens. 2021, 13, 1104. [Google Scholar] [CrossRef]
- Clabaut, É.; Lemelin, M.; Germain, M.; Bouroubi, Y.; St-Pierre, T. Model Specialization for the Use of ESRGAN on Satellite and Airborne Imagery. Remote Sens. 2021, 13, 4044. [Google Scholar] [CrossRef]
- Fan, C.; Wu, C.; Li, G.; Ma, J. Projections onto Convex Sets Super-Resolution Reconstruction Based on Point Spread Function Estimation of Low-Resolution Remote Sensing Images. Sensors 2017, 17, 362. [Google Scholar] [CrossRef] [PubMed]
- Zhou, Q. Superresolution Reconstruction of Remote Sensing Image Based on Generative Adversarial Network. Wirel. Commun. Mob. Comput. 2022, 2022, 9114911. [Google Scholar] [CrossRef]
- Zhu, Z.; Lei, Y.; Qin, Y.; Zhu, C.; Zhu, Y. IRE: Improved Image Super-Resolution Based on Real-ESRGAN. IEEE Access 2023, 11, 45334–45348. [Google Scholar] [CrossRef]
- Ma, J.; Guo, S.; Zhang, L. Text Prior Guided Scene Text Image Super-Resolution. IEEE Trans. Image Process. 2023, 32, 1341–1353. [Google Scholar] [CrossRef]
- Yue, X.; Liu, D.; Wang, L.; Benediktsson, J.A.; Meng, L.; Deng, L. IESRGAN: Enhanced U-Net Structured Generative Adversarial Network for Remote Sensing Image Super-Resolution Reconstruction. Remote Sens. 2023, 15, 3490. [Google Scholar] [CrossRef]
- Long, Y.; Sun, W.; Pang, Y.; Wang, H.; Zhang, G. Research on text detection on building surfaces in smart cities based on deep learning. Soft Comput. 2022, 26, 10103–10114. [Google Scholar] [CrossRef]
- Saydirasulovich, S.N.; Mukhiddinov, M.; Djuraev, O.; Abdusalomov, A.; Cho, Y.-I. An Improved Wildfire Smoke Detection Based on YOLOv8 and UAV Images. Sensors 2023, 23, 8374. [Google Scholar] [CrossRef] [PubMed]
- Zhai, X.; Huang, Z.; Li, T.; Liu, H.; Wang, S. YOLO-Drone: An Optimized YOLOv8 Network for Tiny UAV Object Detection. Electronics 2023, 12, 3664. [Google Scholar] [CrossRef]
- Luan, X.; Zhang, J.; Xu, M.; Silamu, W.; Li, Y. Lightweight Scene Text Recognition Based on Transformer. Sensors 2023, 23, 4490. [Google Scholar] [CrossRef] [PubMed]
- Zuo, L.Q.; Sun, H.M.; Mao, Q.C.; Qi, R.; Jia, R.S. Natural Scene Text Recognition Based on Encoder-Decoder Framework. IEEE Access 2019, 7, 62616–62623. [Google Scholar] [CrossRef]
- Liu, B.; Chen, J. A Super Resolution Algorithm Based on Attention Mechanism and SRGAN Network. IEEE Access 2021, 9, 139138–139145. [Google Scholar] [CrossRef]
- Kacem, A.; Saïdani, A. A texture-based approach for word script and nature identification. Pattern Anal. Appl. 2017, 20, 1157–1167. [Google Scholar] [CrossRef]
- Devi, K.D.; Maheswari, P.U.; Polasi, P.K.; Preetha, R.; Vidhyalakshmi, M. Pattern Matching Model for Recognition of Stone Inscription Characters. Comput. J. 2023, 66, 554–564. [Google Scholar] [CrossRef]
- Qu, Z.; Mei, J.; Liu, L.; Zhou, D.Y. Crack Detection of Concrete Pavement with Cross-Entropy Loss Function and Improved VGG16 Network Model. IEEE Access 2020, 8, 54564–54573. [Google Scholar] [CrossRef]
Methods | Operation Execution |
---|---|
Brightness | Brightness factor = 0.8;1.2;1.4 |
Flip | Horizontally flip; Vertically flip |
Gauss blur | Blur radius =1.5;2;2.5 |
Noise | Noise factor = 30;40;50 |
Evaluating Indicator | YOLOv8 | Faster R-CNN | EfficientNet |
---|---|---|---|
Precision | 97.41% | 93.01% | 96.88% |
Recall | 97.13% | 99.43% | 97.14% |
mAP | 99.63% | 96.84% | 97.33% |
F1_Score | 97.27% | 96.11% | 97.01% |
Evaluating Indicator | YOLOv8 |
---|---|
Precision | 98.54% |
Recall | 96.31% |
mAP | 99.76% |
F1_Score | 97.41% |
Evaluating Indicator | SRGAN |
---|---|
PSNR | 29.54 |
SSIM | 0.83 |
Evaluating Indicator | Mobilenetv1 | Swin_transformer_tiny | VGG16_BN |
---|---|---|---|
Precision | 92.73% | 90.0% | 97.62% |
Recall | 92.73% | 94.32% | 98.86% |
mAP | 93.75% | 92.19% | 98.44% |
F1_Score | 92.73% | 92.11% | 98.24% |
Evaluating Indicator | Precision | Recall | mAP | F1_Score |
---|---|---|---|---|
K1 | 94.96% | 94.42% | 95.18% | 94.69% |
K2 | 94.35% | 94.54% | 93.97% | 94.44% |
K3 | 92.02% | 92.02% | 92.26% | 92.02% |
K4 | 93.15% | 93.22% | 92.52% | 93.18% |
K5 | 95.12% | 93.71% | 95.48% | 94.41% |
Mean Value | 93.92% | 94.06% | 94.21% | 94.49% |
Mean Square Deviation | 0.012 | 0.014 | 0.018 | 0.010 |
Method | Precision | Recall | Top-1 Accuracy | F1_score |
---|---|---|---|---|
Original images | 84.92% | 77.95% | 82.81% | 81.29% |
Super-resolution reconstruction | 97.62% | 98.86% | 98.44% | 98.24% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, D.; Huang, Y. Manhole Cover Classification Based on Super-Resolution Reconstruction of Unmanned Aerial Vehicle Aerial Imagery. Appl. Sci. 2024, 14, 2769. https://doi.org/10.3390/app14072769
Wang D, Huang Y. Manhole Cover Classification Based on Super-Resolution Reconstruction of Unmanned Aerial Vehicle Aerial Imagery. Applied Sciences. 2024; 14(7):2769. https://doi.org/10.3390/app14072769
Chicago/Turabian StyleWang, Dejiang, and Yuping Huang. 2024. "Manhole Cover Classification Based on Super-Resolution Reconstruction of Unmanned Aerial Vehicle Aerial Imagery" Applied Sciences 14, no. 7: 2769. https://doi.org/10.3390/app14072769