Local and Global Context-Enhanced Lightweight CenterNet for PCB Surface Defect Detection
<p>Six different kinds of PCB defects.</p> "> Figure 2
<p>The overall network architecture of the proposed local and global context-enhanced lightweight CenterNet.</p> "> Figure 3
<p>Ratio of PCB defect bounding box area to total image area.</p> "> Figure 4
<p>The structure of local coordinate attention and global self-attention.</p> "> Figure 5
<p>Sparse attention is used to skip computations in the most irrelevant region, and pooling is used to downsample the key and value to reduce FLOPs.</p> "> Figure 6
<p>Detection results of different object detection algorithms. More detection results of the other defects can be found in the <a href="#app1-sensors-24-04729" class="html-app">Supplementary Materials</a>.</p> ">
Abstract
:1. Introduction
2. Related Work
2.1. Traditional Visual Inspection Algorithms
2.2. Machine Learning-Based Algorithms
2.3. Deep Learning-Based Algorithms
3. Methodology
3.1. Review of CenterNet
3.2. Attention Architectures
3.3. The Network Architecture and Loss Function
3.4. Local and Global Context-Enhanced Lightweight Module
4. Experiment
4.1. Experimental Data
4.2. Data Enhancement and Training Parameters
4.3. Experimental Metrics and Experimental Results
4.4. Ablation Study
5. Conclusions
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Bhattacharya, A.; Cloutier, S.G. End-to-end deep learning framework for printed circuit board manufacturing defect classifi-cation. Sci. Rep. 2022, 12, 12559. [Google Scholar] [CrossRef] [PubMed]
- Pham, T.T.A.; Thoi, D.K.T.; Choi, H.; Park, S. Defect detection in printed circuit boards using semi-supervised learning. Sensors 2023, 23, 3246. [Google Scholar] [CrossRef] [PubMed]
- Le, X.; Mei, J.; Zhang, H.; Zhou, B.; Xi, J. A learning-based approach for surface defect detection using small image datasets. Neurocomputing 2020, 408, 112–120. [Google Scholar] [CrossRef]
- Xia, K.; Lv, Z.; Liu, K.; Lu, Z.; Zhou, C.; Zhu, H.; Chen, X. Global contextual attention augmented YOLO with ConvMixer prediction heads for PCB surface defect detection. Sci. Rep. 2023, 13, 9805. [Google Scholar] [CrossRef] [PubMed]
- Kim, J.; Ko, J.; Choi, H.; Kim, H. Printed circuit board defect detection using deep learning via a skip-connected convolutional autoencoder. Sensors 2021, 21, 4968. [Google Scholar] [CrossRef] [PubMed]
- Wan, Y.; Gao, L.; Li, X.; Gao, Y. Semi-supervised defect detection method with data-expanding strategy for pcb quality inspection. Sensors 2022, 22, 7971. [Google Scholar] [CrossRef] [PubMed]
- Zhang, C.; Shi, W.; Li, X.; Zhang, H.; Liu, H. Improved bare PCB defect detection approach based on deep feature learning. J. Eng. 2018, 2018, 1415–1420. [Google Scholar] [CrossRef]
- Gaidhane, V.H.; Hote, Y.V.; Singh, V. An efficient similarity measure approach for PCB surface defect detection. Pattern Anal. Appl. 2017, 21, 277–289. [Google Scholar] [CrossRef]
- Fekri-Ershad, S.; Tajeripour, F. A robust approach for surface defect detection based on one dimensional local binary patterns. Indian J. Sci. Technol. 2012, 5, 3197–3203. [Google Scholar] [CrossRef]
- Ran, G.; Lei, X.; Li, D.; Guo, Z. Research on PCB defect detection using deep convolutional nerual network. In Proceedings of the 2020 5th International Conference on Mechanical, Control and Computer Engineering (ICMCCE), Harbin, China, 25–27 December 2020; pp. 1310–1314. [Google Scholar]
- Noble, W.S. What is a support vector machine. Nat. Biotechnol. 2006, 24, 1565–1567. [Google Scholar] [CrossRef]
- Kotsiantis, S.B. Decision trees: A recent overview. Artif. Intell. Rev. 2011, 39, 261–283. [Google Scholar] [CrossRef]
- De Oliveira, M.; Monteiro, A.; Vieira Filho, J. A new structural health monitoring strategy based on PZT sensors and convolu-tional neural network. Sensors 2018, 18, 2955. [Google Scholar] [CrossRef] [PubMed]
- Luo, J.; Yang, Z.; Li, S.; Wu, Y. FPCB surface defect detection: A decoupled two-stage object detection framework. IEEE Trans. Instrum. Meas. 2021, 70, 5012311. [Google Scholar] [CrossRef]
- Qiao, S.; Chen, L.-C.; Yuille, A. DetectoRS: Detecting objects with recursive feature pyramid and switchable Atrous convolution. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 10213–10224. [Google Scholar]
- Fang, H.; Xia, M.; Liu, H.; Chang, Y.; Wang, L.; Liu, X. Automatic zipper tape defect detection using two-stage multi-scale convolutional networks. Neurocomputing 2021, 422, 34–50. [Google Scholar] [CrossRef]
- Wang, H.; Zhang, R.; Feng, M.; Liu, Y.; Yang, G. Global Context-Based Self-Similarity Feature Augmentation and Bidirectional Feature Fusion for Surface Defect Detection. IEEE Trans. Instrum. Meas. 2023, 72, 5024712. [Google Scholar] [CrossRef]
- Tu, Z.; Wu, S.; Kang, G.; Lin, J. Real-time defect detection of track components: Considering class imbalance and subtle difference between classes. IEEE Trans. Instrum. Meas. 2021, 70, 5017712. [Google Scholar] [CrossRef]
- Duan, K.; Bai, S.; Xie, L.; Qi, H.; Huang, Q.; Tian, Q. CenterNet: Keypoint triplets for object detection. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27–28 October 2019; pp. 6568–6577. [Google Scholar]
- Zhang, H.; Jiang, L.; Li, C. CS-ResNet: Cost-sensitive residual convolutional neural network for PCB cosmetic defect detection. Expert Syst. Appl. 2021, 185, 115673. [Google Scholar] [CrossRef]
- Kang, L.; Ge, Y.; Huang, H.; Zhao, M. Research on PCB defect detection based on SSD. In Proceedings of the 2022 IEEE 4th International Conference on Civil Aviation Safety and Information Technology (ICCASIT), Dali, China, 12–14 October 2022; pp. 1315–1319. [Google Scholar]
- Chen, G.; Xu, F.; Liu, G.; Chen, C.; Liu, M.; Zhang, J.; Niu, X. ESDDNet: Efficient small defect detection network of workpiece surface. Meas. Sci. Technol. 2022, 33, 105007. [Google Scholar] [CrossRef]
- Yu, X.; Li, H.-X.; Yang, H. Collaborative Learning Classification Model for PCBs Defect Detection against Image and Label Uncertainty. IEEE Trans. Instrum. Meas. 2023, 72, 3505008. [Google Scholar] [CrossRef]
- Haase, D.; Amthor, M. Rethinking depthwise separable convolutions: How intra-kernel correlations lead to improved mobilenets. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 14588–14597. [Google Scholar]
- Xu, C.; Li, W.; Cui, X.; Wang, Z.; Zheng, F.; Zhang, X.; Chen, B. Scarcity-GAN: Scarce data augmentation for defect detection via generative adversarial nets. Neuro-Computing 2024, 566, 127061. [Google Scholar] [CrossRef]
- Tsai, D.-M.; Huang, C.-K. Defect detection in electronic surfaces using template-based fourier image reconstruction. IEEE Trans. Compon. Packag. Manuf. Technol. 2018, 9, 163–172. [Google Scholar] [CrossRef]
- Hassanin, A.-A.I.M.; El-Samie, F.E.A.; El Banby, G.M. A real-time approach for automatic defect detection from PCBs based on SURF features and morphological operations. Multimed. Tools Appl. 2019, 78, 34437–34457. [Google Scholar] [CrossRef]
- Kaur, B.; Kaur, G.; Kaur, A. Detection and classification of Printed circuit board defects using image subtraction method. In Proceedings of the 2014 Recent Advances in Engineering and Computational Sciences (RAECS), Chandigarh, India, 6–8 March 2014; pp. 1–5. [Google Scholar]
- Zhou, W.; Fei, M.; Zhou, H.; Li, K. A sparse representation based fast detection method for surface defect detection of bottle caps. Neurocomputing 2014, 123, 406–414. [Google Scholar] [CrossRef]
- Tsai, M.; Lin, B.-T. Defect detection of gold-plated surfaces on PCBs using entropy measures. Int. J. Adv. Manuf. Technol. 2002, 20, 420–428. [Google Scholar] [CrossRef]
- Xie, L.; Huang, R.; Cao, Z. Detection and classification of defect patterns in optical inspection using support vector machines. In Intelligent Computing Theories; Springer: Berlin/Heidelberg, Germany, 2013; pp. 376–384. [Google Scholar]
- Borwankar, R.; Ludwig, R. An optical surface inspection and automatic classification technique using the rotated wavelet transform. IEEE Trans. Instrum. Meas. 2018, 67, 690–697. [Google Scholar] [CrossRef]
- Lu, Z.; He, Q.; Xiang, X.; Liu, H. Defect detection of PCB based on Bayes feature fusion. J. Eng. 2018, 2018, 1741–1745. [Google Scholar] [CrossRef]
- Vafeiadis, T.; Dimitriou, N.; Ioannidis, D.; Wotherspoon, T.; Tinker, G.; Tzovaras, D. A framework for inspection of dies attachment on PCB utilizing machine learning techniques. J. Manag. Anal. 2018, 5, 81–94. [Google Scholar] [CrossRef]
- Malaca, P.; Rocha, L.F.; Gomes, D.; Silva, J.; Veiga, G. Online inspection system based on machine learning techniques: Real case study of fabric textures classification for the automotive industry. J. Intell. Manuf. 2016, 30, 351–361. [Google Scholar] [CrossRef]
- Zhang, D.; Hao, X.; Liang, L.; Liu, W.; Qin, C. A novel deep convolutional neural network algorithm for surface defect detection. J. Comput. Des. Eng. 2022, 9, 1616–1632. [Google Scholar] [CrossRef]
- Xian, Y.; Yu, Y.; Lian, Y.; Fan, J.; Wang, Z. An EA-based pruning on improved YOLOv3 for rapid copper elbow surface defect detection. Eng. Appl. Artif. Intell. 2023, 123, 106412. [Google Scholar] [CrossRef]
- Hu, B.; Wang, J. Detection of PCB surface defects with improved faster-RCNN and feature pyramid network. IEEE Access 2020, 8, 108335–108345. [Google Scholar] [CrossRef]
- Zeng, N.; Wu, P.; Wang, Z.; Li, H.; Liu, W.; Liu, X. A Small-Sized Object Detection Oriented Multi-Scale Feature Fusion Approach with Application to Defect Detection. IEEE Trans. Instrum. Meas. 2022, 71, 3507014. [Google Scholar] [CrossRef]
- Xu, X.; Zhao, M.; Shi, P.; Ren, R.; He, X.; Wei, X.; Yang, H. Crack Detection and Comparison Study Based on Faster R-CNN and Mask R-CNN. Sensors 2022, 22, 1215. [Google Scholar] [CrossRef] [PubMed]
- Tang, J.; Zhao, Y.; Bai, D.; Liu, Q. Rev-RetinaNet: PCB defect detection algorithm based on improved RetinaNet. In Proceedings of the 2023 IEEE 2nd International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA), Changchun, China, 24–26 February 2023; pp. 653–658. [Google Scholar]
- Li, C.; Yu, C.; Lin, H. DesNet: PCB defect detection network based on deformable convolution. In Proceedings of the 2023 IEEE 3rd International Conference on Electronic Technology, Communication and Information (ICETCI), Changchun, China, 26–28 May 2023; pp. 677–681. [Google Scholar]
- Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 779–788. [Google Scholar] [CrossRef]
- Zhang, D.; Hao, X.; Wang, D.; Qin, C.; Zhao, B.; Liang, L.; Liu, W. An efficient lightweight convolutional neural network for industrial surface defect detection. Artif. Intell. Rev. 2023, 56, 10651–10677. [Google Scholar] [CrossRef]
- Hu, H.; Zhu, Z. Sim-YOLOv5s: A method for detecting defects on the end face of lithium battery steel shells. Adv. Eng. Inform. 2023, 55, 101824. [Google Scholar] [CrossRef]
- Yan, R.; Zhang, R.; Bai, J.; Hao, H.; Guo, W.; Gu, X.; Liu, Q. STMS-YOLOv5: A Lightweight Algorithm for Gear Surface Defect Detection. Sensors 2023, 23, 5992. [Google Scholar] [CrossRef]
- Zhu, L.; Wang, X.; Ke, Z.; Zhang, W.; Lau, R. BiFormer: Vision Transformer with Bi-Level Routing Attention. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023; pp. 10323–10333. [Google Scholar]
- Hou, Q.; Zhou, D.; Feng, J. Coordinate Attention for Efficient Mobile Network Design. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 13708–13717. [Google Scholar]
- Huang, W.; Wei, P.; Zhang, M.; Liu, H. HRIPCB: A challenging dataset for PCB defects detection and classification. J. Eng. 2020, 2020, 303–309. [Google Scholar] [CrossRef]
- Wang, K.; Liew, J.H.; Zou, Y.; Zhou, D.; Feng, J. PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 9197–9206. [Google Scholar]
- Lin, T.Y.; Goyal, P.; Girshick, R.; He, K.; Dollar, P. Focal Loss for Dense Object Detection. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 42, 318–327. [Google Scholar] [CrossRef]
- Yang, R.; Ma, H.; Wu, J.; Tang, Y.; Xiao, X.; Zheng, M.; Li, X. ScalableViT: Rethinking the context-oriented generalization of vision transformer. In Proceedings of the 17th European Conference on Computer Vision, Tel Aviv, Isarel, 23–27 October 2022; pp. 480–496. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017; pp. 5998–6008. [Google Scholar]
- He, K.; Chen, X.; Xie, S.; Li, Y.; Dollar, P.; Girshick, R. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 15979–15988. [Google Scholar]
Method | AP50(%) | mAP@50 (%) | mAP@75 (%) | mAP@50:95 (%) | mAPS@50:95 (%) | |||||
---|---|---|---|---|---|---|---|---|---|---|
Sh | Mh | Sp | Mb | Sc | Oc | |||||
YOLOv5s | 0.988 | 0.9931 | 0.9688 | 0.9722 | 0.9814 | 0.9828 | 0.9810 | 0.442 | 0.514 | 0.457 |
YOLOv8n | 0.9672 | 0.9815 | 0.9898 | 0.9444 | 1.0 | 0.969 | 0.9735 | 0.488 | 0.516 | 0.454 |
Retinanet-ResNet18 | 0.9424 | 0.9815 | 0.9936 | 0.9729 | 0.9994 | 0.9886 | 0.9797 | 0.387 | 0.493 | 0.443 |
CenterNet-ResNet18 | 0.9452 | 0.9726 | 0.943 | 0.9848 | 1.0 | 0.9634 | 0.9682 | 0.4 | 0.482 | 0.5 |
YOLOv5m | 0.9893 | 1.0 | 0.9739 | 0.967 | 1.0 | 0.9706 | 0.9835 | 0.477 | 0.53 | 0.455 |
YOLOv8s | 0.9774 | 0.968 | 0.9795 | 0.9715 | 1.0 | 0.9883 | 0.9808 | 0.421 | 0.501 | 0.544 |
Retinanet-ResNet34 | 0.9804 | 0.9918 | 0.9951 | 0.9735 | 0.981 | 0.966 | 0.9813 | 0.42 | 0.494 | 0.482 |
CenterNet-ResNet34 | 0.9756 | 1 | 0.9198 | 0.9687 | 0.9752 | 0.972 | 0.9686 | 0.398 | 0.478 | 0.459 |
YOLOv5l | 0.9907 | 0.9964 | 0.9858 | 0.9693 | 1.0 | 0.9809 | 0.9872 | 0.499 | 0.532 | 0.461 |
YOLOv8m | 0.9663 | 0.9861 | 0.9708 | 0.9672 | 1.0 | 0.9915 | 0.9802 | 0.423 | 0.499 | 0.596 |
Retinanet-ResNet50 | 0.9736 | 0.9948 | 0.9937 | 0.9507 | 0.994 | 0.9786 | 0.9809 | 0.42 | 0.5 | 0.526 |
CenterNet-ResNet50 | 0.9658 | 1.0 | 0.9729 | 0.9476 | 1 | 0.9734 | 0.9766 | 0.424 | 0.494 | 0.469 |
GCC-YOLO | 0.9491 | 0.9834 | 0.9814 | 0.9842 | 0.9995 | 0.9845 | 0.9804 | 0.478 | 0.506 | 0.483 |
Ours | 0.9930 | 1.0 | 0.9972 | 0.9675 | 0.9997 | 0.9796 | 0.9895 | 0.516 | 0.529 | 0.602 |
Method | FLOPs | Paras (M) | FPS |
---|---|---|---|
YOLOv5s | 190.223G | 7.077 | 49.76 |
YOLOv8n | 94.418G | 3.012 | 65.28 |
Retinanet-ResNet18 | 1.436T | 19.875 | 14.89 |
CenterNet-ResNet18 | 648.185G | 14.128 | 49.35 |
YOLOv5m | 583.382G | 21.077 | 26.26 |
YOLOv8s | 329.994G | 11.138 | 44.79 |
Retinanet-ResNet34 | 1.785T | 29.983 | 13.51 |
CenterNet-ResNet34 | 996.870G | 24.236 | 34.74 |
YOLO V5l | 1.320T | 46.658 | 17.31 |
YOLO V8m | 910.626G | 25.860 | 25.8 |
Retinanet-ResNet50 | 1.904T | 36.434 | 12.7 |
CenterNet-ResNet50 | 1.336T | 33.009 | 20.26 |
GCC-YOLO | 202.972G | 6.812 | 35.29 |
Ours | 45.316G | 0.542 | 51.18 |
Model | Darknet53-PANet | DA | Remove P5 | BRA | CLo | LGT | LWH | 1/2 Size | FLOPs (G) | Paras (M) | mAP [IoU = 0.5] (%) |
---|---|---|---|---|---|---|---|---|---|---|---|
Baseline | 648.185 | 14.128 | 0.9682 | ||||||||
Model1 | √ | 134.148 | 2.425 | 0.9711 | |||||||
Model2 | √ | √ | 134.148 | 2.425 | 0.9811 | ||||||
Model3 | √ | √ | √ | 110.273 | 0.643 | 0.9832 | |||||
Model4 | √ | √ | √ | √ | 109.381 | 0.620 | 0.9839 | ||||
Model5 | √ | √ | √ | √ | 111.177 | 0.680 | 0.9843 | ||||
Model6 | √ | √ | √ | √ | 109.952 | 0.649 | 0.9863 | ||||
Model7 | √ | √ | √ | √ | √ | 45.316 | 0.542 | 0.9895 | |||
Model8 | √ | √ | √ | √ | √ | √ | 11.329 | 0.542 | 0.9832 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, W.; Meng, S.; Wang, X. Local and Global Context-Enhanced Lightweight CenterNet for PCB Surface Defect Detection. Sensors 2024, 24, 4729. https://doi.org/10.3390/s24144729
Chen W, Meng S, Wang X. Local and Global Context-Enhanced Lightweight CenterNet for PCB Surface Defect Detection. Sensors. 2024; 24(14):4729. https://doi.org/10.3390/s24144729
Chicago/Turabian StyleChen, Weixun, Siming Meng, and Xueping Wang. 2024. "Local and Global Context-Enhanced Lightweight CenterNet for PCB Surface Defect Detection" Sensors 24, no. 14: 4729. https://doi.org/10.3390/s24144729