Defect Detection in Printed Circuit Boards Using Semi-Supervised Learning
<p>Examples of (<b>a</b>) strip-level PCB images and (<b>b</b>) non-defect and (<b>c</b>) defect images acquired by the AFVI system: the red dashed boxes in (<b>c</b>) represent the defect locations on the circuit board.</p> "> Figure 2
<p>Augmentation process used in this study: (<b>a</b>) augmentation for labeled data and unlabeled data for pseudo prediction (<math display="inline"><semantics> <mrow> <msub> <mi>ψ</mi> <mn>1</mn> </msub> </mrow> </semantics></math>), (<b>b</b>) augmentation for unlabeled data (<math display="inline"><semantics> <mrow> <msub> <mi>ψ</mi> <mn>2</mn> </msub> </mrow> </semantics></math>), and (<b>c</b>) examples of images augmented by the two types.</p> "> Figure 3
<p>Pipelines for the (<b>a</b>) PCB_FS and (<b>b</b>) PCB_SS models; (<b>c</b>) classification process of the validation and test datasets for the trained model from either (<b>a</b>) PCB_FS or (<b>b</b>) PCB_SS (<math display="inline"><semantics> <mrow> <msub> <mi>x</mi> <mi>l</mi> </msub> </mrow> </semantics></math>: labeled image, <math display="inline"><semantics> <mrow> <msub> <mi>y</mi> <mi>l</mi> </msub> </mrow> </semantics></math>: target label, <math display="inline"><semantics> <mrow> <msub> <mi>x</mi> <mi>u</mi> </msub> </mrow> </semantics></math>: unlabeled image, <math display="inline"><semantics> <mrow> <msub> <mi>y</mi> <mi>u</mi> </msub> </mrow> </semantics></math>: pseudo-label, <span class="html-italic">p</span>: probabilities, <math display="inline"><semantics> <mi>τ</mi> </semantics></math>: threshold of pseudo-label, WRN-28-2: WideResnet-28-2, x, y: validation and test images and their labels).</p> "> Figure 4
<p>Architecture of WideResNet-28-2 (WRN-28-2) (FC: fully connected layer).</p> "> Figure 5
<p>Learning curves during training for Label_250, Label_1000, and Label_4000: (<b>a</b>–<b>c</b>) total losses on training data (solid lines) and validation losses (dotted lines), (<b>d</b>–<b>f</b>) validation accuracies (circle and triangle: training steps of top three accuracies for PCB_FS and PCB_SS, respectively), and (<b>g</b>–<b>i</b>) unsupervised losses on the unlabeled data of the PCB_SS model.</p> "> Figure 6
<p>Confusion matrix of the (<b>a</b>) PCB_FS and (<b>b</b>) PCB_SS models, and (<b>c</b>) ROC curves for the two models (solid: PCB_FS, dotted: PCB_SS).</p> "> Figure 7
<p>Error rates (%) of the test data with different ratios of noisy data.</p> "> Figure 8
<p>Original images (top row) and Grad-CAM maps obtained using the PCB_FS model (middle row) and PCB_SS model (bottom row) (<b>a</b>) true predictions for both PCB_FS and PCB_SS models, (<b>b</b>–<b>d</b>) false and true predictions for PCB_FS and PCB_SS models, respectively (green solid box: true prediction, red dotted box: false prediction).</p> "> Figure 9
<p>Examples of defect images (<b>a</b>,<b>b</b>) misclassified by both models.</p> ">
Abstract
:1. Introduction
- (1)
- An SSL approach is developed for significantly varying defect cases with small differences from the non-defect cases, where the golden template and machine-learning approaches are less effective.
- (2)
- By employing the FixMatch concept with varying augmentations, the PCB_SS model was trained using images classified as defects from the AFVI system with unlabeled and manually labeled target data.
- (3)
- An error-resilience test was performed for false detection. The PCB_SS model outperformed PCB_FS for incorrectly labeled data.
- (4)
- By alleviating the false defect detection from the AFVI systems, PCB_SS can reduce the burden of manual defect detection, requiring intensive labor with expertise in PCB inspection.
2. Materials and Methods
2.1. Data Generation
2.1.1. Data Acquisition
2.1.2. Data Preparation
2.2. Deep-Learning Approach
2.2.1. PCB_FS Model
2.2.2. PCB_SS Model
2.2.3. Network Training
Algorithm 1 Pseudocode of PCB_SS model |
|
2.3. Performance Evaluation
3. Results
3.1. Learning Curves
3.2. Performance Evaluation
3.3. Parameter Optimization
3.4. Gradient Visualization (Grad-CAM)
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Moganti, M.; Erçal, F.; Dagli, C.H.; Tsunekawa, S. Automatic PCB Inspection Algorithms: A Survey. Comput. Vis. Image Underst. 1996, 63, 287–313. [Google Scholar] [CrossRef] [Green Version]
- Zheng, X.; Zheng, S.; Kong, Y.; Chen, J. Recent advances in surface defect inspection of industrial products using deep learning techniques. Int. J. Adv. Manuf. Technol. 2021, 113, 35–58. [Google Scholar] [CrossRef]
- Huang, S.-H.; Pan, Y.-C. Automated visual inspection in the semiconductor industry: A survey. Comput. Ind. 2015, 66, 1–10. [Google Scholar] [CrossRef]
- Wang, Y.; Wang, M.; Zhang, Z. Microfocus X-ray printed circuit board inspection system. Optik 2014, 125, 4929–4931. [Google Scholar] [CrossRef]
- Wang, F.; Yue, Z.; Liu, J.; Qi, H.; Sun, W.; Chen, M.; Wang, Y.; Yue, H. Quantitative imaging of printed circuit board (PCB) delamination defects using laser-induced ultrasound scanning imaging. J. Appl. Phys. 2022, 131, 053101. [Google Scholar] [CrossRef]
- Wagh, C.R.; Baru, V.B. Detection of Faulty Region on Printed Circuit Board With IR Thermography. Int. J. Sci. Eng. Res. 2013, 4, 544523. [Google Scholar]
- Nadaf, M.; Kolkure, M.V.S. Detection of Bare PCB Defects by using Morphology Technique. Bus. Mater. Sci. 2016, 120142102. [Google Scholar]
- Benedek, C. Detection of soldering defects in Printed Circuit Boards with Hierarchical Marked Point Processes. Pattern Recognit. Lett. 2011, 32, 1535–1543. [Google Scholar] [CrossRef]
- Oguz, S.H.; Onural, L. An automated system for design-rule-based visual inspection of printed circuit boards. In Proceedings of the 1991 IEEE International Conference on Robotics and Automation, Sacramento, CA, USA, 9–11 April 1991; Volume 2693, pp. 2696–2701. [Google Scholar]
- Deng, Y.-S.; Luo, A.-C.; Dai, M.-J. Building an Automatic Defect Verification System Using Deep Neural Network for PCB Defect Classification. In Proceedings of the 2018 4th International Conference on Frontiers of Signal Processing (ICFSP), Poitiers, France, 24–27 September 2018; pp. 145–149. [Google Scholar]
- Zhang, Z.-Q.; Wang, X.; Liu, S.; Sun, L.; Chen, L.; Guo, Y.-M. An Automatic Recognition Method for PCB Visual Defects. In Proceedings of the 2018 International Conference on Sensing, Diagnostics, Prognostics, and Control (SDPC), Xi’an, China, 15–17 August 2018; pp. 138–142. [Google Scholar]
- Taha, E.M.; Emary, E.E.; Moustafa, K. Automatic Optical Inspection for PCB Manufacturing: A Survey. Int. J. Sci. Eng. Res. 2014, 5, 1095–1102. [Google Scholar]
- Imoto, K.; Nakai, T.; Ike, T.; Haruki, K.; Sato, Y. A CNN-Based Transfer Learning Method for Defect Classification in Semiconductor Manufacturing. IEEE Trans. Semicond. Manuf. 2019, 32, 455–459. [Google Scholar] [CrossRef]
- Miao, Y.; Liu, Z.; Wu, X.; Gao, J. Cost-Sensitive Siamese Network for PCB Defect Classification. Comput. Intell. Neurosci. 2021, 2021, 7550670. [Google Scholar] [CrossRef]
- He, F.; Tang, S.; Mehrkanoon, S.; Huang, X.; Yang, J. A Real-time PCB Defect Detector Based on Supervised and Semi-supervised Learning. In Proceedings of the ESANN, Bruges, Belgium, 2–4 October 2020. [Google Scholar]
- Shi, W.; Zhang, L.; Li, Y.; Liu, H. Adversarial semi-supervised learning method for printed circuit board unknown defect detection. J. Eng. 2020, 2020, 505–510. [Google Scholar] [CrossRef]
- Xie, Q.; Dai, Z.; Hovy, E.H.; Luong, M.-T.; Le, Q.V. Unsupervised Data Augmentation for Consistency Training. arXiv 2019, arXiv:1904.12848. [Google Scholar]
- Sohn, K.; Berthelot, D.; Li, C.-L.; Zhang, Z.; Carlini, N.; Cubuk, E.D.; Kurakin, A.; Zhang, H.; Raffel, C. FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. arXiv 2020, arXiv:abs/2001.07685. [Google Scholar]
- Cubuk, E.D.; Zoph, B.; Shlens, J.; Le, Q.V. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA, 14–19 June 2020; pp. 3008–3017. [Google Scholar]
- Zagoruyko, S.; Komodakis, N. Wide Residual Networks. arXiv 2016, arXiv:abs/1605.07146. [Google Scholar]
- Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the AISTATS, Sardinia, Italy, 13–15 May 2010. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7–13 December 2015; pp. 1026–1034. [Google Scholar]
- Loshchilov, I.; Hutter, F. SGDR: Stochastic Gradient Descent with Warm Restarts. arXiv 2017, arXiv:1608.03983. [Google Scholar]
- Tarvainen, A.; Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proceedings of the NIPS, Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Fawcett, T. An introduction to ROC analysis. Pattern Recognit. Lett. 2006, 27, 861–874. [Google Scholar] [CrossRef]
- Chattopadhyay, A.; Sarkar, A.; Howlader, P.; Balasubramanian, V.N. Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks. arXiv 2018, arXiv:1710.11063. [Google Scholar]
- Chiu, L.C.; Chang, T.S.; Chen, J.Y.; Chang, N.Y. Fast SIFT design for real-time visual feature extraction. IEEE Trans. Image Process. 2013, 22, 3158–3167. [Google Scholar] [CrossRef] [PubMed]
- Chang, C.-C.; Lin, C.-J. Libsvm: A library for Support Vector Machines. ACM Trans. Intell. Syst. Technol. 2011, 2, 1–27. [Google Scholar] [CrossRef]
- Chen, T.; Guestrin, C. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 785–794. [Google Scholar]
- Rosebrock, A. Grid Search Hyperparameter Tuning with Scikit-Learn (GridSearchCV). Available online: https://www.pyimagesearch.com/2021/05/24/grid-search-hyperparameter-tuning-with-scikit-learn-gridsearchcv/ (accessed on 1 April 2022).
- Yang, X.; Song, Z.; King, I.; Xu, Z. A Survey on Deep Semi-supervised Learning. arXiv 2021, arXiv:abs/2103.00550. [Google Scholar] [CrossRef]
- Pham, H.; Xie, Q.; Dai, Z.; Le, Q.V. Meta Pseudo Labels. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 21–25 June 2021; pp. 11552–11563. [Google Scholar]
- Lee, D.; Kim, S.; Kim, I.; Cheon, Y.; Cho, M.; Han, W.-S. Contrastive Regularization for Semi-Supervised Learning. arXiv 2022, arXiv:abs/2201.06247. [Google Scholar]
- Amorim, W.P.; Falcão, A.X.; Papa, J.P.; Carvalho, M.H. Improving semi-supervised learning through optimum connectivity. Pattern Recognit. 2016, 60, 72–85. [Google Scholar] [CrossRef] [Green Version]
- Jin, J.; Feng, W.; Lei, Q.; Gui, G.; Li, X.; Deng, Z.; Wang, W. Defect Detection of Printed Circuit Boards Using EfficientDet. In Proceedings of the 2021 IEEE 6th International Conference on Signal and Image Processing (ICSIP), Nanjing, China, 9–11 July 2021; pp. 287–293. [Google Scholar]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, BC, Canada, 11–17 October 2021; pp. 9992–10002. [Google Scholar]
- Tan, M.; Le, Q.V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv 2019, arXiv:abs/1905.11946. [Google Scholar]
- Huang, W.; Wei, P. A PCB Dataset for Defects Detection and Classification. arXiv 2019, arXiv:abs/1901.08204. [Google Scholar]
Purpose | Set | Defect | Non-Defect | Total |
---|---|---|---|---|
Train | Label_250 | 125 | 125 | 250 |
Label_500 | 250 | 250 | 500 | |
Label_1000 | 500 | 500 | 1000 | |
Label_2000 | 1000 | 1000 | 2000 | |
Label_4000 | 2000 | 2000 | 4000 | |
Unlabeled | 10,791 | 6118 | 16,909 | |
Validation | 250 | 250 | 500 | |
Test | 750 | 750 | 1500 | |
Total | 13,791 | 9118 | 22,909 |
Set with Noisy | Defect | Non-Defect | Total | ||
---|---|---|---|---|---|
Correct | Noisy | Correct | Noisy | ||
0% | 2000 | 0 | 2000 | 0 | 4000 |
3% | 1940 | 60 | 1940 | 60 | 4000 |
9% | 1820 | 180 | 1820 | 180 | 4000 |
12.5% | 1750 | 250 | 1750 | 250 | 4000 |
Unlabeled | 10,791 | 6118 | 16,909 |
Transformation | Range |
---|---|
Rotation ) | [−30, 30] |
Translation ) | [−0.3, 0.3] |
Translation ) | [−0.45, 0.45] |
Scaling | [0.9, 1.1] |
Shearing | [−0.3, 0.3] |
Brightness Color Contrast Sharpness | [0.1, 1.9] |
Configuration | PCB_FS Model | PCB_SS Model |
---|---|---|
Loss function | Cross entropy (Equation (1)) | Cross entropy-based (Equation (6)) |
Optimizer | SGD | SGD |
Initial learning rate | 0.001 | 0.001 |
Learning rate schedule | ||
Batch size | 16 | 16 (labeled) & 144 (unlabeled) |
Total number of training steps | 65,536 | 65,536 |
Number of evaluation steps | 256 | 256 |
Threshold | - | 0.9 |
Coefficient of unlabeled batch size | - | 9 |
Coefficient of unlabeled loss | - | 1 |
EMA decay coefficient | 0.999 | 0.999 |
Weight decay coefficient | 0.0005 | 0.0005 |
Model | Label_250 | Label_500 | Label_1000 | Label_2000 | Label_4000 |
---|---|---|---|---|---|
PCB_FS | 22.27 ± 0.33 | 18.82 ± 0.31 | 16.36 ± 0.49 | 13.00 ± 0.55 | 11.22 ± 0.91 |
PCB_SS | 14.02 ± 0.11 | 11.98 ± 0.88 | 12.29 ± 0.60 | 8.71 ± 0.44 | 7.82 ± 0.08 |
Metric | XG-Boost | EfficientNet | PCB_FS | PCB_SS |
---|---|---|---|---|
Accuracy (%) | 80.07 | 88.60 | 89.67 | 92.27 |
Recall | 0.80 | 0.88 | 0.90 | 0.94 |
Precision | 0.81 | 0.88 | 0.89 | 0.91 |
AUC | 0.96 | 0.95 | 0.96 | 0.97 |
Model | Noisy (0%) | Noisy (3%) | Noisy (9%) | Noisy (12.5%) |
---|---|---|---|---|
PCB_FS | 11.22 ± 0.91 | 13.04 ± 0.38 | 15.18 ± 1.15 | 15.60 ± 0.48 |
PCB_SS | 7.82 ± 0.08 | 8.27 ± 0.25 | 8.04 ± 0.42 | 15.11 ± 0.99 |
Model | Noisy (0%) | Noisy (3%) | Noisy (9%) | Noisy (12.5%) |
---|---|---|---|---|
PCB_FS | 11.29 ± 1.34 | 13.2 ± 0.27 | 15.88 ± 1.01 | 16.85 ± 0.27 |
PCB_SS | 8.37 ± 0.49 | 8.18 ± 0.56 | 8.96 ± 0.84 | 14.22 ± 1.31 |
Augmentation | PCB_FS | PCB_SS |
---|---|---|
13.6 ± 0.23 | 15.78 ± 0.867 | |
14.98 ± 1.01 | 7.82 ± 0.08 | |
11.2 ± 0.91 | 9.36 ± 1.47 |
Threshold (τ) | Recall | Precision | Error Rate |
---|---|---|---|
0.4 | 0.91 ± 0.02 | 0.94 ± 0.01 | 7.44 ± 0.64 |
0.7 | 0.91 ± 0.02 | 0.94 ± 0.01 | 6.95 ± 0.65 |
0.9 | 0.93 ± 0.01 | 0.92 ± 0.01 | 7.82 ± 0.08 |
1.0 | 0.85 ± 0.02 | 0.84 ± 0.01 | 15.36 ± 0.51 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Pham, T.T.A.; Thoi, D.K.T.; Choi, H.; Park, S. Defect Detection in Printed Circuit Boards Using Semi-Supervised Learning. Sensors 2023, 23, 3246. https://doi.org/10.3390/s23063246
Pham TTA, Thoi DKT, Choi H, Park S. Defect Detection in Printed Circuit Boards Using Semi-Supervised Learning. Sensors. 2023; 23(6):3246. https://doi.org/10.3390/s23063246
Chicago/Turabian StylePham, Thi Tram Anh, Do Kieu Trang Thoi, Hyohoon Choi, and Suhyun Park. 2023. "Defect Detection in Printed Circuit Boards Using Semi-Supervised Learning" Sensors 23, no. 6: 3246. https://doi.org/10.3390/s23063246