Abstract
Automatic micro-defect detection is crucial for promoting efficiency in the production lines of patterned OLED panels. Recently, deep learning algorithms have emerged as promising solutions for micro-defect detection. However, in real-world industrial scenarios, the scarcity of training data or annotations results in a drop in performance. A multi-stage few-shot micro-defect detection approach is proposed for patterned OLED panels to deal with this problem. Firstly, we introduce a converter from defective to defect-free images based on our redesigned Vector Quantized-Variational AutoEncoder (VQ-VAE), aiming to inpaint defects with normal textures. Next, we exploit a region-growing method with automatic seed points to obtain the defect’s segmentation and geometric parameters in each image block. Reliable seed points are provided by structural similarity index maps between defective sub-blocks and reconstructed reference. Finally, a multi-scale Siamese neural network is proposed to identify the category of extracted defects. With our proposed approach, detection and classification results of defects can be obtained successively. Our experimental results on samples at different array processes demonstrate the superb adaptability of VQ-VAE, with a defect detection rate ranging from 90.0% to 96.0%. Additionally, compared with existing classification models, our multi-scale Siamese neural network exhibits an impressive 98.6% classification accuracy for a long-tailed defect dataset without overfitting. In summary, the proposed approach shows great potential for practical micro-defect detection in industrial scenarios with limited training data.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The datasets generated and analyzed during the current study are not publicly available due the fact that they constitute an excerpt of research in progress but are available from the corresponding author on reasonable request.
References
Antoniou, A., Storkey, A., & Edwards, H. (2018). Data augmentation generative adversarial networks. arXiv:1711.04340.
Bao, Y., Song, K., Liu, J., Wang, Y., Yan, Y., Yu, H., & Li, X. (2021). Triplet-graph reasoning network for few-shot metal generic surface defect segmentation. IEEE Transactions on Instrumentation and Measurement, 70, 1–11. https://doi.org/10.1109/TIM.2021.3083561
Bottou, L., & Bousquet, O. (2008). The tradeoffs of large scale learning. In NIPS'07: Proceedings of the 20th international conference on neural information processing systems (pp. 161–168).
Cen, Y., Zhao, R., Cen, L., Cui, L., Miao, Z., & Wei, Z. (2015). Defect inspection for TFT-LCD images based on the low-rank matrix reconstruction. Neurocomputing, 149, 1206–1215. https://doi.org/10.1016/j.neucom.2014.09.007
Chen, Z., Fu, Y., Zhang, Y., Jiang, Y., Xue, X., & Sigal, L. (2019). Multi-level semantic feature augmentation for one-shot learning. IEEE Transactions on Image Processing, 28, 4594–4605. https://doi.org/10.1109/TIP.2019.2910052
Ding, R., Dai, L., Li, G., & Liu, H. (2019). TDD-net: A tiny defect detection network for printed circuit boards. CAAI Transactions on Intelligence Technology, 4(2), 110–116. https://doi.org/10.1049/trit.2019.0019
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 770–778. https://doi.org/10.1109/CVPR.2016.90
Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (GELUs). arXiv. https://doi.org/10.48550/arXiv.1606.08415
Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., Adam, H., & Le, Q. (2019). Searching for MobileNetV3. IEEE/CVF International Conference on Computer Vision (ICCV), 2019, 1314–1324. https://doi.org/10.1109/ICCV.2019.00140
Huang, H., Zhang, J., Yu, L., Zhang, J., Wu, Q., & Xu, C. (2022). TOAN: Target-oriented alignment network for fine-grained image categorization with few labeled samples. IEEE Transactions on Circuits and Systems for Video Technology, 32(2), 853–866. https://doi.org/10.1109/TCSVT.2021.3065693
Kingma, D., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv. https://doi.org/10.48550/arXiv.1412.6980
Krizhevsky, A., Sutskever, I., & Hinton, G. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90. https://doi.org/10.1145/3065386
Li, K., Jiang, X., Chen, L., Wang, S., Wang, S., et al. (2022). Wafer defect pattern labeling and recognition using semi-supervised learning. IEEE Transactions on Semiconductor Manufacturing, 35(2), 291–299. https://doi.org/10.1109/TSM.2022.3159246
Ling, Z., Zhang, A., Ma, D., Shi, Y., & Wen, H. (2022). Deep siamese semantic segmentation network for PCB welding defect detection. IEEE Transactions on Instrumentation and Measurement, 71, 1–11. https://doi.org/10.1109/TIM.2022.3154814
Liu, J., Wang, C., Su, H., Du, B., & Tao, D. (2020). Multi-stage GAN for fabric defect detection. IEEE Transactions on Image Processing, 29, 3388–3400. https://doi.org/10.1109/TIP.2019.2959741
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. IEEE/CVF International Conference on Computer Vision (ICCV), 2021, 9992–10002. https://doi.org/10.1109/ICCV48922.2021.00986
Liu, Z., Mao, H., Wu, C., Feichtenhofer, C., Darrell, T., & Xie, S. (2022a). A ConvNet for the 2020s. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, 11966–11976. https://doi.org/10.1109/CVPR52688.2022.01167
Liu, Z., Song, Y., Tang, R., et al. (2022b). Few-shot defect recognition of metal surfaces via attention-embedding and self-supervised learning. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-022-02022-y
Lu, C., & Tsai, D. (2005). Automatic defect inspection for LCDs using singular value decomposition. The International Journal of Advanced Manufacturing Technology (AMT), 25, 53–61. https://doi.org/10.1007/s00170-003-1832-6
Luo, J., Yang, Z., Li, S., & Wu, Y. (2021). FPCB surface defect detection: A decoupled two-stage object detection framework. IEEE Transactions on Instrumentation and Measurement, 70, 1–11. https://doi.org/10.1109/TIM.2021.3092510
Min, Y., & Li, Y. (2022). Self-supervised railway surface defect detection with defect removal variational autoencoders. Energies, 15, 3592. https://doi.org/10.3390/en15103592
Oord, A., Vinyals, Oriol., & Kavukcuoglu, K. (2017). Neural discrete representation learning. The 31st international conference on neural information processing systems (pp. 6309–6318).
Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context encoders: Feature learning by inpainting. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 2536–2544. https://doi.org/10.1109/CVPR.2016.278
Sandfort, V., Yan, K., Pickhardt, P. J., & Summers, R. M. (2019). Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks. Scientific Reports. https://doi.org/10.1038/s41598-019-52737-x
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L. (2018). MobileNetV2: Inverted residuals and linear bottlenecks. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, 4510–4520. https://doi.org/10.1109/CVPR.2018.00474
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. IEEE International Conference on Computer Vision (ICCV), 2017, 618–626. https://doi.org/10.1109/ICCV.2017.74
Shao, L., Zhang, E., Ma, Q., & Li, M. (2022). Pixel-wise semisupervised fabric defect detection method combined with multitask mean teacher. IEEE Transactions on Instrumentation and Measurement, 71, 1–11. https://doi.org/10.1109/TIM.2022.3162286
Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. The 3rd international conference on learning representations (ICLR2015). https://arxiv.org/abs/1409.1556
Snell, J., Swersky, K., & Zemel, R. (2017). Prototypical networks for few-shot learning. In Proceedings of the 31st international conference on neural information processing systems (pp. 4080–4090).
Song, Y., Liu, Z., Liang, S., Tang, R., Duan, G., & Tan, J. (2022). Coarse-to-fine few-shot defect recognition with dynamic weighting and joint metric. IEEE Transactions on Instrumentation and Measurement, 71, 1–10. https://doi.org/10.1109/TIM.2022.3193204
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2015.7298594
Tan, M., & Le, Q. (2019). EfficientNet: rethinking model scaling for convolutional neural networks. The 36th International Conference on Machine Learning (pp. 6150–6114). arXiv: 1905.11946
Vahdat, A., & Kautz, J. (2020). NVAE: a deep hierarchical variational autoencoder. In NIPS'20: Proceedings of the 34th international conference on neural information processing systems (Vol. 1650, pp.19667–19679).
Vidal, R., Ma, Y., & Sastry, S. (2016). Robust principal component analysis. Journal of the ACM, 58(3), 1–37. https://doi.org/10.1145/1970392
Vinyals, O., Blundell, C., Lillicrap, T., & Kavukcuoglu, K. (2016). Matching networks for one shot learning. In Proceedings of the 30th international conference on neural information processing systems (pp. 3637–3645).
Wang, H., Li, Z., & Wang, H. (2022a). Few-shot steel surface defect detection. IEEE Transactions on Instrumentation and Measurement, 71, 1–12. https://doi.org/10.1109/TIM.2021.3128208
Wang, S., Chen, H., Liu, K., Zhou, Y., & Feng, H. (2022b). Meta-FSDet: A meta-learning based detector for few-shot defects of photovoltaic modules. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-022-02001-3
Wang, S., Zhong, Z., Zhao, Y., & Zuo, L. (2021). A variational autoencoder enhanced deep learning model for wafer defect imbalanced classification. IEEE Transactions on Components, Packaging and Manufacturing Technology, 11(12), 2055–2060. https://doi.org/10.1109/TCPMT.2021.3126083
Wang, Y., Wei, Y., & Wang, H. (2023). A class imbalanced wafer defect classification framework based on variational autoencoder generative adversarial network. Measurement Science and Technology. https://doi.org/10.1088/1361-6501/ac9ed3
Wang, Y., Yao, Q., Kwok, J., & Ni, L. (2020). Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys, 53(3), 1–34. https://doi.org/10.1145/3386252
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612. https://doi.org/10.1109/TIP.2003.819861
Wu, X., Qiu, L., Gu, X., & Long, Z. (2021). Deep learning-based generic automatic surface defect detection (ASDI) with pixelwise segmentation. IEEE Transactions on Instrumentation and Measurement, 70, 1–10. https://doi.org/10.1109/TIM.2020.3026801
Wu, X., Wang, T., Li, Y., Li, P., & Liu, Y. (2022). A CAM-based weakly supervised method for surface defect inspection. IEEE Transactions on Instrumentation and Measurement, 71, 1–10. https://doi.org/10.1109/TIM.2022.3168895
Xiao, W., Song, K., Liu, J., & Yan, Y. (2022). Graph embedding and optimal transport for few-shot classification of metal surface defect. IEEE Transactions on Instrumentation and Measurement, 71, 1–10. https://doi.org/10.1109/TIM.2022.3169547
Yang, H., Song, K., Mao, F., & Yin, Z. (2021). Autolabeling-enhanced active learning for cost-efficient surface defect visual classification. IEEE Transactions on Instrumentation and Measurement, 70, 1–15. https://doi.org/10.1109/TIM.2020.3032190
Yu, R., Guo, B., & Yang, K. (2022). Selective prototype network for few-shot metal surface defect segmentation. IEEE Transactions on Instrumentation and Measurement, 71, 1–10. https://doi.org/10.1109/TIM.2022.3196447
Zhang, G., Cui, K., Hung, T., & Lu, S. (2021a). Defect-GAN: high-fidelity defect synthesis for automated defect detection. IEEE Winter Conference on Applications of Computer Vision (WACV). https://doi.org/10.1109/WACV48630.2021.00257
Zhang, J., Su, H., Zou, W., Gong, X., Zhang, Z., & Shen, F. (2021b). CADN: A weakly supervised learning-based category-aware object detection network for surface defect detection. Pattern Recognition, 109, 1–10. https://doi.org/10.1016/j.patcog.2020.107571
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018). ShuffleNet: An extremely efficient convolutional neural network for mobile devices. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, 6848–6856. https://doi.org/10.1109/CVPR.2018.00716
Zhao, W., Song, K., Wang, Y., Liang, S., & Yan, Y. (2023). FaNet: feature-aware network for few shot classification of strip steel surface defects. Measurement. https://doi.org/10.1016/j.measurement.2023.112446
Zheng, Y., & Cui, L. (2022). Defect detection on new samples with siamese defect-aware attention network. Applied Intelligence, 53, 4563–4578. https://doi.org/10.1007/s10489-022-03595-0
Zhou, T., & Tao, D. (2011). GoDec: randomized low-rank & sparse matrix decomposition in noisy case. In ICML'11: Proceedings of the 28th international conference on international conference on machine learning (pp. 33–40).
Zhou, Z. (2017). A brief introduction to weakly supervised learning. National Science Review, 5, 44–53. https://doi.org/10.1093/nsr/nwx106
Zhu, J., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. IEEE International Conference on Computer Vision (ICCV), 2017, 2242–2251. https://doi.org/10.1109/ICCV.2017.244
Funding
This work is supported by National Natural Science Foundation of China (52275527, 51975161 and 52275526); Key Research and Development Program of Heilongjiang (Grant No. 2022ZX01A27); CGN-HIT Advanced Nuclear and New Energy Research Institute (CGN-HIT202201).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ye, S., Wang, Z., Xiong, P. et al. Multi-stage few-shot micro-defect detection of patterned OLED panel using defect inpainting and multi-scale Siamese neural network. J Intell Manuf 35, 2653–2669 (2024). https://doi.org/10.1007/s10845-023-02168-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-023-02168-3