Abstract
Defect detection is an essential way to ensure the quality of parts made by laser powder bed fusion (LPBF) and industrial cameras are one of the commonly used tools for defect monitoring. Different lighting environments affect the visibility of defects in the images, and the illumination condition becomes one of the most important factors affecting the defect detection effect of industrial cameras, but the modification of the equipment lighting environment will increase the complexity and cost of monitoring. In this study, only an off-axis CMOS camera monitoring system is used and the lighting facilities are not changed to improve the effect of defect detection under uneven lighting conditions. A dual-input convolutional neural network fusing defect parameter vectors and layerwise images is proposed for real-time online monitoring of defects in the LPBF process using a paraxial CMOS camera monitoring system. The model integrates the image and the parameter information related to defect generation, and can distinguish some defects that are not easily discerned by images alone. To a certain extent, it avoids the problem that the same defects are visually indistinguishable in images caused by uneven light distribution and reflections on metal surfaces. The results indicate that the method has better performance than the method with a single image input, with recognition accuracies above 80.00% for all defect categories. In addition, the method is more suitable for real-time online monitoring scenarios due to its low parameter number, short training time and fast prediction speed compared to classical deep learning algorithms.
Similar content being viewed by others
Data availability
The data set used in this study is not publicly available at this time and may be obtained from the corresponding author upon reasonable request and permission of the authors.
References
Aminzadeh, M., & Kurfess, T. R. (2019). Online quality inspection using Bayesian classification in powder-bed additive manufacturing from high-resolution visual camera images. Journal of Intelligent Manufacturing, 30(6), 2505–2523. https://doi.org/10.1007/s10845-018-1412-0
Batista, G., Bazzan, A., & Monard, M.-C. (2003). Balancing Training Data for Automated Annotation of Keywords: a Case Study. In the Proc. Of Workshop on Bioinformatics (pp. 10–18).
Caggiano, A., Zhang, J., Alfieri, V., Caiazzo, F., Gao, R., & Teti, R. (2019). Machine learning-based image processing for on-line defect recognition in additive manufacturing. CIRP Annals. https://doi.org/10.1016/j.cirp.2019.03.021
Chen, R., Imani, F., Reutzel, E. W., & Yang, H. (2018). From design complexity to build quality in additive manufacturing—A sensor-based perspective. IEEE Sensors Journal, 3(1), 1–4. https://doi.org/10.1109/LSENS.2018.2880747
Chen, R., Rao, P., Lu, Y., Reutzel, E. W., & Yang, H. (2021). Recurrence network analysis of design-quality interactions in additive manufacturing. Additive Manufacturing, 39, 101861. https://doi.org/10.1016/j.addma.2021.101861
Coeck, S., Bisht, M., Plas, J., & Verbist, F. (2019). Prediction of lack of fusion porosity in selective laser melting based on melt pool monitoring data. Additive Manufacturing, 25, 347–356. https://doi.org/10.1016/j.addma.2018.11.015
Conner, B. P., Manogharan, G. P., Martof, A. N., Rodomsky, L. M., Rodomsky, C. M., Jordan, D. C., & Limperos, J. W. (2014). Making sense of 3-D printing: Creating a map of additive manufacturing products and services. Additive Manufacturing, 1–4, 64–76. https://doi.org/10.1016/j.addma.2014.08.005
Craeghs, T., Clijsters, S., Yasa, E., & Kruth, J.-P. (2011). Online quality control of selective laser melting. In 22nd Annual International Solid Freeform Fabrication Symposium—An Additive Manufacturing Conference, SFF 2011.
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., & Belongie, S. (2019). Class-balanced loss based on effective number of samples. arXiv e-prints, arXiv:1901.05555.
Davis, G., Nagarajah, R., Palanisamy, S., Rashid, R. A. R., Rajagopal, P., & Balasubramaniam, K. (2019). Laser ultrasonic inspection of additive manufactured components. The International Journal of Advanced Manufacturing Technology, 102(5), 2571–2579. https://doi.org/10.1007/s00170-018-3046-y
Terris, T. de, Andreau, O., Peyre, P., Adamski, F., Koutiri, I., Gorny, C., & Dupuy, C. (2019). Optimization and comparison of porosity rate measurement methods of Selective Laser Melted metallic parts. Additive Manufacturing, 28, 802–813. https://doi.org/10.1016/j.addma.2019.05.035
Ding, X., Guo, Y., Ding, G., & Han, J. (2019). ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 1911–1920). https://doi.org/10.1109/ICCV.2019.00200
Dowling, L., Kennedy, J., O’Shaughnessy, S., & Trimble, D. (2019). A review of critical repeatability and reproducibility issues in powder bed fusion. Materials & Design, 186, 108346. https://doi.org/10.1016/j.matdes.2019.108346
Everton, S. K., Hirsch, M., Stravroulakis, P., Leach, R. K., & Clare, A. T. (2016). Review of in-situ process monitoring and in-situ metrology for metal additive manufacturing. Materials & Design, 95, 431–445. https://doi.org/10.1016/j.matdes.2016.01.099
Fischer, F. G., Zimmermann, M. G., Praetzsch, N., & Knaak, C. (2022). Monitoring of the powder bed quality in metal additive manufacturing using deep transfer learning. Materials & Design, 222, 111029. https://doi.org/10.1016/j.matdes.2022.111029
Foster, B. K., Reutzel, E. W., Nassar, A. R., Hall, B. T., Brown, S. W., & Dickman, C. J. (2015). Optical, layerwise monitoring of powder bed fusion. Solid Freeform Fabrication Symposium Proceedings, Austin, TX, 295–307. https://doi.org/10.1017/CBO9781107415324.004
Gobert, C., Reutzel, E. W., Petrich, J., Nassar, A. R., & Phoha, S. (2018). Application of supervised machine learning for defect detection during metallic powder bed fusion additive manufacturing using high resolution imaging. Additive Manufacturing, 21, 517–528. https://doi.org/10.1016/j.addma.2018.04.005
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., et al. (2014). Generative Adversarial Networks. arXiv e-prints, arXiv:1406.2661.
Grasso, M., & Colosimo, B. (2017). Process defects and In-situ monitoring methods in metal powder bed fusion: A review. Measurement Science and Technology, 28, 1–25. https://doi.org/10.1088/1361-6501/aa5c4f
Grasso, M., Laguzza, V., Semeraro, Q., & Colosimo, B. (2016). In-process monitoring of selective laser melting: Spatial detection of defects via image data analysis. Journal of Manufacturing Science and Engineering, 139, 051001–051011. https://doi.org/10.1115/1.4034715
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., et al. (2018). Recent advances in convolutional neural networks. Pattern Recognition: The Journal of the Pattern Recognition Society, 77, 354–377.
Guerra, M. G., Errico, V., Fusco, A., Lavecchia, F., Campanelli, S. L., & Galantucci, L. M. (2022). High resolution-optical tomography for in-process layerwise monitoring of a laser-powder bed fusion technology. Additive Manufacturing, 55, 102850. https://doi.org/10.1016/j.addma.2022.102850
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 770–778. https://doi.org/10.1109/CVPR.2016.90
Imani, F., Chen, R., Diewald, E. P., Reutzel, E. W., & Yang, H. (2019). Deep learning of Variant Geometry in Layerwise Imaging Profiles for Additive Manufacturing Quality Control. Journal of Manufacturing Science and Engineering, 141(11), 1. https://doi.org/10.1115/1.4044420
Imani, F., Gaikwad, A., Montazeri, M., Rao, P., Yang, H., & Reutzel, E. (2018). Process mapping and in-process monitoring of porosity in laser powder bed fusion using layerwise optical imaging. Journal of Manufacturing Science and Engineering. American Society of Mechanical Engineers (ASME). https://doi.org/10.1115/1.4040615
Jo, D. T., & Japkowicz, N. (2004). Class imbalances versus small disjuncts. SIGKDD Explorations, 6, 40–49. https://doi.org/10.1145/1007730.1007737
Kleszczynski, S., Zur Jacobsmühlen, J., Sehrt, J., & Witt, G. (2012). Error detection in laser beam melting systems by high resolution imaging. In 23rd Annual International Solid Freeform Fabrication Symposium—An Additive Manufacturing Conference, SFF 2012 (pp. 975–987).
Krizhevsky, A., Sutskever, I., & Hinton, G. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in neural information processing systems, 25(2).
Li, J., Zhou, Q., Huang, X., Li, M., & Cao, L. (2023). In situ quality inspection with layer-wise visual images based on deep transfer learning during selective laser melting. Journal of Intelligent Manufacturing, 34(2), 853–867. https://doi.org/10.1007/s10845-021-01829-5
Lin, M., Chen, Q., & Yan, S. (2014). Network In Network. arXiv e-prints. https://doi.org/10.48550/arXiv.1312.4400
Mozaffar, M., Paul, A., Al-Bahrani, R., Wolff, S., Choudhary, A., Agrawal, A., et al. (2018). Data-driven prediction of the high-dimensional thermal history in directed energy deposition processes via recurrent neural networks. Manufacturing Letters, 18, 35–39. https://doi.org/10.1016/j.mfglet.2018.10.002
Pagani, L., Grasso, M., Scott, P. J., & Colosimo, B. M. (2020). Automated layerwise detection of geometrical distortions in laser powder bed fusion. Additive Manufacturing, 36, 101435. https://doi.org/10.1016/j.addma.2020.101435
Petrich, J., Snow, Z., Corbin, D., & Reutzel, E. W. (2021). Multi-modal sensor fusion with machine learning for data-driven process monitoring for additive manufacturing. Additive Manufacturing, 48, 102364. https://doi.org/10.1016/j.addma.2021.102364
Qin, J., Hu, F., Liu, Y., Witherell, P., Wang, C. C. L., Rosen, D. W., et al. (2022). Research and application of machine learning for additive manufacturing. Additive Manufacturing, 52, 102691. https://doi.org/10.1016/j.addma.2022.102691
Razvi, S. S., Feng, S., Narayanan, A., Lee, Y.-T., & Witherell, P. (2019). A Review of Machine Learning Applications in Additive Manufacturing. In Proceedings of the ASME 2019 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (Vol. 1, pp. 18–21). Anaheim, California, USA. https://doi.org/10.1115/DETC2019-98415
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (Vol. abs/1505.04597, pp. 234–241). http://arxiv.org/abs/1505.04597
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018). MobileNetV2: Inverted Residuals and Linear Bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4510–4520). https://doi.org/10.1109/CVPR.2018.00474
Schweier, M., Heins, J. F., Haubold, M. W., & Zaeh, M. F. (2013). Spatter formation in laser welding with beam oscillation. Physics Procedia, 41, 20–30. https://doi.org/10.1016/j.phpro.2013.03.047
Scime, L., & Beuth, J. (2018a). A multi-scale convolutional neural network for autonomous anomaly detection and classification in a laser powder bed fusion additive manufacturing process. Additive Manufacturing, 24, 273–286. https://doi.org/10.1016/j.addma.2018.09.034
Scime, L., & Beuth, J. (2018b). Anomaly detection and classification in a laser powder bed additive manufacturing process using a trained computer vision algorithm. Additive Manufacturing, 19, 114–126. https://doi.org/10.1016/j.addma.2017.11.009
Scime, L., Siddel, D., Baird, S., & Paquit, V. (2020). Layer-wise anomaly detection and classification for powder bed additive manufacturing processes: A machine-agnostic algorithm for real-time pixel-wise semantic segmentation. Additive Manufacturing, 36, 101453. https://doi.org/10.1016/j.addma.2020.101453
Shao, J., Chen, L., & Wu, Y. (2021). SRWGANTV: Image Super-Resolution Through Wasserstein Generative Adversarial Networks with Total Variational Regularization. In 2021 IEEE 13th International Conference on Computer Research and Development (ICCRD) (pp. 21–26). https://doi.org/10.1109/ICCRD51685.2021.9386518
Shelhamer, E., Long, J., & Darrell, T. (2016). Fully Convolutional Networks for Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 1–1. https://doi.org/10.1109/TPAMI.2016.2572683
Shi, B., & Chen, Z. (2021). A layer-wise multi-defect detection system for powder bed monitoring: Lighting strategy for imaging, adaptive segmentation and classification. Materials and Design. https://doi.org/10.1016/j.matdes.2021.110035
Simonyan, K., & Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 1409.1556. https://doi.org/10.48550/arxiv.1409.1556
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S. E., Anguelov, D., et al. (2014). Going Deeper with Convolutions. CoRR, abs/1409.4842. http://arxiv.org/abs/1409.4842
Tan, M., & Le, Q. V. (2021). EfficientNetV2: Smaller Models and Faster Training. CoRR, abs/2104.00298. https://arxiv.org/abs/2104.00298
Tian, C., Xu, Y., Zuo, W., Lin, C.-W., & Zhang, D. (2021). Asymmetric CNN for image super-resolution. CoRR, abs/2103.13634. https://arxiv.org/abs/2103.13634
Wang, D., Wu, S., Fu, F., Mai, S., Yang, Y., Liu, Y., & Song, C. (2017). Mechanisms and characteristics of spatter generation in SLM processing and its effect on the properties. Materials & Design, 117, 121–130. https://doi.org/10.1016/j.matdes.2016.12.060
Zhang, Y., & Yan, W. (2023). Applications of machine learning in metal powder-bed fusion in-process monitoring and control: Status and challenges. Journal of Intelligent Manufacturing, 34(6), 2557–2580. https://doi.org/10.1007/s10845-022-01972-7
Zhou, B., Cui, Q., Wei, X.-S., & Chen, Z.-M. (2020). BBN: Bilateral-Branch Network With Cumulative Learning for Long-Tailed Visual Recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 9716–9725). https://doi.org/10.1109/CVPR42600.2020.00974
Zhu, B., & Dong, X. (2017). Studies on distortions of metal parts in selective laser melting (Master Degree). Zhejiang University of Technology.
Zhu, X., Zhou, H., Wang, T., Hong, F., Ma, Y., Li, W., et al. (2021). Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 9934–9943). https://doi.org/10.1109/CVPR46437.2021.00981
Zur Jacobsmühlen, J., Kleszczynski, S., Schneider, D., & Witt, G. (2013). High resolution imaging for inspection of Laser Beam Melting systems. In Conference Record - IEEE Instrumentation and Measurement Technology Conference (pp. 707–712). https://doi.org/10.1109/I2MTC.2013.6555507
Zur Jacobsmühlen, J., Kleszczynski, S., Witt, G., & Merhof, D. (2015). Detection of Elevated Regions in Surface Images from Laser Beam Melting Processes. In 41st Annual Conference of the IEEE Industrial Electronics Society (pp. 1270–1275). https://doi.org/10.1109/IECON.2015.7392275
Funding
The authors would like to acknowledge the supports from the grants of Zhong Yang Gao Xiao (No. 2022ZYGXZR060), Natural Science Foundation of Guandong Province (No. 2022A1515011563), Basic and Applied Basic Research Programs of Guanzhou City (No. 202102020680).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Jiang, Z., Zhang, A., Chen, Z. et al. A deep convolutional network combining layerwise images and defect parameter vectors for laser powder bed fusion process anomalies classification. J Intell Manuf 35, 2929–2959 (2024). https://doi.org/10.1007/s10845-023-02183-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-023-02183-4