LeanNet: An Efficient Convolutional Neural Network for Digital Number Recognition in Industrial Products
<p>Flowchart of the whole project. The above side shows the process from acquiring the data set to evaluate the model and finally to deploy the model, the below side shows and compares the structure of VGG-16 and LeanNet in detail. The label number part of the flowchart is a collection of label parts on some industrial electronic equipment parts, from 1 to 8.</p> "> Figure 2
<p>Pruning a channel results in the removal of both the corresponding filter in the previous layer and related convolution kernels in the next layer.</p> "> Figure 3
<p>We use the trainable parameter <math display="inline"><semantics> <mi>γ</mi> </semantics></math> in the batch normalization layer as the scaling factor and each channel corresponds to a scaling factor. After sparse these scaling factors, the channels with small scaling factors are taken as trivial channels and removed. After pruning, we can obtain a slim network.</p> "> Figure 4
<p>Left: Flowchart of different pruning methods, the dotted-line represents iterative pruning. Right: The blue line represents the One-shot Pruning method and the yellow line represents the Iterative Pruning method.</p> "> Figure 5
<p>The number label on the workpiece and its location. (<b>a</b>) The industrial electronic equipment parts and the captured digital number. (<b>b</b>) The Digital number on some industrial electronic equipment parts, from 1 to 8, respectively representing the corresponding mold number of different workpieces.</p> "> Figure 6
<p>Visualization of 8 channels randomly selected in the first convolutional layer of VGG-16 trained on our dataset. Different channels reflect different information about the input data.</p> "> Figure 7
<p>Curves of training loss and test accuracy for different networks.</p> "> Figure 8
<p>Comparison of parameters and computation under different networks.</p> "> Figure 9
<p>Distributions of scaling factors under different convolutional layers.</p> "> Figure 10
<p>The scaling factor of different channels under different convolutional layers changes dynamically with the increase of the epoch of training rounds. The deeper layer is, the faster the channel’s scaling factor of this layer will to be near zero.</p> "> Figure 11
<p>The distribution of scaling factors in four groups of VGG16 under different sparsity regularization (controlled by the parameter <math display="inline"><semantics> <mi>λ</mi> </semantics></math>). As <math display="inline"><semantics> <mi>λ</mi> </semantics></math> increases, the scaling factor becomes sparse.</p> ">
Abstract
:1. Introduction
- The proposed LeanNet can effectively reduce the parameters and computational complexity, and the accuracy on our dataset is close to the original accuracy by retraining the network, so it can be deployed on some devices with limited computing resources.
- Experimental results show that LeanNet achieves better performance on reductions in model size and computation compared to some lightweight networks like MobileNet and SqueezeNet.
2. Related Work
3. Pruning Network
3.1. Determining the Sparse Level
3.2. Determining Which Channel Should Be Pruned
- For the fully connected network, the mean value and standard variance of the output data of neurons in the upper layer are calculated first.
- Normalizing them to obtain the following Formula (1).
- Finally, the data obtained through the above normalization processing are reconstructed to get the following Formula (2):
3.3. Determining the Pruning Ratio and Method
4. Experiments
4.1. Dataset
4.2. Network Models
4.3. Implementation Details
4.4. Results
5. Discussion
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Chowdhary, C.L.; Patel, P.V.; Kathrotia, K.J.; Attique, M.; Perumal, K.; Ijaz, M.F. Analytical study of hybrid techniques for image encryption and decryption. Sensors 2020, 20, 5162. [Google Scholar] [CrossRef] [PubMed]
- Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 580–587. [Google Scholar]
- Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [Google Scholar]
- Le, N.Q.K. Fertility-GRU: Identifying fertility-related proteins by incorporating deep-gated recurrent units and original position-specific scoring matrix profiles. J. Proteome Res. 2019, 18, 3503–3511. [Google Scholar] [CrossRef] [PubMed]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv 2017, arXiv:1704.04861. [Google Scholar]
- Iandola, F.N.; Han, S.; Moskewicz, M.W.; Ashraf, K.; Dally, W.J.; Keutzer, K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size. arXiv 2016, arXiv:1602.07360. [Google Scholar]
- Peng, Y.; Ganesh, A.; Wright, J.; Xu, W.; Ma, Y. RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 34, 2233–2246. [Google Scholar] [CrossRef] [PubMed]
- Wright, J.; Ganesh, A.; Rao, S.; Peng, Y.; Ma, Y. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. In Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 7–10 December 2009; pp. 2080–2088. [Google Scholar]
- Jaderberg, M.; Vedaldi, A.; Zisserman, A. Speeding up convolutional neural networks with low rank expansions. arXiv 2014, arXiv:1405.3866. [Google Scholar]
- Denton, E.L.; Zaremba, W.; Bruna, J.; LeCun, Y.; Fergus, R. Exploiting linear structure within convolutional networks for efficient evaluation. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 1269–1277. [Google Scholar]
- Tai, C.; Xiao, T.; Zhang, Y.; Wang, X. Convolutional neural networks with low-rank regularization. arXiv 2015, arXiv:1511.06067. [Google Scholar]
- LeCun, Y.; Denker, J.S.; Solla, S.A. Optimal brain damage. In Proceedings of the Advances in Neural Information Processing Systems, Denver, CO, USA, 26–29 November 1990; pp. 598–605. [Google Scholar]
- Hassibi, B.; Stork, D.G. Second order derivatives for network pruning: Optimal brain surgeon. In Proceedings of the Advances in Neural Information Processing Systems, Denver, CO, USA, 27–30 November 1993; pp. 164–171. [Google Scholar]
- Han, S.; Pool, J.; Tran, J.; Dally, W. Learning both weights and connections for efficient neural network. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 7–12 December 2015; pp. 1135–1143. [Google Scholar]
- Han, S.; Mao, H.; Dally, W.J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv 2015, arXiv:1510.00149. [Google Scholar]
- Zhou, H.; Alvarez, J.M.; Porikli, F. Less is more: Towards compact cnns. In Proceedings of the European Conference on Computer Vision, Amsterdam, the Netherlands, 8–10 October 2016; pp. 662–677. [Google Scholar]
- Li, H.; Kadav, A.; Durdanovic, I.; Samet, H.; Graf, H.P. Pruning filters for efficient convnets. arXiv 2016, arXiv:1608.08710. [Google Scholar]
- Wen, W.; Wu, C.; Wang, Y.; Chen, Y.; Li, H. Learning structured sparsity in deep neural networks. In Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016; pp. 2074–2082. [Google Scholar]
- Liu, Z.; Li, J.; Shen, Z.; Huang, G.; Yan, S.; Zhang, C. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2736–2744. [Google Scholar]
- Wu, J.; Leng, C.; Wang, Y.; Hu, Q.; Cheng, J. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 4820–4828. [Google Scholar]
- Tung, F.; Mori, G. Clip-q: Deep network compression learning by in-parallel pruning-quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7873–7882. [Google Scholar]
- Courbariaux, M.; Hubara, I.; Soudry, D.; El-Yaniv, R.; Bengio, Y. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or −1. arXiv 2016, arXiv:1602.02830. [Google Scholar]
- Hinton, G.; Vinyals, O.; Dean, J. Distilling the knowledge in a neural network. arXiv 2015, arXiv:1503.02531. [Google Scholar]
- Romero, A.; Ballas, N.; Kahou, S.E.; Chassang, A.; Gatta, C.; Bengio, Y. Fitnets: Hints for thin deep nets. arXiv 2014, arXiv:1412.6550. [Google Scholar]
- Prakosa, S.W.; Leu, J.S.; Chen, Z.H. Improving the accuracy of pruned network using knowledge distillation. Pattern Anal. Appl. 2020, 24, 819–830. [Google Scholar] [CrossRef]
- Zhang, X.; Zhou, X.; Lin, M.; Sun, J. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 6848–6856. [Google Scholar]
- Yuan, C.; Wu, Y.; Qin, X.; Qiao, S.; Pan, Y.; Huang, P.; Liu, D.; Han, N. An effective image classification method for shallow densely connected convolution networks through squeezing and splitting techniques. Appl. Intell. 2019, 49, 3570–3586. [Google Scholar] [CrossRef]
- Ioffe, S.; Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv 2015, arXiv:1502.03167. [Google Scholar]
- Yang, Y.; Qin, N.; Huang, D.; Shah, A. Label number Recognition Based on Convolutional Neural Networks in industrial products. IFAC-PapersOnLine 2019, 52, 207–212. [Google Scholar] [CrossRef]
- Lin, M.; Chen, Q.; Yan, S. Network in network. arXiv 2013, arXiv:1312.4400. [Google Scholar]
- Chollet, F. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1251–1258. [Google Scholar]
Dataset | Model | Accuracy (%) | FLOP (Giga) | Parameters (M) | Model Size (MB) | Inference Time (s) |
---|---|---|---|---|---|---|
LN | VGG-16 | 98.37 | 7.02 | 14.72 | 117.2 | 7.56/633.85 |
LeanNet | 97.17 | 1.59 | 0.55 | 4.5 | 5.36/120.58 | |
SqueezeNet | 94.37 | 3.96 | 0.73 | 5.9 | 6.19/453.97 | |
MobileNet | 99.05 | 0.29 | 3.22 | 25.9 | 5.15/112.01 | |
CIFAR-10 | VGG-16 | 93.43 | 0.31 | 14.72 | 117.2 | 4.75/387.25 |
LeanNet | 91.95 | 0.05 | 0.55 | 4.5 | 2.95/65.27 | |
SqueezeNet | 92.97 | 0.13 | 0.73 | 5.9 | 3.24/238.59 | |
MobileNet | 91.85 | 0.01 | 3.22 | 25.9 | 2.79/63.34 | |
MNIST | VGG-16 | 99.65 | 0.21 | 14.72 | 117.2 | 3.07/264.35 |
LeanNet | 99.53 | 0.04 | 0.55 | 4.5 | 2.47/55.27 | |
SqueezeNet | 99.57 | 0.09 | 0.73 | 5.9 | 2.26/165.65 | |
MobileNet | 99.45 | 0.01 | 3.22 | 25.9 | 2.68/61.37 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Qin, N.; Liu, L.; Huang, D.; Wu, B.; Zhang, Z. LeanNet: An Efficient Convolutional Neural Network for Digital Number Recognition in Industrial Products. Sensors 2021, 21, 3620. https://doi.org/10.3390/s21113620
Qin N, Liu L, Huang D, Wu B, Zhang Z. LeanNet: An Efficient Convolutional Neural Network for Digital Number Recognition in Industrial Products. Sensors. 2021; 21(11):3620. https://doi.org/10.3390/s21113620
Chicago/Turabian StyleQin, Na, Longkai Liu, Deqing Huang, Bi Wu, and Zonghong Zhang. 2021. "LeanNet: An Efficient Convolutional Neural Network for Digital Number Recognition in Industrial Products" Sensors 21, no. 11: 3620. https://doi.org/10.3390/s21113620