Abstract
The capsule network (CapsNet) is a promising model in computer vision. It has achieved excellent results on MNIST, but it is still slightly insufficient in real images. Deepening capsule architectures is an effective way to improve performance, but the computational cost hinders their development. To overcome parameter growth and build an efficient architecture, this paper proposes a tensor capsule layer based on multistage separable convolutions and a dense capsule architecture. Multistage separable convolutions can effectively reduce the parameters at the cost of a small performance loss. In the dense capsule architecture, the use of dense connections allows the capsule network to be deeper and easier to train. Combining these two can achieve a novel lightweight dense capsule network. Experiments show that this network uses only 0.05% of the parameters of the CapsNet, but the performance is improved by 8.25% on CIFAR10. In addition, the full tensor capsule method is proposed to solve the problem of capsule network parameters changing with image scale. Experiments prove that this method can keep the parameters unchanged while affecting the performance in a small amount. In order to lighten the fully connected capsule layer, a dynamic routing based on separable matrices is proposed. In addition to applying it to our models, this algorithm also compresses the CapsNet by 41.25% while losing only 0.47% performance on CIFAR10. The parameter utilization index is proposed to quantify the relationship between parameters and performance. To our knowledge, this is the first paper to study lightweight capsule network.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Cheng X, He J, He J, Xu H (2019) Cv-CapsNet: complex-valued capsule network. IEEE Access 7:85492–85499. https://doi.org/10.1109/ACCESS.2019.2924548
Choi J, Seo H, Im S, Kang M (2019) Attention routing between capsules. In: IEEE/CVF international conference on computer vision workshop (ICCVW), pp 1981–1989. https://doi.org/10.1109/ICCVW.2019.00247
Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: IEEE conference on computer vision and pattern recognition, pp 1800–1807. https://doi.org/10.1109/CVPR.2017.195
Deliege A, Cioppa A, Droogenbroeck MV (2018) Hitnet: a neural network with capsules embedded in a hit-or-miss layer, extended with hybrid data augmentation and ghost capsules. ArXiv preprint arXiv:1806.06519
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90
Hinton GE, Sabour S, Frosst N (2018) Matrix capsules with em routing. In: International conference on learning representations. https://openreview.net/forum?id=HJWLfGWRb
Hoogi A, Wilcox BM, Gupta Y, Rubin DL (2019) Self-attention capsule networks for image classification. ArXiv preprint arXiv:1904.12483
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. ArXiv preprint arXiv:1704.04861
Huang G, Liu Z, van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: IEEE conference on computer vision and pattern recognition, pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243
Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. ArXiv preprint arXiv:1502.03167
Kingma DP, Ba J (2014) ADAM: a method for stochastic optimization. ArXiv preprint arXiv:1412.6980
Kosiorek A, Sabour S, Teh YW, Hinton GE (2019) Stacked capsule autoencoders. Adv Neural Inf Process Syst 32:15512–15522
Krizhevsky A (2009) Learning multiple layers of features from tiny images. Tech. rep., University Toronto, Toronto
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105
LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791
Lin M, Chen Q, Yan S (2013) Network in network. ArXiv preprint arXiv:1312.4400
Marchisio A, Bussolino B, Colucci A, Hanif MA, Martina M, Masera G, Shafique M (2019) X-traincaps: accelerated training of capsule nets through lightweight software optimizations. ArXiv preprint arXiv:1905.10142
Mukhometzianov R, Carrillo J (2018) Capsnet comparative performance evaluation for image classification. ArXiv preprint arXiv:1606.07356
Paik I, Kwak T, Kim I (2019) Capsule networks need an improved routing algorithm. ArXiv preprint arXiv:1907.13327
Peer D, Stabinger S, Rodriguez-Sanchez A (2019) Limitations of routing-by-agreement based capsule networks. ArXiv preprint arXiv:1905.08744
Phaye SSR, Sikka A, Dhall A, Bathula DR (2018) Dense and diverse capsule networks: making the capsules learn better. ArXiv preprint arXiv:1805.04001
Rajasegaran J, Jayasundara V, Jayasekara S, Jayasekara H, Seneviratne S, Rodrigo R (2019) Deepcaps: going deeper with capsule networks. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 10725–10733. https://doi.org/10.1109/CVPR.2019.01098
Ren H, Su J, Lu H (2019a) Evaluating generalization ability of convolutional neural networks and capsule networks for image classification via top-2 classification. ArXiv preprint arXiv:1901.10112
Ren Q, Shang S, He L (2019b) Adaptive routing between capsules. ArXiv preprint arXiv:1911.08119
Rosario VMd, Borin E, Breternitz M (2019) The multi-lane capsule network. IEEE Signal Process Lett 26(7):1006–1010. https://doi.org/10.1109/LSP.2019.2915661
Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. Adv Neural Inf Process Syst 30:3856–3866
Srivastava RK, Greff K, Schmidhuber J (2015) Training very deep networks. Adv Neural Inf Process Syst 28:2377–2385
Sun K, Yuan L, Xu H, Wen X (2020) Deep tensor capsule network. IEEE. Access 8:96920–96933. https://doi.org/10.1109/ACCESS.2020.2996282
Wang D, Liu Q (2018) An optimization view on dynamic routing between capsules. In: International conference on learning representations workshop. https://openreview.net/forum?id=HJjtFYJDf
Xi E, Bing S, Jin Y (2017) Capsule network performance on complex data. ArXiv preprint arXiv:1712.03480
Xiang C, Zhang L, Tang Y, Zou W, Xu C (2018) Ms-CapsNet: a novel multi-scale capsule network. IEEE Signal Process Lett 25(12):1850–1854. https://doi.org/10.1109/LSP.2018.2873892
Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. ArXiv preprint arXiv:1708.07747
Yin J, Li S, Zhu H, Luo X (2019) Hyperspectral image classification using CapsNet with well-initialized shallow layers. IEEE Geosci Remote Sens Lett 16(7):1095–1099. https://doi.org/10.1109/LGRS.2019.2891076
Zhang S, Zhao W, Wu X, Zhou Q (2019) Fast dynamic routing based on weighted kernel density estimation. Pract Exp, Concurr Comput. https://doi.org/10.1002/cpe.5281
Zhao Z, Kleinhans A, Sandhu G, Patel I, Unnikrishnan KP (2019) Capsule networks with max–min normalization. ArXiv preprint arXiv:1903.09662
Acknowledgements
This study was supported by the National Natural Science Foundation of China under Grant 61472278 and Major project of Tianjin under Grant 18ZXZNGX00150, and the Key Project of Natural Science Foundation of Tianjin University under Grant 2017ZD13, and the Research Project of Tianjin Municipal Education Commission under Grant 2017KJ255, and Natural Science Foundation of Tianjin under Grant 18JCYBJC84800.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sun, K., Wen, X., Yuan, L. et al. Dense capsule networks with fewer parameters. Soft Comput 25, 6927–6945 (2021). https://doi.org/10.1007/s00500-021-05774-6
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-021-05774-6