Abstract
Dropout is an effective strategy for the regularization of deep neural networks. Applying tabu to the units that have been dropped in the recent epoch and retaining them for training ensures diversification in dropout. In this paper, we improve the Tabu Dropout mechanism for training deep neural networks in two ways. Firstly, we propose to use tabu tenure, or the number of epochs a particular unit will not be dropped. Different tabu tenures provide diversification to boost the training of deep neural networks based on the search landscape. Secondly, we propose an adaptive tabu algorithm that automatically selects the tabu tenure based on the training performances through epochs. On several standard benchmark datasets, the experimental results show that the adaptive tabu dropout and tabu tenure dropout diversify and perform significantly better compared to the standard dropout and basic tabu dropout mechanisms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, Y., Yi, Z.: Adaptive sparse dropout: learning the certainty and uncertainty in deep neural networks. Neurocomputing 450, 354–361 (2021)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
FAN, X., Zhang, S., Tanwisuth, K., Qian, X., Zhou, M.: Contextual dropout: an efficient sample-dependent dropout module. In: International Conference on Learning Representations. https://par.nsf.gov/biblio/10273825
Glover, F.: Tabu search-part i. ORSA J. Comput. 1(3), 190–206 (1989)
Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)
Hu, J., Chen, Y., Zhang, L., Yi, Z.: Surrogate dropout: learning optimal drop rate through proxy. Knowl.-Based Syst. 206, 106340 (2020)
İrsoy, O., Alpaydın, E.: Dropout regularization in hierarchical mixture of experts. Neurocomputing 419, 148–156 (2021)
Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 972–981 (2017)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural. Inf. Process. Syst. 25, 1097–1105 (2012)
LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)
Li, H., et al.: Adaptive dropout method based on biological principles. IEEE Trans. Neural Networks Learn. Syst. (2021)
Ma, Z., Sattar, A., Zhou, J., Chen, Q., Su, K.: Dropout with tabu strategy for regularizing deep neural networks. Comput. J. 63(7), 1031–1038 (2020)
Morerio, P., Cavazza, J., Volpi, R., Vidal, R., Murino, V.: Curriculum dropout. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3544–3552 (2017)
Pham, H., Le, Q.V.: Autodropout: Learning dropout patterns to regularize deep networks. arXiv preprint arXiv:2101.01761 1(2), 3 (2021)
Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT press (2018)
Tang, Y., et al.: Beyond dropout: Feature map distortion to regularize deep neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 5964–5971 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hasan, M.T. et al. (2023). Adaptive Tabu Dropout for Regularization of Deep Neural Networks. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Lecture Notes in Computer Science, vol 13623. Springer, Cham. https://doi.org/10.1007/978-3-031-30105-6_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-30105-6_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30104-9
Online ISBN: 978-3-031-30105-6
eBook Packages: Computer ScienceComputer Science (R0)