Abstract
Few-shot learning aims to use a limited amount of data to complete the model’s training, but complex models often face overfitting. To mitigate overfitting risks, we propose a new meta-learning method termed Meta-Pruning, which diverges from traditional pruning methods by treating pruning as a learnable task and training the model to discern and select beneficial network connections for new tasks. We propose to set the corresponding learning rate for each neural network parameter, use the learning rate as a learnable parameter, and adopt a new parameter updating method that reduces reliance on initial learning rate settings and lowers the manual tuning burden. Empirical evaluations on data sets indicate that our proposed algorithm achieves higher classification accuracy with similar computational complexity than state-of-the-art algorithms. We have also conducted learning rate comparison experiments to prove that the initial learning rate no longer affects our method.
Supported by HeiLongJiang Excellent Young Scientists Foundation (YQJH2023303), Heilongjiang Provincial Natural Science Foundation of China (YQ2023D008) and National Natural Science Foundation of China (62301179).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Jiang, S., et al.: Explainable text classification via attentive and targeted mixing data augmentation, vol. 2023-August, (Macao, China), pp. 5085–5094 (2023). Attention mechanisms; Augmentation methods; Classifieds; Data augmentation; Data Selection; Loss functions; Model training; Selection framework; Text classification; Training sample;
Zhong, T., Chi, Z., Gu, L., Wang, Y., Yu, Y., Tang, J.: Meta-DMoE: adapting to domain shift by meta-distillation from mixture-of-experts. Adv. Neural. Inf. Process. Syst. 35, 22243–22257 (2022)
Chu, Y., Sun, X., Songhao, J., Xie, T., Wang, Z., Shan, W.: Imbalanced few-shot learning based on meta-transfer learning. In: International Conference on Artificial Neural Networks, pp. 357–369. Springer (2023). https://doi.org/10.1007/978-3-031-44198-1_30
Chen, J., Zhang, R., Mao, Y., Xu, J.: ContrastNet: a contrastive learning framework for few-shot text classification. Proc. AAAI Conf. Artif. Intell. 36, 10492–10500 (2022)
Liu, J., et al.: Discrimination-aware network pruning for deep model compression. IEEE Trans. Pattern Anal. Mach. Intell. 44(8), 4035–4051 (2021)
Yu, S., Mazaheri, A., Jannesari, A.: Topology-aware network pruning using multi-stage graph embedding and reinforcement learning. In: International Conference on Machine Learning, pp. 25656–25667, PMLR (2022)
He, Z., Xie, Z., Zhu, Q., Qin, Z.: Sparse double descent: where network pruning aggravates overfitting. In: International Conference on Machine Learning, pp. 8635–8659, PMLR (2022)
Geng, B., Yuan, F., Xu, Q., Shen, Y., Xu, R., Yang, M.: Continual learning for task-oriented dialogue system with iterative network pruning, expanding and masking. Ann. Meeting Assoc. Comput. Linguist., 517–523 (2021). abs/2107.08173
Guo, D., Rush, A.M., Kim, Y.: Parameter-efficient transfer learning with diff pruning. Ann. Meeting Assoc. Comput. Linguist. (2020)
Nilsback, M.-E., Zisserman, A.: Automated flower classification over a large number of classes. In: 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722–729. IEEE (2008)
Vinyals, O., Blundell, C., Lillicrap, T.P., kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning. Adv. Neural Inf. Process. Syst. 29, 3637–3645 (2016)
Lake, B.M., Salakhutdinov, R., Tenenbaum, J.B.: Human-level concept learning through probabilistic program induction. Science 350(6266), 1332–1338 (2015)
Bertinetto, L., Henriques, J.F., Torr, P.H.S., Vedaldi, A.: Meta-learning with differentiable closed-form solvers. Int. Conf. Learn. Represent. (2019). abs/1805.08136
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Chu, Y., Liu, K., Jiang, S., Sun, X., Wang, B., Wang, Z. (2024). Meta-pruning: Learning to Prune on Few-Shot Learning. In: Cao, C., Chen, H., Zhao, L., Arshad, J., Asyhari, T., Wang, Y. (eds) Knowledge Science, Engineering and Management. KSEM 2024. Lecture Notes in Computer Science(), vol 14884. Springer, Singapore. https://doi.org/10.1007/978-981-97-5492-2_6
Download citation
DOI: https://doi.org/10.1007/978-981-97-5492-2_6
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5491-5
Online ISBN: 978-981-97-5492-2
eBook Packages: Computer ScienceComputer Science (R0)