Abstract
Extensive research has been dedicated to studying the substantial challenge posed by non-IID data, which hinders the performance of federated learning (FL), a popular distributed learning paradigm. However, a notable challenge encountered by current FL algorithms in real-world applications is the presence of long-tailed data distributions. This issue often results in inadequate model accuracy when dealing with rare but crucial classes in classification tasks. To cope with this, recent studies have proposed various classifier retraining (CR) approaches. Though effective, they lack a deep understanding of how these methods affect the classifier’s performance. In this work, we first present a systematic study informed by mutual information indicators in FL. Based on this study, we propose a novel and effective CR method for FL scenarios, coined CRFDC, to address non-IID and long-tailed data challenges. Extensive experiments on standard FL benchmarks show that CRFDC can improve the model accuracy by up to 8.16% in generalization and 10.02% in personalization, as compared to the state-of-the-art approaches. The code is available at https://github.com/harrylee999/CRFDC.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Acar, D.A.E., Zhao, Y., Matas, R., Mattina, M., Whatmough, P., Saligrama, V.: Federated learning based on dynamic regularization. In: International Conference on Learning Representations (2021)
Cao, K., Wei, C., Gaidon, A., Arechiga, N., Ma, T.: Learning imbalanced datasets with label-distribution-aware margin loss. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 1567–1578 (2019)
Chen, H.Y., Chao, W.L.: On bridging generic and personalized federated learning for image classification. In: International Conference on Learning Representations (2022)
Chen, Z., et al.: Towards federated long-tailed learning. arXiv preprint arXiv:2206.14988 (2022)
Collins, L., Hassani, H., Mokhtari, A., Shakkottai, S.: Exploiting shared representations for personalized federated learning. In: International Conference on Machine Learning, pp. 2089–2099. PMLR (2021)
Cui, Y., Cao, K., Cao, G., Qiu, M., Wei, T.: Client scheduling and resource management for efficient training in heterogeneous IoT-edge federated learning. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 41(8), 2407–2420 (2021)
Dai, Y., Chen, Z., Li, J., Heinecke, S., Sun, L., Xu, R.: Tackling data heterogeneity in federated learning with class prototypes. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 7314–7322 (2023)
Darlow, L.N., Crowley, E.J., Antoniou, A., Storkey, A.J.: CINIC-10 is not ImageNet or CIFAR-10. arXiv preprint arXiv:1810.03505 (2018)
Fallah, A., Mokhtari, A., Ozdaglar, A.: Personalized federated learning with theoretical guarantees: a model-agnostic meta-learning approach. Adv. Neural. Inf. Process. Syst. 33, 3557–3568 (2020)
Holste, G., et al.: Towards long-tailed, multi-label disease classification from chest X-ray: overview of the CXR-LT challenge. arXiv preprint arXiv:2310.16112 (2023)
Krizhevsky, A., Geoffrey, H.: Learning multiple layers of features from tiny images. Citeseer (2009)
Li, Q., Diao, Y., Chen, Q., He, B.: Federated learning on non-IID data silos: an experimental study. In: 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 965–978. IEEE (2022)
Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. Proc. Mach. Learn. Syst. 2, 429–450 (2020)
Luo, J., Wu, S.: Adapt to adaptation: learning personalization for cross-silo federated learning. In: IJCAI: Proceedings of the Conference, vol. 2022, p. 2166. NIH Public Access (2022)
Luo, M., Chen, F., Hu, D., Zhang, Y., Liang, J., Feng, J.: No fear of heterogeneity: classifier calibration for federated learning with non-IID data. Adv. Neural. Inf. Process. Syst. 34, 5972–5984 (2021)
Makansi, O., Cicek, Ö., Marrakchi, Y., Brox, T.: On exposing the challenging long tail in future prediction of traffic actors. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13147–13157 (2021)
McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)
Oh, J., et al.: Fedbabu: toward enhanced representation for federated image classification. In: International Conference on Learning Representations (2022)
Shang, X., Lu, Y., Huang, G., Wang, H.: Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features. arXiv preprint arXiv:2204.13399 (2022)
Shwartz-Ziv, R., Tishby, N.: Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 (2017)
Wang, L., Xu, S., Wang, X., Zhu, Q.: Addressing class imbalance in federated learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 10165–10173 (2021)
Yang, S., Liu, L., Xu, M.: Free lunch for few-shot learning: distribution calibration. In: International Conference on Learning Representations (2021)
Zhang, J., et al.: Fedala: adaptive local aggregation for personalized federated learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 11237–11244 (2023)
Zhang, Y., Kang, B., Hooi, B., Yan, S., Feng, J.: Deep long-tailed learning: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 45(9), 10795–10816 (2023)
Acknowledgements
This work is sponsored by Chenguang Program No. 20CG47 supported by Shanghai Education Development Foundation and Shanghai Municipal Education Commission, and Grant No. 22ZR1423700 from Shanghai Science and Technology Committee.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, Y., Liu, T., Shen, W., Cui, Y., Lu, W. (2024). Improving Generalization and Personalization in Long-Tailed Federated Learning via Classifier Retraining. In: Carretero, J., Shende, S., Garcia-Blas, J., Brandic, I., Olcoz, K., Schreiber, M. (eds) Euro-Par 2024: Parallel Processing. Euro-Par 2024. Lecture Notes in Computer Science, vol 14802. Springer, Cham. https://doi.org/10.1007/978-3-031-69766-1_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-69766-1_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-69765-4
Online ISBN: 978-3-031-69766-1
eBook Packages: Computer ScienceComputer Science (R0)