Abstract
High-quality data is crucial for the success of machine learning, but labeling large datasets is often a time-consuming and costly process. While semi-supervised learning can help mitigate the need for labeled data, label quality remains an open issue due to ambiguity and disagreement among annotators. Thus, we use proposal-guided annotations as one option which leads to more consistency between annotators. However, proposing a label increases the probability of the annotators deciding in favor of this specific label. This introduces a bias which we can simulate and remove. We propose a new method CleverLabel for Cost-effective LabEling using Validated proposal-guidEd annotations and Repaired LABELs. CleverLabel can reduce labeling costs by up to 30.0%, while achieving a relative improvement in Kullback-Leibler divergence of up to 29.8% compared to the previous state-of-the-art on a multi-domain real-world image classification benchmark. CleverLabel offers a novel solution to the challenge of efficiently labeling large datasets while also improving the label quality.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Addison, P.F.E.E., et al.: A new wave of marine evidence-based management: emerging challenges and solutions to transform monitoring, evaluating, and reporting. ICES J. Mar. Sci. 75(3), 941–952 (2018). https://doi.org/10.1093/icesjms/fsx216
Arazo, E., Ortego, D., Albert, P., O’Connor, N.E., McGuinness, K.: Pseudo-labeling and confirmation bias in deep semi-supervised learning. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2020). https://doi.org/10.1109/IJCNN48605.2020.9207304
Basile, V., et al.: We need to consider disagreement in evaluation. In: Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pp. 15–21 (2021)
Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.A.: Mixmatch: a holistic approach to semi-supervised learning. In: Advances in Neural Information Processing Systems, pp. 5050–5060 (2019)
Beyer, L., et al.: Are we done with ImageNet? arXiv preprint arXiv:2006.07159 (2020)
Brünger, J., Dippel, S., Koch, R., Veit, C.: ‘Tailception’: using neural networks for assessing tail lesions on pictures of pig carcasses. Animal 13(5), 1030–1036 (2019). https://doi.org/10.1017/S1751731118003038
Collier, M., Mustafa, B., Kokiopoulou, E., Jenatton, R., Berent, J.: Correlated input-dependent label noise in large-scale image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1551–1560 (2021)
Collins, K.M., Bhatt, U., Weller, A.: Eliciting and learning with soft labels from every annotator. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 10, no. 1 (2022)
Cortes, C., Lawrence, N.D.: Inconsistency in conference peer review: revisiting the 2014 neurips experiment. arXiv preprint arXiv:2109.09774 (2021)
Davani, A.M., Díaz, M., Prabhakaran, V.: Dealing with disagreements: looking beyond the majority vote in subjective annotations. Trans. Assoc. Comput. Linguist. 10, 92–110 (2022). https://doi.org/10.1162/tacl_a_00449
Desmond, M., Duesterwald, E., Brimijoin, K., Brachman, M., Pan, Q.: Semi-automated data labeling. In: NeurIPS 2020 Competition and Demonstration Track, pp. 156–169. PMLR (2021)
Desmond, M., et al.: Increasing the speed and accuracy of data labeling through an AI assisted interface. In: 26th International Conference on Intelligent User Interfaces, pp. 392–401. Association for Computing Machinery (2021). https://doi.org/10.1145/3397481.3450698
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: ICLR (2021)
Gao, Z., et al.: Learning from multiple annotator noisy labels via sample-wise label fusion. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022, Part XXIV. LNCS, vol. 13684, pp. 407–422. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20053-3_24
Gordon, M.L., Zhou, K., Patel, K., Hashimoto, T., Bernstein, M.S.: The disagreement deconvolution: bringing machine learning performance metrics in line with reality. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14. ACM (2021). https://doi.org/10.1145/3411764.3445423
Grossmann, V., Schmarje, L., Koch, R.: Beyond hard labels: investigating data label distributions. In: ICML 2022 Workshop DataPerf: Benchmarking Data for Data-Centric AI (2022)
Gu, K., Masotto, X., Bachani, V., Lakshminarayanan, B., Nikodem, J., Yin, D.: An instance-dependent simulation framework for learning with label noise. Mach. Learn. 1–26 (2022)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009 (2022)
Hendrycks, D., Mazeika, M., Kadavath, S., Song, D.: Using self-supervised learning can improve model robustness and uncertainty. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Jachimowicz, J.M., Duncan, S., Weber, E.U., Johnson, E.J.: When and why defaults influence decisions: a meta-analysis of default effects. Behavioural Public Policy 3(2), 159–186 (2019)
Jung, H., Park, Y., Lease, M.: Predicting next label quality: a time-series model of crowdwork. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 2, no. 1, pp. 87–95 (2014). https://doi.org/10.1609/hcomp.v2i1.13165
Kolesnikov, A., et al.: Big transfer (BiT): general visual representation learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) ECCV 2020. LNCS, vol. 12350, pp. 491–507. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58558-7_29
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, Citeseer (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 60, pp. 1097–1105. Association for Computing Machinery (2012). https://doi.org/10.1145/3065386
Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Statist. 22(1), 79–86 (1951). https://doi.org/10.1214/aoms/1177729694
Lee, D.H.: Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges in Representation Learning, ICML, vol. 3, p. 2 (2013)
Li, J., Socher, R., Hoi, S.C.H.: DivideMix: learning with noisy labels as semi-supervised learning. In: International Conference on Learning Representations, pp. 1–14 (2020)
Li, Y.F., Liang, D.M.: Safe semi-supervised learning: a brief introduction. Front. Comp. Sci. 13(4), 669–676 (2019)
Lopresti, D., Nagy, G.: Optimal data partition for semi-automated labeling. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR 2012), pp. 286–289. IEEE (2012)
Lukasik, M., Bhojanapalli, S., Menon, A.K., Kumar, S.: Does label smoothing mitigate label noise? In: International Conference on Machine Learning, pp. 6448–6458. PMLR (2020)
Lukov, T., Zhao, N., Lee, G.H., Lim, S.N.: Teaching with soft label smoothing for mitigating noisy labels in facial expressions. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022, Part XII. LNCS, vol. 13672, pp. 648–665. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19775-8_38
Mazeika, M., et al.: How would the viewer feel? Estimating wellbeing from video scenarios. Adv. Neural. Inf. Process. Syst. 35, 18571–18585 (2022)
Misra, I., van der Maaten, L., van der Maaten, L.: Self-supervised learning of pretext-invariant representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707–6717 (2020)
Motamedi, M., Sakharnykh, N., Kaldewey, T.: A data-centric approach for training deep neural networks with less data. In: NeurIPS 2021 Data-Centric AI workshop (2021)
Müller, R., Kornblith, S., Hinton, G.: When does label smoothing help? In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Naeem, A., Farooq, M.S., Khelifi, A., Abid, A.: Malignant melanoma classification using deep learning: datasets, performance measurements, challenges and opportunities. IEEE Access 8, 110575–110597 (2020)
Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., Schmidt, L.: Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP, pp. 1–46 (2022)
Northcutt, C.G., Athalye, A., Mueller, J.: Pervasive label errors in test sets destabilize machine learning benchmarks. In: 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks (2021)
Northcutt, C.G., Jiang, L., Chuang, I.L.: Confident learning: estimating uncertainty in dataset labels. J. Artif. Intell. Res. 70, 1373–1411 (2021). https://doi.org/10.1613/JAIR.1.12125
Ooms, E.A., et al.: Mammography: interobserver variability in breast density assessment. Breast 16(6), 568–576 (2007). https://doi.org/10.1016/j.breast.2007.04.007
Papadopoulos, D.P., Weber, E., Torralba, A.: Scaling up instance annotation via label propagation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15364–15373 (2021)
Patel, B.N., et al.: Human-machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ Digit. Med. 2(1), 1–10 (2019). https://doi.org/10.1038/s41746-019-0189-7
Peterson, J., Battleday, R., Griffiths, T., Russakovsky, O.: Human uncertainty makes classification more robust. In: Proceedings of the IEEE International Conference on Computer Vision 2019-October, pp. 9616–9625 (2019). https://doi.org/10.1109/ICCV.2019.00971
Ren, P., et al.: A survey of deep active learning. ACM Comput. Surv. (CSUR) 54(9), 1–40 (2021)
Saleh, A., Laradji, I.H., Konovalov, D.A., Bradley, M., Vazquez, D., Sheaves, M.: A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis. Sci. Rep. 10(1), 1–10 (2020). https://doi.org/10.1038/s41598-020-71639-x
Schmarje, L., Brünger, J., Santarossa, M., Schröder, S.M., Kiko, R., Koch, R.: Fuzzy overclustering: semi-supervised classification of fuzzy labels with overclustering and inverse cross-entropy. Sensors 21(19), 6661 (2021). https://doi.org/10.3390/s21196661
Schmarje, L., et al.: Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation. Adv. Neural. Inf. Process. Syst. 35, 33215–33232 (2022)
Schmarje, L., Grossmann, V., Zelenka, C., Koch, R.: Annotating Ambiguous Images: General Annotation Strategy for High-Quality Data with Real-World Biomedical Validation. arXiv preprint arXiv:2306.12189 (2023)
Schmarje, L., et al.: A data-centric approach for improving ambiguous labels with combined semi-supervised classification and clustering. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)
Schröder, S.M., Kiko, R., Koch, R.: MorphoCluster: efficient annotation of plankton images by clustering. Sensors 20 (2020)
Schulz, C., et al.: Analysis of automatic annotation suggestions for hard discourse-level tasks in expert domains. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2761–2772. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/P19-1265
Schustek, P., Moreno-Bote, R.: Instance-based generalization for human judgments about uncertainty. PLoS Comput. Biol. 14(6), e1006205 (2018). https://doi.org/10.1371/journal.pcbi.1006205
Shah, F.A., Sirts, K., Pfahl, D.: The impact of annotation guidelines and annotated data on extracting app features from app reviews. arXiv preprint arXiv:1810.05187 (2018)
Sheng, V.S., Provost, F.: Get another label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers Categories and Subject Descriptors, New York, pp. 614–622 (2008)
Singh, A., Nowak, R., Zhu, J.: Unlabeled data: now it helps, now it doesn’t. In: Advances in Neural Information Processing Systems, vol. 21 (2008)
Sohn, K., et al.: FixMatch: simplifying semi-supervised learning with consistency and confidence. In: Advances in Neural Information Processing Systems 33 pre-proceedings (NeurIPS 2020) (2020)
Tarling, P., Cantor, M., Clapés, A., Escalera, S.: Deep learning with self-supervision and uncertainty regularization to count fish in underwater images. PLoS ONE 17(5), 1–22 (2021)
Tifrea, A., Clarysse, J., Yang, F.: Uniform versus uncertainty sampling: when being active is less efficient than staying passive. arXiv preprint arXiv:2212.00772 (2022)
Uijlings, J., Mensink, T., Ferrari, V.: The Missing Link: Finding label relations across datasets (2022)
Vasudevan, V., Caine, B., Gontijo-Lopes, R., Fridovich-Keil, S., Roelofs, R.: When does dough become a bagel? Analyzing the remaining mistakes on ImageNet. Adv. Neural. Inf. Process. Syst. 35, 6720–6734 (2022)
Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., Liu, Y.: Learning with noisy labels revisited: a study using real-world human annotations. In: ICLR, pp. 1–23 (2021)
Wei, X., Cong, H., Zhang, Z., Peng, J., Chen, G., Li, J.: Faint Features Tell: Automatic Vertebrae Fracture Screening Assisted by Contrastive Learning (2022)
Yun, S., Oh, S.J., Heo, B., Han, D., Choe, J., Chun, S.: Re-labeling imagenet: from single to multi-labels, from global to localized labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2340–2350 (2021)
Acknowledgments
We acknowledge funding of L. Schmarje by the ARTEMIS project (Grant number 01EC1908E) funded by the Federal Ministry of Education and Research (BMBF, Germany). We further acknowledge the funding of J. Nazarenus by the OP der Zukunft project funded by the Business Development and Technlogy Transfer Corporation of Schleswig Holstein (WTSH, Germany) within the REACT-EU program. We also acknowledge the funding of M. Santarossa by the KI-SIGS project (grant number FKZ 01MK20012E) and the funding of V. Grossmann by the Marispace-X project (grant number 68GX21002E), both funded by the Federal Ministry for Economic Affairs and Climate Action (BMWK, Germany).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Schmarje, L. et al. (2024). Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Quality. In: Köthe, U., Rother, C. (eds) Pattern Recognition. DAGM GCPR 2023. Lecture Notes in Computer Science, vol 14264. Springer, Cham. https://doi.org/10.1007/978-3-031-54605-1_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-54605-1_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54604-4
Online ISBN: 978-3-031-54605-1
eBook Packages: Computer ScienceComputer Science (R0)