Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Quality

1. Addison, P.F.E.E., et al.: A new wave of marine evidence-based management: emerging challenges and solutions to transform monitoring, evaluating, and reporting. ICES J. Mar. Sci. 75(3), 941–952 (2018). https://doi.org/10.1093/icesjms/fsx216

   Article  Google Scholar 

2. Arazo, E., Ortego, D., Albert, P., O’Connor, N.E., McGuinness, K.: Pseudo-labeling and confirmation bias in deep semi-supervised learning. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2020). https://doi.org/10.1109/IJCNN48605.2020.9207304

3. Basile, V., et al.: We need to consider disagreement in evaluation. In: Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pp. 15–21 (2021)

   Google Scholar 

4. Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.A.: Mixmatch: a holistic approach to semi-supervised learning. In: Advances in Neural Information Processing Systems, pp. 5050–5060 (2019)

   Google Scholar 

5. Beyer, L., et al.: Are we done with ImageNet? arXiv preprint arXiv:2006.07159 (2020)

6. Brünger, J., Dippel, S., Koch, R., Veit, C.: ‘Tailception’: using neural networks for assessing tail lesions on pictures of pig carcasses. Animal 13(5), 1030–1036 (2019). https://doi.org/10.1017/S1751731118003038

   Article  Google Scholar 

7. Collier, M., Mustafa, B., Kokiopoulou, E., Jenatton, R., Berent, J.: Correlated input-dependent label noise in large-scale image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1551–1560 (2021)

   Google Scholar 

8. Collins, K.M., Bhatt, U., Weller, A.: Eliciting and learning with soft labels from every annotator. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 10, no. 1 (2022)

   Google Scholar 

9. Cortes, C., Lawrence, N.D.: Inconsistency in conference peer review: revisiting the 2014 neurips experiment. arXiv preprint arXiv:2109.09774 (2021)

10. Davani, A.M., Díaz, M., Prabhakaran, V.: Dealing with disagreements: looking beyond the majority vote in subjective annotations. Trans. Assoc. Comput. Linguist. 10, 92–110 (2022). https://doi.org/10.1162/tacl_a_00449

    Article  Google Scholar 

11. Desmond, M., Duesterwald, E., Brimijoin, K., Brachman, M., Pan, Q.: Semi-automated data labeling. In: NeurIPS 2020 Competition and Demonstration Track, pp. 156–169. PMLR (2021)

    Google Scholar 

12. Desmond, M., et al.: Increasing the speed and accuracy of data labeling through an AI assisted interface. In: 26th International Conference on Intelligent User Interfaces, pp. 392–401. Association for Computing Machinery (2021). https://doi.org/10.1145/3397481.3450698

13. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: ICLR (2021)

    Google Scholar 

14. Gao, Z., et al.: Learning from multiple annotator noisy labels via sample-wise label fusion. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022, Part XXIV. LNCS, vol. 13684, pp. 407–422. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20053-3_24

    Chapter  Google Scholar 

15. Gordon, M.L., Zhou, K., Patel, K., Hashimoto, T., Bernstein, M.S.: The disagreement deconvolution: bringing machine learning performance metrics in line with reality. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14. ACM (2021). https://doi.org/10.1145/3411764.3445423

16. Grossmann, V., Schmarje, L., Koch, R.: Beyond hard labels: investigating data label distributions. In: ICML 2022 Workshop DataPerf: Benchmarking Data for Data-Centric AI (2022)

    Google Scholar 

17. Gu, K., Masotto, X., Bachani, V., Lakshminarayanan, B., Nikodem, J., Yin, D.: An instance-dependent simulation framework for learning with label noise. Mach. Learn. 1–26 (2022)

    Google Scholar 

18. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009 (2022)

    Google Scholar 

19. Hendrycks, D., Mazeika, M., Kadavath, S., Song, D.: Using self-supervised learning can improve model robustness and uncertainty. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

20. Jachimowicz, J.M., Duncan, S., Weber, E.U., Johnson, E.J.: When and why defaults influence decisions: a meta-analysis of default effects. Behavioural Public Policy 3(2), 159–186 (2019)

    Article  Google Scholar 

21. Jung, H., Park, Y., Lease, M.: Predicting next label quality: a time-series model of crowdwork. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 2, no. 1, pp. 87–95 (2014). https://doi.org/10.1609/hcomp.v2i1.13165

22. Kolesnikov, A., et al.: Big transfer (BiT): general visual representation learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) ECCV 2020. LNCS, vol. 12350, pp. 491–507. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58558-7_29

    Chapter  Google Scholar 

23. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, Citeseer (2009)

    Google Scholar 

24. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 60, pp. 1097–1105. Association for Computing Machinery (2012). https://doi.org/10.1145/3065386

25. Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Statist. 22(1), 79–86 (1951). https://doi.org/10.1214/aoms/1177729694

    Article  MathSciNet  Google Scholar 

26. Lee, D.H.: Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges in Representation Learning, ICML, vol. 3, p. 2 (2013)

    Google Scholar 

27. Li, J., Socher, R., Hoi, S.C.H.: DivideMix: learning with noisy labels as semi-supervised learning. In: International Conference on Learning Representations, pp. 1–14 (2020)

    Google Scholar 

28. Li, Y.F., Liang, D.M.: Safe semi-supervised learning: a brief introduction. Front. Comp. Sci. 13(4), 669–676 (2019)

    Article  Google Scholar 

29. Lopresti, D., Nagy, G.: Optimal data partition for semi-automated labeling. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR 2012), pp. 286–289. IEEE (2012)

    Google Scholar 

30. Lukasik, M., Bhojanapalli, S., Menon, A.K., Kumar, S.: Does label smoothing mitigate label noise? In: International Conference on Machine Learning, pp. 6448–6458. PMLR (2020)

    Google Scholar 

31. Lukov, T., Zhao, N., Lee, G.H., Lim, S.N.: Teaching with soft label smoothing for mitigating noisy labels in facial expressions. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022, Part XII. LNCS, vol. 13672, pp. 648–665. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19775-8_38

    Chapter  Google Scholar 

32. Mazeika, M., et al.: How would the viewer feel? Estimating wellbeing from video scenarios. Adv. Neural. Inf. Process. Syst. 35, 18571–18585 (2022)

    Google Scholar 

33. Misra, I., van der Maaten, L., van der Maaten, L.: Self-supervised learning of pretext-invariant representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707–6717 (2020)

    Google Scholar 

34. Motamedi, M., Sakharnykh, N., Kaldewey, T.: A data-centric approach for training deep neural networks with less data. In: NeurIPS 2021 Data-Centric AI workshop (2021)

    Google Scholar 

35. Müller, R., Kornblith, S., Hinton, G.: When does label smoothing help? In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

36. Naeem, A., Farooq, M.S., Khelifi, A., Abid, A.: Malignant melanoma classification using deep learning: datasets, performance measurements, challenges and opportunities. IEEE Access 8, 110575–110597 (2020)

    Article  Google Scholar 

37. Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., Schmidt, L.: Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP, pp. 1–46 (2022)

    Google Scholar 

38. Northcutt, C.G., Athalye, A., Mueller, J.: Pervasive label errors in test sets destabilize machine learning benchmarks. In: 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks (2021)

    Google Scholar 

39. Northcutt, C.G., Jiang, L., Chuang, I.L.: Confident learning: estimating uncertainty in dataset labels. J. Artif. Intell. Res. 70, 1373–1411 (2021). https://doi.org/10.1613/JAIR.1.12125

    Article  MathSciNet  Google Scholar 

40. Ooms, E.A., et al.: Mammography: interobserver variability in breast density assessment. Breast 16(6), 568–576 (2007). https://doi.org/10.1016/j.breast.2007.04.007

41. Papadopoulos, D.P., Weber, E., Torralba, A.: Scaling up instance annotation via label propagation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15364–15373 (2021)

    Google Scholar 

42. Patel, B.N., et al.: Human-machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ Digit. Med. 2(1), 1–10 (2019). https://doi.org/10.1038/s41746-019-0189-7

43. Peterson, J., Battleday, R., Griffiths, T., Russakovsky, O.: Human uncertainty makes classification more robust. In: Proceedings of the IEEE International Conference on Computer Vision 2019-October, pp. 9616–9625 (2019). https://doi.org/10.1109/ICCV.2019.00971

44. Ren, P., et al.: A survey of deep active learning. ACM Comput. Surv. (CSUR) 54(9), 1–40 (2021)

    Article  Google Scholar 

45. Saleh, A., Laradji, I.H., Konovalov, D.A., Bradley, M., Vazquez, D., Sheaves, M.: A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis. Sci. Rep. 10(1), 1–10 (2020). https://doi.org/10.1038/s41598-020-71639-x

    Article  Google Scholar 

46. Schmarje, L., Brünger, J., Santarossa, M., Schröder, S.M., Kiko, R., Koch, R.: Fuzzy overclustering: semi-supervised classification of fuzzy labels with overclustering and inverse cross-entropy. Sensors 21(19), 6661 (2021). https://doi.org/10.3390/s21196661

    Article  Google Scholar 

47. Schmarje, L., et al.: Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation. Adv. Neural. Inf. Process. Syst. 35, 33215–33232 (2022)

    Google Scholar 

48. Schmarje, L., Grossmann, V., Zelenka, C., Koch, R.: Annotating Ambiguous Images: General Annotation Strategy for High-Quality Data with Real-World Biomedical Validation. arXiv preprint arXiv:2306.12189 (2023)

49. Schmarje, L., et al.: A data-centric approach for improving ambiguous labels with combined semi-supervised classification and clustering. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)

    Google Scholar 

50. Schröder, S.M., Kiko, R., Koch, R.: MorphoCluster: efficient annotation of plankton images by clustering. Sensors 20 (2020)

    Google Scholar 

51. Schulz, C., et al.: Analysis of automatic annotation suggestions for hard discourse-level tasks in expert domains. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2761–2772. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/P19-1265

52. Schustek, P., Moreno-Bote, R.: Instance-based generalization for human judgments about uncertainty. PLoS Comput. Biol. 14(6), e1006205 (2018). https://doi.org/10.1371/journal.pcbi.1006205

    Article  Google Scholar 

53. Shah, F.A., Sirts, K., Pfahl, D.: The impact of annotation guidelines and annotated data on extracting app features from app reviews. arXiv preprint arXiv:1810.05187 (2018)

54. Sheng, V.S., Provost, F.: Get another label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers Categories and Subject Descriptors, New York, pp. 614–622 (2008)

    Google Scholar 

55. Singh, A., Nowak, R., Zhu, J.: Unlabeled data: now it helps, now it doesn’t. In: Advances in Neural Information Processing Systems, vol. 21 (2008)

    Google Scholar 

56. Sohn, K., et al.: FixMatch: simplifying semi-supervised learning with consistency and confidence. In: Advances in Neural Information Processing Systems 33 pre-proceedings (NeurIPS 2020) (2020)

    Google Scholar 

57. Tarling, P., Cantor, M., Clapés, A., Escalera, S.: Deep learning with self-supervision and uncertainty regularization to count fish in underwater images. PLoS ONE 17(5), 1–22 (2021)

    Google Scholar 

58. Tifrea, A., Clarysse, J., Yang, F.: Uniform versus uncertainty sampling: when being active is less efficient than staying passive. arXiv preprint arXiv:2212.00772 (2022)

59. Uijlings, J., Mensink, T., Ferrari, V.: The Missing Link: Finding label relations across datasets (2022)

    Google Scholar 

60. Vasudevan, V., Caine, B., Gontijo-Lopes, R., Fridovich-Keil, S., Roelofs, R.: When does dough become a bagel? Analyzing the remaining mistakes on ImageNet. Adv. Neural. Inf. Process. Syst. 35, 6720–6734 (2022)

    Google Scholar 

61. Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., Liu, Y.: Learning with noisy labels revisited: a study using real-world human annotations. In: ICLR, pp. 1–23 (2021)

    Google Scholar 

62. Wei, X., Cong, H., Zhang, Z., Peng, J., Chen, G., Li, J.: Faint Features Tell: Automatic Vertebrae Fracture Screening Assisted by Contrastive Learning (2022)

    Google Scholar 

63. Yun, S., Oh, S.J., Heo, B., Han, D., Choe, J., Chun, S.: Re-labeling imagenet: from single to multi-labels, from global to localized labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2340–2350 (2021)

    Google Scholar 

Download references