Incremental Learning for Dermatological Imaging Modality Classification
<p>Examples of images belonging to each dermatological modality.</p> "> Figure 2
<p>Examples of images from each image modality selected for the incremental phase.</p> "> Figure 3
<p>Confusion matrices of the base models tested on Task A.</p> "> Figure 4
<p>Model comparison in terms of backward transfer. Results averaged over 10 iterations (±SD).</p> "> Figure 5
<p>Confusion matrices of Task A test images after the training of Task B with the VGG-16 model.</p> "> Figure 6
<p>Confusion matrices of Task A test images after the training of Task B with the MobileNetV2 model.</p> "> Figure 7
<p>Examples of images from Task A correctly classified after the first training but misclassified after the incremental training.</p> "> Figure 8
<p>Comparison of the different incremental learning strategies in terms of global test accuracy, training time, and RAM. The circles’ diameter is proportional to the required RAM.</p> ">
Abstract
:1. Introduction
- The proposed work presents models able to classify dermatological imaging modalities, allowing a better organization of medical records to improve teledermatological processes.
- It explores different incremental learning strategies to enable the continuous training of the developed classification algorithms, without losing performance on the previously learned concepts, as the images acquired in teledermatological consultations may present different properties over time.
1.1. Related Work
1.1.1. Medical Imaging Modality Classification
1.1.2. Incremental Learning
2. Materials and Methods
2.1. Problem Definition
2.2. Database
2.3. Image Modality Classification
2.4. Incremental Learning of Image Modalities
2.4.1. Incremental Learning Strategies
2.4.2. Incremental Task Training
2.4.3. Incremental Learning Evaluation
3. Results and Discussion
3.1. Image Modality Classification
3.2. Incremental Learning of Image Modalities
4. Conclusions and Future Work
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Cullen, J.K.; Simmons, J.L.; Parsons, P.G.; Boyle, G.M. Topical treatments for skin cancer. Adv. Drug Deliv. Rev. 2020, 153, 54–64. [Google Scholar] [CrossRef] [PubMed]
- Cancer Today. Available online: https://gco.iarc.fr/today/online-analysis-table?v=2020&include_nmsc_other=0 (accessed on 4 July 2021).
- American Telemedicine Association, Practice Guidelines for Dermatology. Available online: https://www.americantelemed.org/wp-content/themes/ata-custom/download.php?id=1559 (accessed on 16 December 2020).
- Lee, J.J.; English, J.C. Teledermatology: A review and update. Am. J. Clin. Dermatol. 2018, 19, 253–260. [Google Scholar] [CrossRef] [PubMed]
- Abedini, M.; Codella, N.C.; Connell, J.H.; Garnavi, R.; Merler, M.; Pankanti, S.; Syeda-Mahmood, T. A generalized framework for medical image classification and recognition. IBM J. Res. Dev. 2015, 59, 1:1–1:18. [Google Scholar] [CrossRef]
- Singh, S.; Ho-Shon, K.; Karimi, S.; Hamey, L. Modality classification and concept detection in medical images using deep transfer learning. In Proceedings of the 2018 International Conference on Image and Vision Computing New Zealand (IVCNZ), Auckland, New Zealand, 19–21 November 2018. [Google Scholar]
- Öztürk, Ş. Class-driven content-based medical image retrieval using hash codes of deep features. Biomed. Signal Process. Control 2021, 68, 102601. [Google Scholar] [CrossRef]
- Dimitrovski, I.; Kocev, D.; Kitanovski, I.; Loskovska, S.; Džeroski, S. Improved medical image modality classification using a combination of visual and textual features. Comput. Med. Imaging Graph. 2015, 39, 14–26. [Google Scholar] [CrossRef]
- Yu, Y.; Lin, H.; Yu, Q.; Meng, J.; Zhao, Z.; Li, Y.; Zuo, L. Modality classification for medical images using multiple deep convolutional neural networks. J. Comput. Inf. Syst. 2015, 11, 5403–5413. [Google Scholar]
- Direção-Geral da Saúde, Telerrastreio Dermatológico. Available online: https://www.dgs.pt/directrizes-da-dgs/normas-e-circulares-normativas/norma-n-0052014-de-08042014-pdf.aspx (accessed on 28 June 2021).
- Kalpathy-Cramer, J.; Hersh, W. Automatic image modality based classification and annotation to improve medical image retrieval. In Proceedings of the 12th World Congress on Health (Medical) Informatics, Brisbane, Australia, 20–24 August 2007; p. 1334. [Google Scholar]
- Lesort, T.; Lomonaco, V.; Stoian, A.; Maltoni, D.; Filliat, D.; Díaz-Rodríguez, N. Continual learning for robotics: Definition, framework, learning strategies. Oppor. Chall. Inf. Fusion 2020, 58, 52–68. [Google Scholar] [CrossRef] [Green Version]
- Gepperth, A.; Hammer, B. Incremental learning algorithms and applications. In Proceedings of the European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium, 27–29 April 2016. [Google Scholar]
- McCloskey, M.; Cohen, N.J. Catastrophic interference in connectionist networks: The sequential learning problem. Psychol. Learn. Motiv. 1989, 24, 109–165. [Google Scholar]
- Parisi, G.I.; Kemker, R.; Part, J.L.; Kanan, C.; Wermter, S. Continual lifelong learning with neural networks: A review. Neural Netw. 2019, 113, 54–71. [Google Scholar] [CrossRef]
- Wu, H.; He, L. Combining visual and textual features for medical image modality classification with lp- norm multiple kernel learning. Neurocomputing 2015, 147, 387–394. [Google Scholar] [CrossRef]
- Arias, J.; Martinez-Gomez, J.; Gamez, J.A.; de Herrera, A.G.S.; Müller, H. Medical image modality classification using discrete Bayesian networks. Comput. Vis. Image Underst. 2016, 151, 61–71. [Google Scholar] [CrossRef] [Green Version]
- Kitanovski, I.; Dimitrovski, I.; Loskovska, S. FCSE at Medical Tasks of ImageCLEF 2013. In Proceedings of the CLEF (Working Notes), Valencia, Spain, 23–26 September 2013. [Google Scholar]
- Pelka, O.; Friedrich, C.M. Modality prediction of biomedical literature images using multimodal feature representation. GMS Med. Inform. Biom. Und Epidemiol. 2016, 12, Doc04. [Google Scholar] [CrossRef]
- Csurka, G.; Clinchant, S.; Jacquet, G. Medical image modality classification and retrieval. In Proceedings of the 2011 9th International Workshop on Content-Based Multimedia Indexing (CBMI), Madrid, Spain, 13–15 June 2011. [Google Scholar]
- Markonis, D.; Eggel, I.; de Herrera, A.G.S.; Müller, H. The medGIFT Group in ImageCLEFmed 2011. In Proceedings of the CLEF (Notebook Papers/Labs/Workshop), Amsterdam, The Netherlands, 19–22 September 2011. [Google Scholar]
- Kumar, A.; Kim, J.; Lyndon, D.; Fulham, M.; Feng, D. An ensemble of fine-tuned convolutional neural networks for medical image classification. IEEE J. Biomed. Health Inform. 2016, 21, 31–40. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Yu, Y.; Lin, H.; Meng, J.; Wei, X.; Guo, H.; Zhao, Z. Deep transfer learning for modality classification of medical images. Information 2017, 8, 91. [Google Scholar] [CrossRef] [Green Version]
- Zhang, J.; Xia, Y.; Wu, Q.; Xie, Y. Classification of medical images and illustrations in the biomedical literature using synergic deep learning. arXiv 2017, arXiv:1706.09092. [Google Scholar]
- Hassan, M.; Ali, S.; Alquhayz, H.; Safdar, K. Developing intelligent medical image modality classification system using deep transfer learning and LDA. Sci. Rep. 2020, 10, 1–14. [Google Scholar]
- Roy, D.; Panda, P.; Roy, K. Tree-CNN: A hierarchical deep convolutional neural network for incremental learning. Neural Netw. 2020, 121, 148–160. [Google Scholar] [CrossRef]
- Rusu, A.A.; Rabinowitz, N.C.; Desjardins, G.; Soyer, H.; Kirkpatrick, J.; Kavukcuoglu, K.; Hadsell, R. Progressive neural networks. arXiv 2016, arXiv:1606.04671. [Google Scholar]
- Aljundi, R.; Chakravarty, P.; Tuytelaars, T. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 3366–3375. [Google Scholar]
- Lomonaco, V.; Maltoni, D. Core50: A new dataset and benchmark for continuous object recognition. In Proceedings of the 1st Annual Conference on Robot Learning, Mountain View, CA, USA, 13–15 November 2017; pp. 17–26. [Google Scholar]
- Mallya, A.; Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7765–7773. [Google Scholar]
- Mallya, A.; Davis, D.; Lazebnik, S. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 67–82. [Google Scholar]
- Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A.A.; Hadsell, R. Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. USA 2017, 114, 3521–3526. [Google Scholar] [CrossRef] [Green Version]
- Zenke, F.; Poole, B.; Ganguli, S. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017; pp. 3987–3995. [Google Scholar]
- Li, Z.; Hoiem, D. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 40, 2935–2947. [Google Scholar] [CrossRef] [Green Version]
- Lopez-Paz, D.; Ranzato, M.A. Gradient episodic memory for continual learning. Adv. Neural Inf. Process. Syst. 2017, 30, 6467–6476. [Google Scholar]
- Chaudhry, A.; Ranzato, M.A.; Rohrbach, M.; Elhoseiny, M. Efficient lifelong learning with a-gem. arXiv 2018, arXiv:1812.00420. [Google Scholar]
- Chaudhry, A.; Rohrbach, M.; Elhoseiny, M.; Ajanthan, T.; Dokania, P.K.; Torr, P.H.; Ranzato, M.A. On tiny episodic memories in continual learning. arXiv 2019, arXiv:1902.10486. [Google Scholar]
- Rebuffi, S.A.; Kolesnikov, A.; Sperl, G.; Lampert, C.H. iCarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2001–2010. [Google Scholar]
- Meng, Q.; Shin’ichi, S. ADINet: Attribute driven incremental network for retinal image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 4033–4042. [Google Scholar]
- Hofmanninger, J.; Perkonigg, M.; Brink, J.A.; Pianykh, O.; Herold, C.; Langs, G. Dynamic memory to alleviate catastrophic forgetting in continuous learning settings. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Lima, Peru, 4–8 October 2020; Springer: Cham, Switzerland, 2020; pp. 359–368. [Google Scholar]
- Ravishankar, H.; Venkataramani, R.; Anamandra, S.; Sudhakar, P.; Annangi, P. Feature transformers: Privacy preserving lifelong learners for medical imaging. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China, 13–17 October 2019; Springer: Cham, Switzerland, 2019; pp. 347–355. [Google Scholar]
- Van Garderen, K.; van der Voort, S.; Incekara, F.; Smits, M.; Klein, S. Towards continuous learning for glioma segmentation with elastic weight consolidation. arXiv 2019, arXiv:1909.11479. [Google Scholar]
- Baweja, C.; Glocker, B.; Kamnitsas, K. Towards continual learning in medical imaging. arXiv 2018, arXiv:1811.02496. [Google Scholar]
- Karani, N.; Chaitanya, K.; Baumgartner, C.; Konukoglu, E. A lifelong learning approach to brain MR segmentation across scanners and protocols. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Granada, Spain, 16–20 September 2018; Springer: Cham, Switzerland, 2018; pp. 476–484. [Google Scholar]
- Lomonaco, V.; Pellegrini, L.; Cossu, A.; Carta, A.; Graffieti, G.; Hayes, T.L.; Maltoni, D. Avalanche: An End-to-End Library for Continual Learning. arXiv 2021, arXiv:2104.00405. [Google Scholar]
Task A | Task B (Incremental) | ||||||
---|---|---|---|---|---|---|---|
Train | Validation | Test | Train | Test | Total | ||
Full-body | 180 | 60 | 60 | 43 | 10 | 353 | |
Anatomic | 803 | 268 | 268 | 190 | 48 | 1577 | |
Macroscopic | 880 | 294 | 294 | 209 | 51 | 1728 | |
Dermoscopic | 661 | 220 | 221 | 156 | 39 | 1297 | |
Clinical reports | 521 | 173 | 173 | 123 | 30 | 1020 | |
Total | 3045 | 1015 | 1016 | 721 | 178 | 5975 |
Model | Modality | Accuracy | Precision | Recall | F1-Score |
---|---|---|---|---|---|
VGG-16 | Full-body | 0.9084 | 0.6486 | 0.8000 | 0.7164 |
Anatomic | 0.9032 | 0.7313 | 0.8082 | ||
Macroscopic | 0.8636 | 0.9694 | 0.9135 | ||
Dermoscopic | 0.9955 | 1.0000 | 0.9977 | ||
Clinical reports | 1.0000 | 1.0000 | 1.0000 | ||
MobileNetV2 | Full-body | 0.8837 | 0.6232 | 0.7167 | 0.6667 |
Anatomic | 0.8071 | 0.7649 | 0.7854 | ||
Macroscopic | 0.8658 | 0.8776 | 0.8716 | ||
Dermoscopic | 0.9865 | 0.9955 | 0.9910 | ||
Clinical reports | 1.0000 | 0.9942 | 0.9971 |
VGG-16 | 0.9084 | 0.8652 | 0.9019 |
MobileNetV2 | 0.8837 | 0.8596 | 0.8801 |
VGG−16 | MobileNetV2 | |||
---|---|---|---|---|
BWT | BWT | |||
Naive | 0.8459 ± 0.0025 | −0.0767 ± 0.0029 | 0.8313 ± 0.0049 | −0.0688 ± 0.0047 |
EWC100 | 0.8500 ± 0.0025 | −0.0718 ± 0.0029 | 0.8337 ± 0.0072 | −0.0659 ± 0.0065 |
EWC50 | 0.8517 ± 0.0039 | −0.0699 ± 0.0046 | 0.8367 ± 0.0065 | −0.0628 ± 0.0065 |
EWC1 | 0.8500 ± 0.0028 | −0.0718 ± 0.0033 | 0.8348 ± 0.0059 | −0.0652 ± 0.0057 |
EWC0.5 | 0.8500 ± 0.0032 | −0.0719 ± 0.0038 | 0.8339 ± 0.0060 | −0.0655 ± 0.0055 |
A−GEM50 | 0.8495 ± 0.0036 | −0.0724 ± 0.0042 | 0.8432 ± 0.0063 | −0.0567 ± 0.0065 |
A−GEM100 | 0.8502 ± 0.0030 | −0.0716 ± 0.0035 | 0.8436 ± 0.0055 | −0.0563 ± 0.0064 |
A−GEM150 | 0.8522 ± 0.0030 | −0.0693 ± 0.0035 | 0.8453 ± 0.0034 | −0.0551 ± 0.0044 |
Replay100 | 0.8602 ± 0.0056 | −0.0600 ± 0.0071 | 0.8516 ± 0.0059 | −0.0448 ± 0.0070 |
Replay250 | 0.8695 ± 0.0043 | −0.0482 ± 0.0050 | 0.8588 ± 0.0051 | −0.0368 ± 0.0040 |
Replay500 | 0.8786 ± 0.0045 | −0.0372 ± 0.0056 | 0.8604 ± 0.0065 | −0.0344 ± 0.0070 |
Strategy | |||||
---|---|---|---|---|---|
VGG-16 | Naive | 0.9084 | 0.8652 | 0.8316 ± 0.0029 | 0.9270 ± 0.0000 |
EWC50 | 0.8385 ± 0.0046 | 0.9270 ± 0.0000 | |||
AGEM150 | 0.8391 ± 0.0035 | 0.9270 ± 0.0000 | |||
Replay500 | 0.8711 ± 0.0056 | 0.9213 ± 0.0038 | |||
MobileNetV2 | Naive | 0.8837 | 0.8596 | 0.8150 ± 0.0047 | 0.9242 ± 0.0100 |
EWC50 | 0.8209 ± 0.0065 | 0.9270 ± 0.0102 | |||
AGEM150 | 0.8287 ± 0.0044 | 0.9404 ± 0.0054 | |||
Replay500 | 0.8494 ± 0.0079 | 0.9236 ± 0.0100 |
Model | Modality | Naive | EWC50 | A-GEM150 | Replay500 |
---|---|---|---|---|---|
F1-Score | F1-Score | F1-Score | F1-Score | ||
VGG-16 | Full-body | 0.5583 | 0.5684 | 0.5665 | 0.6461 |
Anatomic | 0.5730 | 0.6016 | 0.5928 | 0.7123 | |
Macroscopic | 0.8732 | 0.8757 | 0.8778 | 0.8832 | |
Dermoscopic | 0.9868 | 0.9865 | 0.9903 | 0.9957 | |
Clinical reports | 1.0000 | 1.0000 | 1.0000 | 1.0000 | |
MobileNetV2 | Full-body | 0.5348 | 0.5512 | 0.5683 | 0.6177 |
Anatomic | 0.5626 | 0.5810 | 0.6153 | 0.6867 | |
Macroscopic | 0.8503 | 0.8500 | 0.8499 | 0.8507 | |
Dermoscopic | 0.9799 | 0.9810 | 0.9797 | 0.9833 | |
Clinical reports | 0.9968 | 0.9971 | 0.9971 | 0.9971 |
VGG-16 | 0.8802 ± 0.0034 | 0.9224 ± 0.0062 | 0.8865 ± 0.0023 |
MobileNetV2 | 0.8617 ± 0.0087 | 0.9371 ± 0.0047 | 0.8742 ± 0.0079 |
Strategy | Time/Epoch(s) | RAM (MB) | |
---|---|---|---|
VGG-16 | Naive | 59.38 ± 2.66 | 3805.22 ± 1360.07 |
EWC50 | 60.98 ± 0.64 | 4017.37 ± 149.99 | |
AGEM150 | 166.01 ± 3.44 | 6127.38 ± 284.36 | |
Replay500 | 123.54 ± 3.46 | 4441.09 ± 105.13 | |
MobileNetV2 | Naive | 53.04 ± 2.46 | 4437.26 ± 226.91 |
EWC50 | 54.88 ± 2.23 | 4428.37 ± 7.97 | |
AGEM150 | 83.91 ± 2.97 | 6457.83 ± 3.98 | |
Replay500 | 109.56 ± 3.35 | 4554.59 ± 144.08 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Morgado, A.C.; Andrade, C.; Teixeira, L.F.; Vasconcelos, M.J.M. Incremental Learning for Dermatological Imaging Modality Classification. J. Imaging 2021, 7, 180. https://doi.org/10.3390/jimaging7090180
Morgado AC, Andrade C, Teixeira LF, Vasconcelos MJM. Incremental Learning for Dermatological Imaging Modality Classification. Journal of Imaging. 2021; 7(9):180. https://doi.org/10.3390/jimaging7090180
Chicago/Turabian StyleMorgado, Ana C., Catarina Andrade, Luís F. Teixeira, and Maria João M. Vasconcelos. 2021. "Incremental Learning for Dermatological Imaging Modality Classification" Journal of Imaging 7, no. 9: 180. https://doi.org/10.3390/jimaging7090180