Abstract
The science of solving clinical problems by analyzing images generated in clinical practice is known as medical image analysis. The aim is to extract information in an affective and efficient manner for improved clinical diagnosis. The recent advances in the field of biomedical engineering have made medical image analysis one of the top research and development area. One of the reasons for this advancement is the application of machine learning techniques for the analysis of medical images. Deep learning is successfully used as a tool for machine learning, where a neural network is capable of automatically learning features. This is in contrast to those methods where traditionally hand crafted features are used. The selection and calculation of these features is a challenging task. Among deep learning techniques, deep convolutional networks are actively used for the purpose of medical image analysis. This includes application areas such as segmentation, abnormality detection, disease classification, computer aided diagnosis and retrieval. In this study, a comprehensive review of the current state-of-the-art in medical image analysis using deep convolutional networks is presented. The challenges and potential of these techniques are also highlighted.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Greenspan, H., van Ginneken, B., and Summers, R. M., Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35(5):1153–1159, 2016.
Wang, G., A perspective on deep imaging. IEEE Access 4:8914–8924, 2016.
Liu, Y., Cheng, H., Huang, J., Zhang, Y., Tang, X., Tian, J.-W., and Wang, Y., Computer aided diagnosis system for breast cancer based on color doppler flow imaging. J. Med. Syst. 36(6):3975–3982, 2012.
Diao, X.-F., Zhang, X.-Y., Wang, T.-F., Chen, S.-P., Yang, Y., and Zhong, L., Highly sensitive computer aided diagnosis system for breast tumor based on color doppler flow images. J. Med. Syst. 35(5):801–809, 2011.
Wan, J., Wang, D., Hoi, S. C. H., Wu, P., Zhu, J., Zhang, Y., and Li, J.: Deep learning for content-based image retrieval: A comprehensive study. In: Proceedings of the 22nd ACM international conference on Multimedia. ACM, pp. 157–166, 2014
Deng, L., Yu, D., et al., Deep learning: Methods and applications. Foundations and Trends®, in Signal Processing 7(3–4):197–387, 2014.
Shi, S., Wang, Q., Xu, P., and Chu, X.: Benchmarking state-of-the-art deep learning software tools. In: 2016 7th International Conference on Cloud Computing and Big Data (CCBD). IEEE, pp. 99–104, 2016
Janowczyk, A., and Madabhushi, A.: Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases, Journal of pathology informatics 7
Lakhani, P., Gray, D. L., Pett, C. R., Nagy, P., and Shih, G., Hello world deep learning in medical imaging. J. Digit. Imaging 31(3):283–289, 2018.
Heidenreich, A., Desgrandschamps, F., and Terrier, F., Modern approach of diagnosis and management of acute flank pain: Review of all imaging modalities. Eur. Urol. 41(4):351–362, 2002.
Rahman, M. M., Desai, B.C., and Bhattacharya, P., Medical image retrieval with probabilistic multi-class support vector machine classifiers and adaptive similarity fusion. Comput. Med. Imaging Graph. 32(2):95–108, 2008.
Sáez, A., Sánchez-Monedero, J., Gutiérrez, P. A., and Hervás-Martínez, C., Machine learning methods for binary and multiclass classification of melanoma thickness from dermoscopic images. IEEE Trans. Med. Imaging 35(4):1036–1045, 2016.
Miri, M. S., Abràmoff, M. D., Lee, K., Niemeijer, M., Wang, J.-K., Kwon, Y. H., and Garvin, M. K., Multimodal segmentation of optic disc and cup from sd-oct and color fundus photographs using a machine-learning graph-based approach. IEEE Trans. Med. Imaging 34(9):1854–1866, 2015.
Gao, Y., Zhan, Y., and Shen, D., Incremental learning with selective memory (ilsm): Towards fast prostate localization for image guided radiotherapy. IEEE Trans. Med. Imaging 33(2):518–534, 2014.
Tao, Y., Peng, Z., Krishnan, A., and Zhou, X. S., Robust learning-based parsing and annotation of medical radiographs. IEEE Trans. Med. Imaging 30(2):338–350, 2011.
Ahmad, J., Muhammad, K., Lee, M. Y., and Baik, S. W., Endoscopic image classification and retrieval using clustered convolutional features. J. Med. Syst. 41(12):196, 2017.
Ahmad, J., Muhammad, K., and Baik, S. W., Medical image retrieval with compact binary codes generated in frequency domain using highly reactive convolutional features. J. Med. Syst. 42(2):24, 2018.
Jenitta, A., and Ravindran, R. S., Image retrieval based on local mesh vector co-occurrence pattern for medical diagnosis from mri brain images. J. Med. Syst. 41(10):157, 2017.
Zhang, L., and Ji, Q., A bayesian network model for automatic and interactive image segmentation. IEEE Trans. Image Process. 20(9):2582–2593, 2011.
Sharma, M. M.: Brain tumor segmentation techniques: A survey. Brain 4 (4): 220–223
Vishnuvarthanan, G., Rajasekaran, M. P., Subbaraj, P., and Vishnuvarthanan, A., An unsupervised learning method with a clustering approach for tumor identification and tissue segmentation in magnetic resonance brain images. Appl. Soft Comput. 38:190–212, 2016.
Feng, Y., Zhao, H., Li, X., Zhang, X., and Li, H., A multi-scale 3d otsu thresholding algorithm for medical image segmentation. Digital Signal Process. 60:186–199, 2017.
Gupta, D., and Anand, R., A hybrid edge-based segmentation approach for ultrasound medical images. Biomed. Signal Process. Control 31:116–126, 2017.
von Landesberger, T., Basgier, D., and Becker, M., Comparative local quality assessment of 3d medical image segmentations with focus on statistical shape model-based algorithms. IEEE Trans. Vis. Comput. Graph. 22 (12):2537–2549, 2016.
Anwar, S., Yousaf, S., and Majid, M.: Brain timor segmentation on multimodal mri scans using emap algorithm. In: Engineering in medicine and biology soceity (EMBC), International Conference of the IEEE. IEEE, pp. 1-4, 2018
Cabria, I., and Gondra, I., Mri segmentation fusion for brain tumor detection. Information Fusion 36:1–9, 2017.
Soulami, K. B., Saidi, M. N., and Tamtaoui, A.: A cad system for the detection of abnormalities in the mammograms using the metaheuristic algorithm particle swarm optimization (pso). In: Advances in Ubiquitous Networking 2. Springer, pp. 505–517, 2017
Kobayashi, Y., Kobayashi, H., Giles, J. T., Yokoe, I., Hirano, M., Nakajima, Y., and Takei, M., Detection of left ventricular regional dysfunction and myocardial abnormalities using complementary cardiac magnetic resonance imaging in patients with systemic sclerosis without cardiac symptoms: A pilot study. Intern. Med. 55(3): 237–243, 2016.
Mosquera-Lopez, C., Agaian, S., Velez-Hoyos, A., and Thompson, I., Computer-aided prostate cancer diagnosis from digitized histopathology: A review on texture-based systems. IEEE Rev. Biomed. Eng. 8:98–113, 2015.
Ma, H.-Y., Zhou, Z., Wu, S., Wan, Y.-L., and Tsui, P.-H., A computer-aided diagnosis scheme for detection of fatty liver in vivo based on ultrasound kurtosis imaging. J. Med. Syst. 40(1):33, 2016.
Remeseiro, B., Mosquera, A., and Penedo, M. G., Casdes: A computer-aided system to support dry eye diagnosis based on tear film maps. IEEE journal of biomedical and health informatics 20(3):936–943, 2016.
Torrents-Barrena, J., Lazar, P., Jayapathy, R., Rathnam, M., Mohandhas, B., and Puig, D., Complex wavelet algorithm for computer-aided diagnosis of alzheimer’s disease. Electron. Lett. 51(20):1566–1568, 2015.
Saha, M., Mukherjee, R., and Chakraborty, C., Computer-aided diagnosis of breast cancer using cytological images: A systematic review. Tissue Cell 48(5):461–474, 2016.
Salam, A. A., Akram, M. U., Wazir, K., Anwar, S. M., and Majid, M.: Autonomous glaucoma detection from fundus image using cup to disc ratio and hybrid features. In: IEEE International Symposium on Signal processing and information technology (ISSPIT) 2015. IEEE, pp. 370-374, 2015
Salam, A. A., Akram, M. U., Abbas, S., and Anwar, S. M.: Optic disc localization using local vessel based features and support vector machine. In: IEEE 15th International Conference on Bioinformatics and Bioengineering (BIBE), 2015. IEEE, pp. 1–6, 2015
Altaf, T., Anwar, S. M., Gul, N., Majeed, M. N., and Majid, M., Multi-class alzheimer’s disease classification using image and clinical features. Biomed. Signal Process. Control 43:64–74, 2018.
Hwang, K. H., Lee, H., and Choi, D., Medical image retrieval: Past and present. Healthcare informatics research 18(1):3–9, 2012.
Müller, H., Rosset, A., Vallée, J.-P., Terrier, F., and Geissbuhler, A., A reference data set for the evaluation of medical image retrieval systems. Comput. Med. Imaging Graph. 28(6):295–305, 2004.
Müller, H., Michoux, N., Bandon, D., and Geissbuhler, A., A review of content-based image retrieval systems in medical applications—clinical benefits and future directions. Int. J. Med. Inform. 73(1):1–23, 2004.
Mizotin, M., Benois-Pineau, J., Allard, M., and Catheline, G.: Feature-based brain mri retrieval for alzheimer disease diagnosis. In: 2012 19th IEEE International Conference on Image Processing (ICIP). IEEE, pp. 1241–1244, 2012
Brahmi, D., and Ziou, D.: Improving cbir systems by integrating semantic features. In: 2004 Proceedings of the 1st Canadian Conference on Computer and robot vision. IEEE, pp. 233-240, 2004
Chang, N.-S., and Fu, K.-S., Query-by-pictorial-example. IEEE Trans. Softw. Eng. SE-6(6):519–524, 1980.
Thakur, M. S., and Singh, M., Content based image retrieval using line edge singular value pattern (lesvp): A review paper. International Journal of Advanced Research in Computer Science and Software Engineering 5(3): 648–652, 2015.
Jiji, G. W., and Raj, P. S. J. D., Content-based image retrieval in dermatology using intelligent technique. IET Image Process. 9(4):306–317, 2014.
Rahman, M. M., Antani, S. K., and Thoma, G. R., A learning-based similarity fusion and filtering approach for biomedical image retrieval using svm classification and relevance feedback. IEEE Trans. Inf. Technol. Biomed. 15(4):640–646, 2011.
Anwar, S. M., Arshad, F., and Majid, M.: Fast wavelet based image characterization for content based medical image retrieval. In: 2017 International Conference on communication, computing and digital systems (C-CODE). IEEE, pp.351-356, 2017
Deng, L., Yu, D., et al., Deep learning: Methods and applications. Foundations and Trends®, in Signal Processing 7(3–4):197–387, 2014.
Premaladha, J., and Ravichandran, K., Novel approaches for diagnosing melanoma skin lesions through supervised and deep learning algorithms. J. Med. Syst. 40(4):96, 2016.
Kharazmi, P., Zheng, J., Lui, H., Wang, Z. J., and Lee, T. K., A computer-aided decision support system for detection and localization of cutaneous vasculature in dermoscopy images via deep feature learning. J. Med. Syst. 42(2):33, 2018.
Wang, S.-H., Phillips, P., Sui, Y., Liu, B., Yang, M., and Cheng, H., Classification of alzheimer’s disease based on eight-layer convolutional neural network with leaky rectified linear unit and max pooling. J. Med. Syst. 42(5):85, 2018.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P., Gradient-based learning applied to document recognition. Proc. IEEE 86(11):2278–2324, 1998.
LeCun, Y., Bengio, Y., and Hinton, G., Deep learning. Nature 521(7553):436, 2015.
Ding, S., Lin, L., Wang, G., and Chao, H., Deep feature learning with relative distance comparison for person re-identification. Pattern Recog. 48(10):2993–3003, 2015.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R., Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15(1):1929–1958, 2014.
Ioffe, S., and Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167
Kooi, T., Litjens, G., van Ginneken, B., Gubern-Mérida, A., Sánchez, C. I., Mann, R., den Heeten, A., and Karssemeijer, N., Large scale deep learning for computer aided detection of mammographic lesions. Med. Image Anal. 35:303–312, 2017. https://doi.org/10.1016/j.media.2016.07.007. http://www.sciencedirect.com/science/article/pii/S1361841516301244.
Perez, L., and Wang, J.: The effectiveness of data augmentation in image classification using deep learning. arXiv:1712.04621
Hussain, S., Anwar, S. M., and Majid, M., Segmentation of glioma tumors in brain using deep convolutional neural network. Neurocomputing 282:248–261, 2018.
Ma, J., Wu, F., Zhu, J., Xu, D., and Kong, D., A pre-trained convolutional neural network based method for thyroid nodule diagnosis. Ultrasonics 73:221–230, 2017.
Sun, W., Tseng, T.-L. B., Zhang, J., and Qian, W., Enhancing deep convolutional neural network scheme for breast cancer diagnosis with unlabeled data. Comput. Med. Imaging Graph. 57:4–9 , 2017.
Pratt, H., Coenen, F., Broadbent, D. M., Harding, S. P., and Zheng, Y., Convolutional neural networks for diabetic retinopathy. Procedia Comput. Sci. 90:200–205, 2016.
Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., and Mougiakakou, S., Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imaging 35 (5):1207–1216, 2016.
van Tulder, G., and de Bruijne, M., Combining generative and discriminative representation learning for lung ct analysis with convolutional restricted boltzmann machines. IEEE Trans. Med. Imaging 35(5):1262–1272, 2016.
Yan, Z., Zhan, Y., Peng, Z., Liao, S., Shinagawa, Y., Zhang, S., Metaxas, D. N., and Zhou, X. S., Multi-instance deep learning: Discover discriminative local anatomies for bodypart recognition. IEEE Trans. Med. Imaging 35(5):1332–1343, 2016.
Sirinukunwattana, K., Raza, S. E. A., Tsang, Y.-W., Snead, D. R., Cree, I. A., and Rajpoot, N. M., Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans. Med. Imaging 35(5):1196–1206, 2016.
Qayyum, A., Anwar, S. M., Awais, M., and Majid, M., Medical image retrieval using deep convolutional neural network. Neurocomputing 266:8–20, 2017.
Chowdhury, M., Bulo, S. R., Moreno, R., Kundu, M. K., and Smedby, Ö.: An efficient radiographic image retrieval system using convolutional neural network. In: 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE, pp. 3134–3139, 2016
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., and Larochelle, H., Brain tumor segmentation with deep neural networks. Med. Image Anal. 35:18–31, 2017.
Pereira, S., Pinto, A., Alves, V., and Silva, C. A., Brain tumor segmentation using convolutional neural networks in mri images. IEEE Trans. Med. Imaging 35(5):1240–1251 , 2016.
Jodoin, A. C., Larochelle, H., Pal, C., and Bengio, Y.: Brain tumor segmentation with deep neural networks
Kamnitsas, K., Ledig, C., Newcombe, V. F., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., and Glocker, B., Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Med. Image Anal. 36:61–78 , 2017.
Tseng, K.-L., Lin, Y.-L., Hsu, W., and Huang, C.-Y.: Joint sequence learning and cross-modality convolution for 3d biomedical segmentation. arXiv:1704.07754
Casamitjana, A., Puch, S., Aduriz, A., Sayrol, E., and Vilaplana, V.: 3d convolutional networks for brain tumor segmentation. Proceedings of the MICCAI Challenge on Multimodal Brain Tumor Image Segmentation (BRATS), pp. 65–68 , 2016
Farooq, A., Anwar, S., Awais, M., and Rehman, S.: A deep cnn based multi-class classification of alzheimer’s disease using mri. In: 2017 IEEE International Conference on Imaging systems and techniques (IST). IEEE, pp. 1–6, 2017
Farooq, A., Anwar, S., Awais, M., and Alnowami, M.: Artificial intelligence based smart diagnosis of alzheimer’s disease and mild cognitive impairment. In: 2017 International Smart cities conference (ISC2). IEEE, pp. 1–4, 2017
Gangeh, M. J., Sørensen, L., Shaker, S. B., Kamel, M. S., De Bruijne, M., and Loog, M.: A texton-based approach for the classification of lung parenchyma in ct images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 595–602, 2010
Sorensen, L., Shaker, S. B., and De Bruijne, M., Quantitative analysis of pulmonary using local binary patterns. IEEE Trans. Med. Imaging 29(2):559–569, 2010.
Anthimopoulos, M., Christodoulidis, S., Christe, A., and Mougiakakou, S.: Classification of interstitial lung disease patterns using local dct features and random forest. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp. 6040–6043, 2014
Chen, M., Shi, X., Zhang, Y., Wu, D., and Guizani, M.: Deep features learning for medical image analysis with convolutional autoencoder neural network. IEEE Transactions on Big Data (1) 1–1. https://doi.org/10.1109/TBDATA.2017.2717439, 2017
Hoo-Chang, S., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., and Summers, R. M., Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35(5):1285, 2016.
Simonyan, K., and Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., and Ronneberger, O.: 3D u-net Learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M. R., Unal, G., and Wells, W. (Eds.) Medical image computing and computer-assisted intervention – MICCAI, Vol. 2016, pp. 424–432. Springer International Publishing, Cham, 2016.
Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N., and Liang, J.: Unet++: A nested u-net architecture for medical image segmentation. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer, pp. 3–11, 2018
Chen, W., Zhang, Y., He, J., Qiao, Y., Chen, Y., Shi, H., and Tang, X.: W-net: Bridged u-net for 2d medical image segmentation. arXiv:1807.04459
Milletari, F., Navab, N., and Ahmadi, S.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571. https://doi.org/10.1109/3DV.2016.79, 2016
LaLonde, R., and Bagci, U.: Capsules for object segmentation. arXiv:1804.04241
Chen, H., Dou, Q., Yu, L., and Heng, P.-A.: Voxresnet: Deep voxelwise residual networks for volumetric brain segmentation. arXiv:1608.05895
Setio, A. A. A., Ciompi, F., Litjens, G., Gerke, P., Jacobs, C., Van Riel, S. J., Wille, M. M. W., Naqibullah, M., Sánchez, C. I., and van Ginneken, B., Pulmonary nodule detection in ct images: False positive reduction using multi-view convolutional networks. IEEE Trans. Med. Imaging 35(5):1160–1169, 2016.
Brosch, T., Tang, L. Y., Yoo, Y., Li, D. K., Traboulsee, A., and Tam, R., Deep 3d convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans. Med. Imaging 35(5):1229–1239, 2016.
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., and Ronneberger, O.: 3d u-net: Learning dense volumetric segmentation from sparse annotation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 424–432, 2016
Ceschin, R., Zahner, A., Reynolds, W., Gaesser, J., Zuccoli, G., Lo, C. W., Gopalakrishnan, V., and Panigrahy, A., A computational framework for the detection of subcortical brain dysmaturation in neonatal mri using 3d convolutional neural networks. NeuroImage 178:183–197, 2018.
Ghafoorian, M., Karssemeijer, N., Heskes, T., Bergkamp, M., Wissink, J., Obels, J., Keizer, K., de Leeuw, F.-E., van Ginneken, B., Marchiori, E., et al., Deep multi-scale location-aware 3d convolutional neural networks for automated detection of lacunes of presumed vascular origin. NeuroImage: Clinical 14:391–399, 2017.
Meijs, M., and Manniesing, R.: Artery and vein segmentation of the cerebral vasculature in 4d ct using a 3d fully convolutional neural network. In: Medical Imaging 2018: Computer-Aided Diagnosis, Vol. 10575, International Society for Optics and Photonics, p. 105751Q, 2018
Kamnitsas, K., Ledig, C., Newcombe, V. F., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., and Glocker, B., Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Med. Image Anal. 36:61–78, 2017.
Seong, S.-B., Pae, C., and Park, H.-J., Geometric convolutional neural network for analyzing surface-based neuroimaging data. Frontiers in Neuroinformatics 12:42, 2018.
Tzeng, E., Hoffman, J., Saenko, K., and Darrell, T.: Adversarial discriminative domain adaptation. In: Computer Vision and Pattern Recognition (CVPR), Vol. 1, p. 4, 2017
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is part of the Topical Collection on Image & Signal Processing
Rights and permissions
About this article
Cite this article
Anwar, S.M., Majid, M., Qayyum, A. et al. Medical Image Analysis using Convolutional Neural Networks: A Review. J Med Syst 42, 226 (2018). https://doi.org/10.1007/s10916-018-1088-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10916-018-1088-1