[go: up one dir, main page]
Next Article in Journal
Efficient Memory-Enhanced Transformer for Long-Document Summarization in Low-Resource Regimes
Next Article in Special Issue
Resource Allocation and Data Offloading Strategy for Edge-Computing-Assisted Intelligent Telemedicine System
Previous Article in Journal
An Asynchronous Collision-Tolerant ACRDA Scheme Based on Satellite-Selection Collaboration-Beamforming for LEO Satellite IoT Networks
Previous Article in Special Issue
A Blockchain-Based End-to-End Data Protection Model for Personal Health Records Sharing: A Fully Homomorphic Encryption Approach
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 23
Issue 7
10.3390/s23073548
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

An IoMT-Based Melanoma Lesion Segmentation Using Conditional Generative Adversarial Networks

by
Zeeshan Ali
1,
Sheneela Naz
2,
Hira Zaffar
3,
Jaeun Choi
4,* and
Yongsung Kim
5,*
1
R & D Setups, National University of Computer and Emerging Sciences, Islamabad 44000, Pakistan
2
Department of Computer Science, COMSATS University Islamabad, Islamabad 45550, Pakistan
3
Department of Computer Science, Air University, Aerospace and Aviation Kamra Campus, Islamabad 44000, Pakistan
4
College of Business, Kwangwoon University, Seoul 01897, Republic of Korea
5
Department of Technology Education, Chungnam National University, Daejeon 34134, Republic of Korea
*
Authors to whom correspondence should be addressed.
Sensors 2023, 23(7), 3548; https://doi.org/10.3390/s23073548
Submission received: 20 January 2023 / Revised: 3 February 2023 / Accepted: 25 March 2023 / Published: 28 March 2023
(This article belongs to the Special Issue Internet of Health Things)
Browse Figures
Figure 1
<p>A Pictorial Overview of Proposed Methodology.</p> "> Figure 2
<p>Skin Refinement on ISBI2016 Dataset: Column 1 shows Original Images, Column 2 shows the images after closing morphological operation, and Column 3 shows results after a sharpening kernel is applied.</p> "> Figure 3
<p>Training of Conditional Generative Adversarial Networks for melanoma lesion segmentation.</p> "> Figure 4
<p>Sample images from the datasets: Horizontally Row 1 shows some sample images from the ISCI2016 dataset. Row 2 shows some sample images from the DermQuest dataset and Row 3 shows some sample images from the DermIS dataset.</p> "> Figure 5
<p>The augmented samples of the images from the DermQuest dataset.</p> "> Figure 6
<p>Melanoma segmentation results of DermIS dataset from the skin with the respective masks and contour images.</p> "> Figure 7
<p>Melanoma segmentation results of DermQuest dataset from the skin with the respective masks and contour images.</p> "> Figure 8
<p>Sample melanoma segmentation results of datasets in form of contour images.</p> "> Figure 9
<p>Loss of generator (<b>a</b>) and discriminator (<b>b</b>) of CGANs model.</p> ">
Versions Notes

Abstract

:
Currently, Internet of medical things-based technologies provide a foundation for remote data collection and medical assistance for various diseases. Along with developments in computer vision, the application of Artificial Intelligence and Deep Learning in IOMT devices aids in the design of effective CAD systems for various diseases such as melanoma cancer even in the absence of experts. However, accurate segmentation of melanoma skin lesions from images by CAD systems is necessary to carry out an effective diagnosis. Nevertheless, the visual similarity between normal and melanoma lesions is very high, which leads to less accuracy of various traditional, parametric, and deep learning-based methods. Hence, as a solution to the challenge of accurate segmentation, we propose an advanced generative deep learning model called the Conditional Generative Adversarial Network (cGAN) for lesion segmentation. In the suggested technique, the generation of segmented images is conditional on dermoscopic images of skin lesions to generate accurate segmentation. We assessed the proposed model using three distinct datasets including DermQuest, DermIS, and ISCI2016, and attained optimal segmentation results of 99%, 97%, and 95% performance accuracy, respectively.

1. Introduction

The Internet of Things (IoT) has expanded into many research domains such as smart cities, vehicular communication, cloud computing, smart agriculture, and healthcare systems [1,2,3]. However, the research interest has mainly been focused on the healthcare systems of IoMT-based medical systems. IoMT-based medical systems are based on sensors or smart medical devices for data collection, and they also perform cloud-based data processing [4]. These systems are used for virtual patient care or long-term illness monitoring as well as patient medication tracking. By linking patients to respective physicians as well as allowing healthcare data to be exchanged over a secure site that could be anywhere, the Internet of Things substantially lowered the number of needless hospitalizations [5]. Using IoMT technology, patients and diagnostic laboratories may access data online from anywhere and at any time [6]. IoMT-assisted techniques bring significant advances to a range of medical fields that need rigorous study and supervision, including early diagnoses, such as diabetes [5], heart disease [7], infectious diseases [8], as well cancer diseases [9]. These diseases are detected early on and tracked utilizing IoT-based medical technologies. Due to these advantages, IoMT-based technologies play a remarkable role in the medical sector by designing effective Computer-aided diagnostic (CAD) systems. Similarly, in dermatology, it is critical to detect skin lesions in the very beginning phases to prevent subsequent consequences including melanoma. Annually, about nine thousand people die due to skin cancer. Of all the kinds of skin cancer, about 2% are melanoma, which causes 75% of deaths from skin cancer [10]. To diagnose the medicinal conditions of melanoma lesions, dermatologists visually analyze the symmetry, irregularities in edges, color differences, and diameter of more than 6 mm. All of these are indications of skin cancer, commonly called the ABCDE rule, proposed by the American Society for cancer [11]. On the other hand, the performance accuracy of the manual diagnosis of this rule is around 59~80% [12]. However, performance accuracy only depends on the test called ‘biopsy’ [13]. The diagnosis of melanoma is looked for at its initial phases because it can be cured at this early phase through excision of the affected lesion. Manual diagnosis, on the other hand, first requires an expert dermatologist, and then, in the second step, the decision is sent to for a subjective variation analysis. All of these manual diagnoses take time and may be risky since patients’ lives are in jeopardy [14].
Therefore, to facilitate the patients as well as dermatologists, automatic diagnosis is significant. It is a reliable procedure that also saves time by reducing the series of biopsies tests. The segmentation of skin lesions is crucial in the performance of diagnostics in these automated CAD systems of skin disease diagnostics. Segmentation is an important task to analyze medical images by dividing the pixels of images into appealing regions based on various characteristics including texture, grey level, color, as well as intensities of pixels. Generally, in the CAD system, the underlying modalities, such as a dermoscopic image, are employed to identify skin cancer. In this scenario, accurate segmentation is critical because segmentation localizes the lesion regions, and by using these localized areas, the detector model will result in a more ideal classification of skin cancer, such as melanoma or benign. Poorly located regions or faulty segmentation will result in incorrect disease diagnosis and analysis. Hence, in existing studies, the dermoscopy images approach is utilized to deeply examine the lesion in terms of segmentation and to improve its diagnosis [15]. These images visually enhance the clarity to a certain level, but an accurate diagnosis of melanoma continues to be challenging due to the size of the lesion and arbitrary shape. In addition, variations in hue, edges, form, and noise such as hair and air bubbles also cause difficulty to segment out the lesion. Furthermore, the contrast between the lesion and the surrounding area is low, which also hinders precise segmentation. Numerous researchers have proposed some traditional approaches such as Otsu [16] and Stochastic [17] to segment out melanoma, but all of these thresholding algorithms are not automated and suffer from the problem of under or over-segmentation due to artifacts.
Recent studies have used deep learning-based frameworks in medical applications for melanoma segmentation and detection. However, most research has already taken significant advances toward automating analysis by employing CNN in a variety of image recognition studies [18]. In the case of CNN, it performs automatic feature learning after receiving image data by utilizing various loss functions and parameters based on the task. For instance, we take the naive technique for minimizing the Euclidean distance between predicted and ground truth by initializing CNN. At first, CNN will start producing blur images [19,20] because Euclidean distance can only be minimized by taking the average of all possible outputs. In terms of the loss function, forcing CNN to do what we want is an open problem and requires domain experts. In all of these existing methods, the main problem in melanoma lesion segmentation is the challenge of visual similarities between melanoma lesions and normal skin. Despite dermoscopic images’ increased sharpness, accurate melanoma localization from dermoscopic images remains difficult due to broad variance in color, texture, structure, and the presence of artifacts such as hair, gel bubbles, and clinical rule markings.
Recently, generative deep learning models, notably Generative Adversarial Networks (GANs), have been applied in a number of applications for a variety of tasks. A GAN trains a generative model to minimize the loss after learning the loss associated with classification attempts [16,21]. In the literature context of melanoma lesion segmentation, several methods are proposed, which include traditional methods such as Premaladha [22], thresholding-based [23], etc.; however, these methods fail when some diverse changes are observed in melanoma lesions images since these are parametric methods that are set manually. Recently, deep learning-based methods are widely used for classification and segmentation. The segmentation models include U-Net, FCN, and many more. However, the evolution of generative models opens up new avenues for solving diverse problems. As a result, one key research issue arises: what if these generative models are employed to perform segmentation? Second, the segmented images must be generated from specific dermoscopy images in which the lesion is emphasized, which is why conditional GANs are one of the best variants [21]. In CGANs, the images are generated depending on a Condition. This characteristic makes the cGAN pix-to-pix translation task where we can apply conditions on an input image and can generate its corresponding output images. There are some GAN-based approaches for skin lesion segmentation in existing studies, and the types of GANs used in these studies vary. Some have utilized style-based GANs [24], some have utilized them for augmentation [25], some have utilized traditional deep Generative adversarial networks (DGANs) [26], etc. Here, one major research question arises, what if skin lesions are segmented by structuring the problem in such a manner that generative models generate the segmented images based on the conditions? To answer this question, we have suggested conditional GANs for the task of skin lesion segmentation. Moreover, the major reason behind using cGANs is to assess the performance of segmentation tasks by using more recent and advanced deep learning models in comparison with existing traditional methods especially the melanoma lesions with cGANs. In addition, we also performed the preprocessing on dermoscopy images to enhance them to be more effectively utilized by cGANs. Thirdly, we performed the experimentation on three different datasets having a diverse range of skin lesion images to properly analyze the performance of CGANs for skin cancer lesion segmentation tasks. Our contribution is summarized in the following points:
  • Conditional Generative Adversarial Networks (CGANs) are suggested to carry out the task of melanoma lesion segmentation and multiple types of cancer from a single image.
  • Skin refinement as a preprocessing step is employed to automatically remove the artifact from images.
  • The proposed segmentation technique accurately segments the affected lesion by overcoming the challenges presented in the ISIC2016, DermIS, and DermQuest datasets.
The rest of the paper is organized in the following way: Section 2 thoroughly explains the current approaches. Section 3 explains our proposed methodology in detail. Section 4 explains the experimental detail and results. Lastly, Section 5 provides the conclusion of the paper.

2. Related Work

Melanomic segmentation is a critical method for improving accuracy when developing automated skin cancer detection algorithms. Approaches to automatic segmentation are further classified as classical methods and deep learning methods. The traditional methods mainly include iterative selection [27,28], adaptive threshold [23], iteration merging of regions [17], and the Otsu threshold [29]. However, the performance of threshold-based segmentation approaches is affected due to the presence of artifacts in images [27,29]. In [29], the performance of the study is noticeable, but segmented image boundaries are irregular and reduce image resolution. In [30], the researchers overcame the limitations that are exhibited in [29]. Another group of researchers [17,31] applied a region-merging approach for segmentation. This approach groups the intensely similar regions of the image. To reduce the effects of hues, contrast, and illustration, the region-merging approach is effective. In [17], these similar attributes were utilized for the segmentation of lesions. In [32], for image segmentation, an active contour model governed by an adaptive local pre-fitting energy function relying on Jeffreys divergence is presented. When contrasted with local region-based algorithms as well as previously developed algorithms, their suggested method not only reduces computing costs significantly but also enhances segmentation results. Such active contour models are also utilized in existing studies as a post-processing stage after the segmentation results of the melanoma lesions [33]. These active contour models in conjunction with morphological operations are also utilized as a post-processing step [34]. Moreover, Fuzzy-C means-based clustering is also integrated with active contour models to further enhance skin lesion segmentations [35]. These traditional methods perform well; however, these are parametric methods that require multiple parameters to be manually specified to achieve excellent segmentation, such as thresholding-based methods.
Deep learning-based methods have recently been widely adopted in practically all application domains [36,37,38,39]. They are also very much applicable in medical imaging domains [40,41,42]. For instance, in [43], a deep learning method based on transfer learning is proposed for the classification of skin lesions from images. In this study, automated features are retrieved from images utilizing several pre-trained models such as VGG19, Inception V3, ResNet50, and SqueezeNet, and the best results are obtained. Rather than focusing just on improving feature extraction, feature selection is also considered to increase performance. For instance, in [44], several feature selection methods are proposed such as gradient boosting, statistical methods, and optimization algorithms such as PSO. Likewise, in [45], the deep learning method by using U-Net along with stochastic weighted averaging for the segmentation task of melanoma is suggested. When compared to existing approaches, their suggested technique achieves both high precision and real-time efficiency. Subsequently, in some studies, more challenging skin cancer lesion images are utilized to evaluate the performance of the algorithms [46].
In addition to the aforementioned literature, several advanced deep learning methods, such as Generative adversarial networks (GAN), are now being used in the majority of medical imaging applications. For instance, in [47], adversarial training and deep transfer learning-based methodology are proposed to automate the detection of melanoma detection. Their trained model minimizes the focal loss function, which aids the model in optimal learning from difficult samples while underweighting the simple ones. Similarly, in [26], the skin lesion classification method based on GAN with an improved set of hyperparameters is proposed to overcome the problem of network instability. In addition, in [26], the GAN-based method is utilized for the task of augmentation. They are also utilized in an unsupervised manner for different tasks related to skin cancer lesions such as removing noise from the images as well as other artifacts [48]. Current studies have utilized the cGAN on some discrete labels [16,49,50] text [51], and images as well. Image-conditional frameworks undergo the prediction of images from future maps [52], normal maps [53], generation of images from sparse annotation [54,55], and generation of the product photo [56]. By forcing the output to be conditional on the corresponding input, some studies have utilized GAN and attained high results in style transfer [57], impainting [19], super-resolution [57], and manipulation of images [58].
To diagnose the affected region in dermoscopic images, some approaches [59,60] have applied object-detection algorithms. The hyper-graph technique was utilized in [59] to map saliency by utilizing super-pixel information. Other researchers [14,15,61,62] utilized deep learning-based approaches for segmentation and attained significant results as compared to traditional approaches. In [14], an FCRN, i.e., fully-convo-residual network, was proposed to overcome the overfitting issue in segmenting melanoma. In [15], region-based CNN was utilized to localize the lesion and then utilize the machine learning fuzzy clustering approach.
Moreover, in [61], a 19-layer CNN is designed to enhance the performance accuracy by segmentation of melanoma. In their research, they used Jaccard distance as a loss function. This loss function overcomes the overfitting issue between melanoma and normal skin images, consequently improving performance accuracy. In [62], FRCN, i.e., full-resolution CNN, was proposed to segment melanoma. This CNN learned segmentation of the model through a full image without applying any preprocessing to the dermoscopy images. In [63], to segment melanoma and reduce the effect of artifacts, a hybrid approach was introduced with the combination of Convolutional and Recurrent networks. However, two-phased object detection frameworks such as RCNN generate approximately two thousand regions for each image to estimate melanoma lesions.
As compared with the other approaches, the proposed approach of cGAN is different. We have used U-Net-centered architecture [64] for our generator and utilized a convolutional Patch GAN classifier for the discriminator. For purpose of local statistical analysis, a very similar Patch GAN architecture was utilized in [57].

3. Materials and Methods

In this section, we go over the proposed work step by step. To summarize, the steps of the proposed framework begin with data preprocessing, followed by data augmentation, and finally, using the processed images to train the CGANs model. More precisely, we begin by showing how the melanoma lesion segmentation task can be handled using CGANs, and then in the next part, we outline the problem in more thorough steps, before explaining the internal architectural details and training of the CGANs. In the last, the hyperparameters of the model are also discussed. The main overview of the proposed methodology is depicted in Figure 1.
GAN is a generative model and it learns how to map using a random noise vector z to output images y , G : z → y [21]. On the other hand, conditional GAN is a model that learns mapping by observed images x and random noise-vector z , to y , G :   x , z → y . The G , i.e., generator, is trained to produce output that cannot be distinguished from an original image by D , i.e., an adversarially trained discriminator, which is trained to classify real and fake images.

3.1. Preprocessing

The initial stage of the proposed model is image enhancement because medical images must be preprocessed before being used as input. This technique also leads to highly accurate segmentation since more sharp and well-contrasted images allow the model to pinpoint the lesions because the lesion may have a very similar visual appearance to normal skin in its early stages. The images are usually in the form of a 2D array or a 3D array of pixels. The color (RGB) image is the 3D array of pixels. Frequently utilized preprocessing approaches include smoothing of the images, resizing, ROI detection, and de-noising. Gaussian Smoothing is a highly recommended approach for removing the artifacts from the images. Along with the Gaussian function, we used Dilation followed by Erosion operation (i.e., Morphological closing) and sharpening kernels in this research study to eliminate hair and air bubbles from images while also increasing image contrast. This procedure dilates a dermoscopic image of skin cancer and afterward erodes that resulting image, employing the same structural element including both processes. Morphological closure is beneficial for refilling small gaps in conjunction with maintaining the size as well as the shape of larger holes in dermoscopic images. Figure 2 shows some sample images that undergo these preprocessing steps, where the first column shows the original input images, the second column is the morphological closing result on images, and the last column shows the result of sharpening.

3.2. Data Augmentation

Normally, the training samples that are publicly available for all the classes are not distributed equally, so this will create a class imbalance problem. Second, for deep learning models, an abundance of data will be the ultimate need for improved generalization and performance. The datasets used in this study have a very limited number of samples, specifically DermIS and DermQuest. Hence, in this study, we extend the samples by utilizing different augmentation methods such as flipping, cropping, and rotating at different angles. Table 1 shows these parameters. By applying all these parameters, we generate 15 new samples from a single image. The sole reason we use this step in our framework is to address overfitting issues and improve the model’s performance accuracy.

3.3. Melanoma Lesion Segmentation Using cGANS

Generative Adversarial Networks (GANs) are sophisticated deep learning models that come within the domain of generative models. In general, generative models are used to produce data artificially for a variety of objectives. There are several types of GANs such as Wasserstein GANs [65], Conditional GANs [66], Cycle GANs [67], Style-based GANs [68], Progressive GANs [69], etc. However, in this research study, we employ Conditional GANs for the task of melanoma segmentation. The major rationale for using this GAN is to perform lesion segmentation since, in this type of GAN, the data that are generated artificially are conditioned on certain criteria. For instance, if class labels are provided, those can be utilized as input. However, in basic GANs, data samples are generated at random from random noise vectors and there is no method to control the sorts of images produced. Therefore, we formulate the problem of skin lesion segmentation with the help of cGANs. It involves the generation of images with a condition. In the proposed method this condition is a source image, i.e., the dermoscopic image of skin cancer, while the outcome of the model is segmented images in which the lesion is highlighted. This type of image generation is also considered targeted image generation in which both types of CGAN modules, i.e., generator and discriminator, are conditioned on dermoscopic images during training.

3.3.1. Problem Formulation

To meet the contributions presented in this study, we employ the cGANs for melanoma lesion segmentation. As stated previously, we first preprocess the dermoscopic images to eliminate the noise as well as artifacts from the images. Later on, the preprocessed images and their ground truth images are used as inputs to cGANs to generate the segmented images.
The proposed model has two deep learning models, namely, the generator and discriminator. The objective of the generator is to generate the segmented images of skin lesions, while the discriminator classifies whether the generated segmented image of the lesion is real or fake. The weights of the discriminator are updated by themselves without taking the generator into account, while the weight updates of the generator are performed with the help of the discriminator model. Hence, the objective of conditional GANs is given below [66]:
L C G A N   G ,   D = E x , y   l o g D x , y + E x , z   l o g   1 − D x , G x , z
In the above Equation (1), G is the generator and D is the discriminator, z is the random-noise vector, x is the input (i.e., dermoscopic image and real ground truth image of lesion), and y is the generated segmented images of lesions by the generator, and term L C G A N is the loss function of conditional GANs. Moreover, G is for minimization of objective, whereas D is for maximization, i.e., [66],
G * = a r g m i n G   m a x D   L c G A N   G , D
In the above Equation (2), G * is the objective of the generator.

3.3.2. Design and Architecture of cGAN

The overall architecture of cGANs consists of two further deep learning models, i.e., discriminator and generator. The proposed discriminator model for classifying the segmented skin lesion images is simply the deep convolutional neural network. The input of this CNN is the dermoscopic image of skin lesion (referred to as the source image) and corresponding segmented ground truth images (referred to as the target image). The output of the model is to estimate the likelihood that the segmented ground truth image is a real or fake translation of the dermoscopic image of the skin cancer. The input dimension of both dermoscopic and ground truth images is set to 256 × 256 × 3 . More precisely, the discriminator is built over the improved receptive field in which associations among the outcome of the model are related to the total number of pixels in the dermoscopic image. This model is referred to as patch-GAN in which the outcomes are mapped over the patch of 70 × 70 image provided at input. This module simply penalizes architecture at the patch level. The discriminator attempts to determine if every N × N patch in the segmented ground truth image is real or fake.
Hence, the Patch-GAN-based discriminator first concatenates the two inputs, i.e., dermoscopic image and ground truth image. Later on, the discriminator model consists of five convolution layers having units 64, 128, 256, 512, and 512, respectively. A batch-normalization layer and Leaky ReLu activation function are applied after every convolution layer. The alpha parameter in the Leaky ReLu activation function is set to 0.2. The size of kernels in the discriminator is set to 4 by 4, and the weights of these kernels are initialized with Random Normal with a standard deviation of 0.02. Finally, a 2D convolutional layer is employed that performs the predictions in the form of a patch. This patch output is given to the sigmoid layer, resulting in the label of the image as being real or fake. Following on, the architecture of the generator model is based on UNet including the modules of the encoder and decoder. The input of the generator model is the dermoscopic image (also referred to as the source image) and the output of the model is the segmented image in which the lesion is highlighted as foreground pixels. This can be accomplished by first encoding the image followed by passing the resulting feature maps to the bottleneck layer. Later on, the upsampling path also called the decoder model is employed to upsample the resulting information from the bottleneck path in the form of a segmented image. Skip connections are also inserted among the encoder and decoder modules. The input of the dermoscopic image is passed through seven convolution blocks. Each convolution block consists of one convolution layer followed by batch normalization and leaky ReLu. The kernel size in the convolutional layers is 4 by 4, the stride value is 2, and the weight initialization is Random Normal. There are 7 encoder blocks in total, with a total of 64, 128, 256, 512, 512, 512, and 512 convolutional filters in each block. Similarly, the bottleneck part consists of one convolution layer with the activation function ReLu. This layer’s convolutional filters are similarly 512 with stride value of 2 and kernel size of 4. Following on, there are seven decoder blocks consisting of transposed convolutions as well as batch normalization. A dropout layer of 0.5 is also after batch normalization in the decoder part of the generator model. Finally, a 2D convolutional layer is added to generate 2D segmentation map results or to generate a segmented image in which the skin lesion is highlighted.

3.3.3. Training of cGAN

The discriminator is trained with original ground truth images of skin lesions as well as generated (fake) ground truth images of skin lesions. However, the training of the generator is dependent upon the discriminator. An update of the generator model is undertaken to reduce the loss of discriminator for the segmented skin lesion images that are labeled as real. Through this process, it is expected that the best-quality segmented images will be generated. To reduce the L 1 loss, we also update the generator, as this loss represents the error between the real and generated segmented images of skin lesions. Therefore, it is logical to state that generator weights are updated with both adversarial and L 1 loss. The generator model is stacked over the discriminator, resulting in a new model having weights of the other two models. On the other hand, the discriminator weights are adjusted independently. In addition, to test the importance of conditioning the discriminator, an unconditional version wherein the discriminator does not perceive x is also given in Equation (3) [66]:
L G A N G , D = E y   l o g D y + E x , z   l o g   1 − D G x , z
In the above Equation (3), G is the generator and D is the discriminator, and term L G A N is the loss function of conditional GANs. Recent methods have shown the significance of integrating the objective of GAN in a traditional loss function, such as distance L 2 [19]. The job of D is still the same, but the G is tasked not just to fool the D but also to near the ground truth (i.e., segmented lesion images) of the L 2 sense. We can explore this option by utilizing L 1 as compared with L 2 , as L 1 produces low blur [66].
L L 1   ( G ) = E x , y , z   ‖ y − G x , z ‖ 1
The final objective of the model is given in Equation (5) [66]:
G * = a r g m i n G   m a x D L c G A N G , D + λ L L 1 G
In the above Equation (5), the λ is the weight of the loss function and G * is the final objective of the model. The network could still acquire a map from x to y (i.e., dermoscopic image to segmented skin lesion image) without z , but it would not produce deterministic output and, thus, fail to match any distribution other than a delta function. The previous GANs had acknowledged this and provided Gaussian-noise z as an input to the G , with x [53]. In early findings, no efficient approach exists. The G only learns to overlook noise, which is consistent with the study in [52]. As an alternative, noise is given only in form of dropout in the final model, applied on numerous layers of the model for training and testing time as well. Training of CGANs is depicted in Figure 3.

3.3.4. Hyper-Parameters of cGAN Model

The discriminator’s hyperparameters include the loss function, which is binary-cross entropy, a weight optimizer set to Adam, a learning rate of 0.0002, and a β value in the optimizer set to 0.5; in addition, the padding in convolutional layers remains “same” while the weights of the discriminator were initialized with the “Random Normal” method. Similarly, the hyperparameters of the generator model include weight initialization, weight optimizer, and learning rates, which are similar to those of the discriminator. In addition, the proposed algorithm is implemented in Python with a Keras deep learning framework and simulations were carried out on Google Colab with K80 GPU along with 12 GB of RAM. Moreover, the training time for one simulation of CGANs is about 3.5 h approximately.

4. Results

In this section, we analyzed and discussed the findings of CGAN-based melanoma image segmentation. We also discussed the evaluation metrics and data used in the study.

4.1. Datasets

To look at the cGAN generality, we tested the approach on three different datasets, i.e., DermIS [70], DermQuest [70], and ISCI2016 [71]. In all these datasets, we had color images. The DermQuest dataset contains a total number of 274 images along with their corresponding ground truth images. Similarly, DermIS datasets contain a total of 69 images and their ground truth images. The total number of image samples in both datasets is quite low; thus, we used data augmentation to increase the number of samples for both of the datasets. Likewise, the dataset ISCI2016 comprised 900 melanoma images for training and 379 testing images. Figure 4 shows the sample images from the datasets.

4.2. Evaluation Metrics

To evaluate the performance of the model, we evaluated the datasets utilizing accuracy measures, dice score, specificity, sensitivity, and Jaccard score. The following equations illustrate these measures:
Accuracy   = TP + TN TP + TN + FP + FN
Dice   score   = 2 . TP 2 . TP +   FP + FN
Specificity   = TP TP + FP
Sensitivity   = TP TP + FN
Jaccard   score   = TP TP + FP + FN
In the above equations, TP is the value of true positives, FP is the value of false positives, TN is the value of true negatives and FN is the false negative value, and 2 shows the 2 x values of the corresponding value.

4.3. Results on DermIS Dataset

All experiments are carried out by utilizing melanoma images and their related ground mask images. As previously stated, we used three separate datasets in our research. Artifacts such as hair, air bubbles, and other noise can be seen in the dataset images. The existence of these artifacts has an impact on the outcome of our study. To address this issue, we used several preprocessing steps as well as augmentation methods on just the DermQuest and DermIS datasets. This step is performed to increase the data samples as fewer training samples lead to overfitting and class imbalance issues. Figure 5 shows the image samples after augmentation. For the melanoma lesion segmentation, we utilized the cGAN model. This automatically learns from the input images and provides us with the required melanoma lesion segmentation. Figure 6 shows the segmentation of melanoma on the DermIS dataset from the skin with the respective masks and contour images. In Figure 6, column A shows the preprocessed images of the DermIS dataset. In the next step, column B shows the actual masks of the corresponding input images. Next, column C shows the actual images with a contour. The red dots in column C shows the contour.
The masks that were predicted by the proposed model are shown in column D and the last column E, the sample image that was predicted by the model is shown with a contour mask around their boundaries. Furthermore, as demonstrated in Table 2, the accuracy, dice, Jaccard, and specificity attained on this dataset are 97%, 93%, 97%, and 95%, respectively.

4.4. Results on DermQuest Dataset

Similar to the DermIS dataset, all the experiments are conducted by utilizing dermoscopic images and their ground masks. Likewise, in DermIS, we also applied preprocessing steps on this dataset to reduce the artifacts such as hair, air bubbles, and other noise. To improve our model’s performance, we used augmentation steps on this dataset to increase the number of data samples, as fewer training samples lead to overfitting and class imbalance issues. For the lesion segmentation, we utilized the cGAN model. This automatically learns from the input images and provides us with the required melanoma lesion segmentation.
Figure 7 shows the segmentation of melanoma lesions of the DermQuest dataset from the skin with the respective masks and contour images. In Figure 7, column (A) shows the original DermQuest preprocessed images. In the next step, column (B) shows the actual masks of the corresponding input images. Next, column (C) shows the actual images with contour. The red dots in column C shows the contour. The masks that were predicted by our proposed model are shown in column D and the last column E, the sample image that was predicted by our model is shown with a contour mask around its boundaries. Similarly, the accuracy, dice, Jaccard, and specificity achieved on this dataset are 95%, 95%, 95%, and 90%, respectively, as shown in Table 2. Figure 8 shows the sample test image results from all three datasets, i.e., DermIS, DermQuest, and ISCI 2016, in the form of contour images. In Figure 8, the above row x shows the input images while the second row y shows the results in form of contour images.
In addition, Figure 9a shows the loss of the generator during the training process. The blue dotted lines in this figure represent the generator’s loss.
Similarly, Figure 9b shows the loss value concerning the number of epochs during the training of the network in which dotted blue lines represent discriminator loss on real examples, whereas orange dotted lines show discriminator loss on fake examples (i.e., fake segmented images).

4.5. Comparison with ISCI2016 Challenge

The proposed model was also evaluated on a benchmark dataset named ISIC 2016 by “International Symposium on biomedical images (ISBI) in the challenge of Skin lesion analysis towards Melanoma detection”. This dataset contains 1279 images, 900 for training, and 379 for testing. In this ISBI 2016 challenge, nearly 28 groups participated and submitted their results, as shown in Table 3. The ISBI ordered the participants to segment as per their highest average score of Jaccard. The accuracy, dice, Jaccard, and specificity achieved by the proposed model on the ISCI2016 challenge dataset are about 95%, 90%, 95%, and 91%, respectively, as shown in Table 3. From Table 3, it is observed that to reduce the segmentation error, all the participants utilized deep learning because of precise segmentation and effective backpropagation learning. Moreover, the participants of ISBI 2016 also utilized a pre-trained model such as AlexNet [75], VGG16 [76], and ResNet [77] to estimate the boundaries of melanoma.
In comparison, our technique achieved the best outcomes by outperforming all previous trials. Even though we did not use post-processing to improve segmentation outcomes, the Jaccard score of this study is the greatest among the leading participants. The second top participant in the ISBI challenge is CUMED; they utilized a fully residual network without utilizing any preprocessing and post-processing approaches. We also compared the performances with some traditional methods of segmentation. These types of comparisons on the same ISBI2016 dataset have also been carried out and performed in the existing research studies [15,33]. It is noticeable from Table 2 that these approaches, including Adaptive Thresholding [23], Bootstrap learning [60], Contextual [59], ISO [27], Level set [72], Sparse coding [73], Region growing [31], and FCN [74], give the lower results as compared to the proposed approach. Table 2 presents the comparison between existing techniques with our proposed framework. Besides traditional approaches, we also performed a comparison with deep learning techniques such as FCN. With this method, the Jaccard score is about 86%, which is lower than the proposed method. It was logically concluded that the suggested method is good enough for segmenting the melanoma lesions from given dermoscopy images.
Moreover, if the proposed CGAN model is evaluated in terms of addressing challenges over existing methods then one of the advantages of the proposed CGANS is that it efficiently handles melanoma lesion segmentation in the presence of visual appearance, texture, size, and shape problems; however, traditional methods such as OSTU or thresholding-based segmentation methods fail when some diverse changes are observed in the melanoma lesion images because these are parametric methods that must be set manually. More specifically, the same threshold criterion may not be applicable for all sorts of dermoscopic images having contrast and sharpness issues. The proposed CGAN model does not require any post-processing steps or any threshold for every specific dermoscopic image. CGANs are generative deep learning models utilizing convolution and max-pool layers to enable automated feature learning. These layer-level configurations in the discriminator and generator models are first utilized to extract the features, which are then utilized to generate the required segmentation maps. These layers hold the properties of scale invariance, preserve neighborhood semantics with the help of shared parameters among kernels, and capture different receptive fields using different size kernels. The feature learning process starts from low-level features to high-level features including learning edges, edges to shapes, and shapes to structures. This kind of automated feature learning will handle variability in object size, e.g., in our case, lesion size, shape, and color. Second, because CGANs are a generative class of deep learning models, their distinct learning or training technique will provide an additional benefit in accurate melanoma segmentation. The objective of the discriminator model in CGANs is to classify the real and fake segmented images (i.e., generated by the generator model); however, the generator model generates images that are closer to real images based on the feedback of the discriminator. The generator model is updated to decrease discriminator loss for segmented skin lesion images that are labeled as real. The final objective given in Equation (5) will be able to train the discriminator and generator model in a zero-sum game, and based on this, the generator model will generate more accurate segmentation maps for skin lesions in the presence of variability in lesion size, shape, and color-related problems. Therefore, the strength of the proposed CGAN model is that it can handle these challenges and segment the skin lesions in more precise localizations. The suggested model is based on deep learning, which is a more generalizable and cutting-edge way than conventional classic methods such as active contour, level set, Otsu, and so on. The comparison study in Table 2 and Table 3 demonstrates that the suggested model is more robust and performs well on all performance metrics, including accuracy, dice score, Jaccard, sensitivity, and specificity. The suggested model’s scalability includes the ability to be extended to other forms of medical imaging tasks, such as semantic segmentation of different kinds of cancers, such as blood cancer, in which blood cells are segmented for medical image analysis. Similarly, it may be extended to perform data augmentation in order to address concerns of class imbalance and dataset scarcity. On the other hand, it is essential to discuss the limitations of the model for further future research. As a result, one limitation of the model is that given a skin lesion, it can precisely localize the skin lesion; hence, it may be beneficial if, in addition to localization, the model outputs the category of skin lesion, such as melanoma’s classes or the severity of the lesions. However, because the suggested CGANs are based on generative models, an additional model that classifies the localized segmented images is required to undertake a more comprehensive analysis of dermoscopic images. Second, assessing CGAN performance as cross-data set validation is critical for determining generalizability.
Furthermore, if the suggested model is examined to see how it might be used in real-world clinical settings, various important aspects emerge. For example, in order to deploy it in a real-world scenario, the model must first be converted to deployable APIs. Second, a system with strong computational capabilities, such as GPUs, RAM, and other resources, is required for smooth operation. However, establishing IoMT-assisted medical healthcare systems presents a variety of challenges. These challenges include security threats, cost, and other administrative challenges. The sensors might be faulty sometimes; thus, an appropriate methodology for validating the data collected from the sensors is required. To guarantee that data are gathered effectively, as well as sensors offering reliable information, the IoMT system must adopt a high degree of precision. Moreover, the misinterpretation of data as well as information resulting from devices also leads to adding a limit to the IoMT-based healthcare systems. From the perspective of the proposed model, as well as other CAD solutions for different diseases, it is logical to say that such a system cannot completely replace doctors but can work as an assistant for the doctor, as well as provide advantages to patients in remote locations.

5. Conclusions

In this paper, we suggested an approach for segmenting melanoma lesions using Conditional Generative Adversarial Networks. The proposed method is appropriate for accurately segmenting the lesion. As in melanoma lesion segmentation, the normal and affected regions are visually the same, so this approach learns the loss function, as well as maps input images to corresponding output images, for accurate segmentation. The proposed technique efficiently detects melanoma lesions as compared to the state-of-the-art approaches. This cGAN network automatically learns the loss function and this characteristic makes it applicable to many other medical image segmentation problems. We have attained higher performance accuracy for the DermIS and DermQuest datasets, i.e., 99% and 97%, respectively. While the challenging dataset ISCI 2016 also produces the greatest results, i.e., 95% performance accuracy. Future work will entail improving the cGAN model and adding more challenging images, as well as increasing the number of samples in the training data. In the future, we hope to analyze particular color images rather than RGB color spaces.

Author Contributions

The manuscript was written through the contributions of all authors. Z.A. was responsible for the conceptualization of the topic; article gathering and sorting were carried out by Z.A., S.N. and H.Z.; manuscript writing, original drafting, and formal analysis were carried out by Z.A., S.N., H.Z., J.C. and Y.K.; writing of reviews and editing was carried out by Z.A., S.N., H.Z., J.C. and Y.K.; and Y.K. led the overall research activity. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the research fund of Chungnam National University.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data used in this work are publicly available and can also be accessed by contacting the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Rajab, H.; Cinkelr, T. IoT based smart cities. In Proceedings of the 2018 International Symposium on Networks, Computers and Communications (ISNCC), Rome, Italy, 19–21 June 2018; pp. 1–4. [Google Scholar]
  2. Stergiou, C.; Psannis, K.E.; Kim, B.-G.; Gupta, B. Secure integration of IoT and cloud computing. Future Gener. Comput. Syst. 2018, 78, 964–975. [Google Scholar] [CrossRef]
  3. Elhattab, K.; Abouelmehdi, K.; Elmoutaouakkil, A. Internet of Things (iot) for Smart City, agriculture and healthcare. J. Theor. Appl. Inf. Technol. 2022, 100, 1104–1112. [Google Scholar]
  4. Adeniyi, E.A.; Ogundokun, R.O.; Awotunde, J.B. IoMT-based wearable body sensors network healthcare monitoring system. In IoT in Healthcare and Ambient Assisted Living; Springer: Berlin/Heidelberg, Germany, 2021; pp. 103–121. [Google Scholar]
  5. Islam, M.K.; Kaushal, C.; Amin, M.A.; Algarni, A.D.; Alturki, N.; Soliman, N.F.; Mansour, R.F. A secure framework toward IoMT-assisted data collection, modeling, and classification for intelligent dermatology healthcare services. Contrast Media Mol. Imaging 2022, 2022, 6805460. [Google Scholar] [CrossRef] [PubMed]
  6. Sun, Y.; Lo, F.P.-W.; Lo, B. Security and privacy for the internet of medical things enabled healthcare systems: A survey. IEEE Access 2019, 7, 183339–183355. [Google Scholar] [CrossRef]
  7. Khan, M.A.; Algarni, F. A healthcare monitoring system for the diagnosis of heart disease in the IoMT cloud environment using MSSO-ANFIS. IEEE Access 2020, 8, 122259–122269. [Google Scholar] [CrossRef]
  8. Jain, S.; Nehra, M.; Kumar, R.; Dilbaghi, N.; Hu, T.; Kumar, S.; Kaushik, A.; Li, C.-Z. Internet of medical things (IoMT)-integrated biosensors for point-of-care testing of infectious diseases. Biosens. Bioelectron. 2021, 179, 113074. [Google Scholar] [CrossRef]
  9. Peter Soosai Anandaraj, A.; Gomathy, V.; Amali Angel Punitha, A.; Abitha Kumari, D.; Sheeba Rani, S.; Sureshkumar, S. Internet of medical things (iomt) enabled skin lesion detection and classification using optimal segmentation and restricted Boltzmann machines. Cogn. Internet Med. Things Smart Healthc. Serv. Appl. 2021, 311, 195–209. [Google Scholar]
  10. Esteva, A.; Kuprel, B.; Novoa, R.A.; Ko, J.; Swetter, S.M.; Blau, H.M.; Thrun, S. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017, 542, 115–118. [Google Scholar] [CrossRef]
  11. Razmjooy, N.; Sheykhahmad, F.R.; Ghadimi, N. A hybrid neural network–world cup optimization algorithm for melanoma detection. Open Med. 2018, 13, 9–16. [Google Scholar] [CrossRef]
  12. Masood, A.; Ali Al-Jumaily, A. Computer aided diagnostic support system for skin cancer: A review of techniques and algorithms. Int. J. Biomed. Imaging 2013, 2013, 323268. [Google Scholar] [CrossRef]
  13. Elston, D.M.; Stratman, E.J.; Miller, S.J. Skin biopsy: Biopsy issues in specific diseases. J. Am. Acad. Dermatol. 2016, 74, 1–16. [Google Scholar] [CrossRef] [Green Version]
  14. Yu, L.; Chen, H.; Dou, Q.; Qin, J.; Heng, P.-A. Automated melanoma recognition in dermoscopy images via very deep residual networks. IEEE Trans. Med. Imaging 2016, 36, 994–1004. [Google Scholar] [CrossRef]
  15. Nida, N.; Irtaza, A.; Javed, A.; Yousaf, M.H.; Mahmood, M.T. Melanoma lesion detection and segmentation using deep region based convolutional neural network and fuzzy C-means clustering. Int. J. Med. Inform. 2019, 124, 37–48. [Google Scholar] [CrossRef]
  16. Denton, E.L.; Chintala, S.; Fergus, R. Deep generative image models using a laplacian pyramid of adversarial networks. In Proceedings of the Advances in Neural Information Processing Systems 28: 29th Annual Conference on Neural Information Processing Systems 2015, Montreal, QC, Canada, 7–12 December 2015; pp. 1486–1494. [Google Scholar]
  17. Wong, A.; Scharcanski, J.; Fieguth, P. Automatic skin lesion segmentation via iterative stochastic region merging. IEEE Trans. Inf. Technol. Biomed. 2011, 15, 929–936. [Google Scholar] [CrossRef]
  18. Anwar, S.M.; Majid, M.; Qayyum, A.; Awais, M.; Alnowami, M.; Khan, M.K. Medical image analysis using convolutional neural networks: A review. J. Med. Syst. 2018, 42, 1–13. [Google Scholar] [CrossRef] [Green Version]
  19. Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; Efros, A.A. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, Seattle, WA, USA, 21–23 June 1994; pp. 2536–2544. [Google Scholar]
  20. Zhang, R.; Isola, P.; Efros, A.A. Colorful image colorization. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016, Proceedings, Part III 14; Springer: Berlin/Heidelberg, Germany, 2016; pp. 649–666. [Google Scholar]
  21. Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. Commun. ACM 2020, 63, 2672–2680. [Google Scholar] [CrossRef]
  22. Premaladha, J.; Priya, M.L.; Sujitha, S.; Ravichandran, K. Normalised Otsu’s segmentation algorithm for melanoma diagnosis. Indian J. Sci. Technol. 2015, 8, 1. [Google Scholar] [CrossRef]
  23. Silveira, M.; Nascimento, J.C.; Marques, J.S.; Marçal, A.R.; Mendonça, T.; Yamauchi, S.; Maeda, J.; Rozeira, J. Comparison of segmentation methods for melanoma diagnosis in dermoscopy images. IEEE J. Sel. Top. Signal Process. 2009, 3, 35–45. [Google Scholar] [CrossRef]
  24. Qin, Z.; Liu, Z.; Zhu, P.; Xue, Y. A GAN-based image synthesis method for skin lesion classification. Comput. Methods Programs Biomed. 2020, 195, 105568. [Google Scholar] [CrossRef]
  25. Pollastri, F.; Bolelli, F.; Palacios, R.P.; Grana, C. Improving skin lesion segmentation with generative adversarial networks. In Proceedings of the 2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS), Karlstad, Sweden, 18–21 June 2018; pp. 442–443. [Google Scholar]
  26. Mutepfe, F.; Kalejahi, B.K.; Meshgini, S.; Danishvar, S. Generative adversarial network image synthesis method for skin lesion generation and classification. J. Med. Signals Sens. 2021, 11, 237. [Google Scholar]
  27. Ridler, T.; Calvard, S. Picture thresholding using an iterative selection method. IEEE Trans. Syst. Man Cybern. 1978, 8, 630–632. [Google Scholar]
  28. Wadhawan, T.; Situ, N.; Lancaster, K.; Yuan, X.; Zouridakis, G. SkinScan©: A portable library for melanoma detection on handheld devices. In Proceedings of the 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Chicago, IL, USA, 30 March 2011–2 April 2011; pp. 133–136. [Google Scholar]
  29. Rajinikanth, V.; Madhavaraja, N.; Satapathy, S.C.; Fernandes, S.L. Otsu’s multi-thresholding and active contour snake model to segment dermoscopy images. J. Med. Imaging Health Inform. 2017, 7, 1837–1840. [Google Scholar] [CrossRef]
  30. Alcón, J.F.; Ciuhu, C.; Ten Kate, W.; Heinrich, A.; Uzunbajakava, N.; Krekels, G.; Siem, D.; de Haan, G. Automatic imaging system with decision support for inspection of pigmented skin lesions and melanoma diagnosis. IEEE J. Sel. Top. Signal Process. 2009, 3, 14–25. [Google Scholar] [CrossRef] [Green Version]
  31. Emre Celebi, M.; Kingravi, H.A.; Iyatomi, H.; Alp Aslandogan, Y.; Stoecker, W.V.; Moss, R.H.; Malters, J.M.; Grichnik, J.M.; Marghoob, A.A.; Rabinovitz, H.S. Border detection in dermoscopy images using statistical region merging. Ski. Res. Technol. 2008, 14, 347–353. [Google Scholar] [CrossRef]
  32. Ge, P.; Chen, Y.; Wang, G.; Weng, G. An active contour model driven by adaptive local pre-fitting energy function based on Jeffreys divergence for image segmentation. Expert Syst. Appl. 2022, 210, 118493. [Google Scholar] [CrossRef]
  33. Albahli, S.; Nida, N.; Irtaza, A.; Yousaf, M.H.; Mahmood, M.T. Melanoma lesion detection and segmentation using YOLOv4-DarkNet and active contour. IEEE Access 2020, 8, 198403–198414. [Google Scholar] [CrossRef]
  34. Abduljaleel, H.K. Deep CNN based skin lesion image denoising and segmentation using active contour method. Eng. Technol. J. 2019, 37, 464–469. [Google Scholar]
  35. Soomro, S.; Munir, A.; Choi, K.N. Fuzzy c-means clustering based active contour model driven by edge scaled region information. Expert Syst. Appl. 2019, 120, 387–396. [Google Scholar] [CrossRef]
  36. Bukhari, M.; Yasmin, S.; Sammad, S.; El-Latif, A.; Ahmed, A. A Deep Learning Framework for Leukemia Cancer Detection in Microscopic Blood Samples Using Squeeze and Excitation Learning. Math. Probl. Eng. 2022, 2022, 2801227. [Google Scholar] [CrossRef]
  37. Bukhari, M.; Bajwa, K.B.; Gillani, S.; Maqsood, M.; Durrani, M.Y.; Mehmood, I.; Ugail, H.; Rho, S. An efficient gait recognition method for known and unknown covariate conditions. IEEE Access 2020, 9, 6465–6477. [Google Scholar] [CrossRef]
  38. Yasmin, S.; Durrani, M.Y.; Gillani, S.; Bukhari, M.; Maqsood, M.; Zghaibeh, M. Small obstacles detection on roads scenes using semantic segmentation for the safe navigation of autonomous vehicles. J. Electron. Imaging 2022, 31, 061806. [Google Scholar] [CrossRef]
  39. Nawaz, H.; Maqsood, M.; Afzal, S.; Aadil, F.; Mehmood, I.; Rho, S. A deep feature-based real-time system for Alzheimer disease stage detection. Multimed. Tools Appl. 2021, 80, 35789–35807. [Google Scholar] [CrossRef]
  40. Al-Antari, M.A.; Al-Masni, M.A.; Choi, M.-T.; Han, S.-M.; Kim, T.-S. A fully integrated computer-aided diagnosis system for digital X-ray mammograms via deep learning detection, segmentation, and classification. Int. J. Med. Inform. 2018, 117, 44–54. [Google Scholar] [CrossRef]
  41. Hosny, K.M.; Kassem, M.A. Refined residual deep convolutional network for skin lesion classification. J. Digit. Imaging 2022, 35, 258–280. [Google Scholar] [CrossRef]
  42. Kassem, M.A.; Hosny, K.M.; Fouad, M.M. Skin lesions classification into eight classes for ISIC 2019 using deep convolutional neural network and transfer learning. IEEE Access 2020, 8, 114822–114832. [Google Scholar] [CrossRef]
  43. Khamparia, A.; Singh, P.K.; Rani, P.; Samanta, D.; Khanna, A.; Bhushan, B. An internet of health things-driven deep learning framework for detection and classification of skin cancer using transfer learning. Trans. Emerg. Telecommun. Technol. 2021, 32, e3963. [Google Scholar] [CrossRef]
  44. Maiti, A.; Chatterjee, B. The Effect of Different Feature Selection Methods for Classification of Melanoma. In Proceedings of the International Symposium on Signal and Image Processing, Online, 21–22 October 2020; pp. 123–133. [Google Scholar]
  45. Tang, P.; Liang, Q.; Yan, X.; Xiang, S.; Sun, W.; Zhang, D.; Coppola, G. Efficient skin lesion segmentation using separable-Unet with stochastic weight averaging. Comput. Methods Programs Biomed. 2019, 178, 289–301. [Google Scholar] [CrossRef]
  46. Navarro, F.; Escudero-Vinolo, M.; Bescós, J. Accurate segmentation and registration of skin lesion images to evaluate lesion change. IEEE J. Biomed. Health Inform. 2018, 23, 501–508. [Google Scholar] [CrossRef]
  47. Zunair, H.; Hamza, A.B. Melanoma detection using adversarial training and deep transfer learning. Phys. Med. Biol. 2020, 65, 135005. [Google Scholar] [CrossRef]
  48. Kim, D.; Hong, B.-W. Unsupervised feature elimination via generative adversarial networks: Application to hair removal in melanoma classification. IEEE Access 2021, 9, 42610–42620. [Google Scholar] [CrossRef]
  49. Gauthier, J. Conditional generative adversarial nets for convolutional face generation. Class Project for Stanford CS231N: Convolutional Neural Networks for Visual Recognition. Winter Semester 2014, 2014, 2. [Google Scholar]
  50. Mirza, M.; Osindero, S. Conditional generative adversarial nets. arXiv 2014, arXiv:1411.1784. [Google Scholar]
  51. Reed, S.; Akata, Z.; Yan, X.; Logeswaran, L.; Schiele, B.; Lee, H. Generative adversarial text to image synthesis. arXiv 2016, arXiv:1605.05396. [Google Scholar]
  52. Mathieu, M.; Couprie, C.; LeCun, Y. Deep multi-scale video prediction beyond mean square error. arXiv 2015, arXiv:1511.05440. [Google Scholar]
  53. Wang, X.; Gupta, A. Generative image modeling using style and structure adversarial networks. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016, Proceedings, Part IV 14; Springer: Berlin/Heidelberg, Germany, 2016; pp. 318–335. [Google Scholar]
  54. Karacan, L.; Akata, Z.; Erdem, A.; Erdem, E. Learning to generate images of outdoor scenes from attributes and semantic layouts. arXiv 2016, arXiv:1612.00215. [Google Scholar]
  55. Reed, S.E.; Akata, Z.; Mohan, S.; Tenka, S.; Schiele, B.; Lee, H. Learning what and where to draw. In Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain, 5–10 December 2016; pp. 217–225. [Google Scholar]
  56. Yoo, D.; Kim, N.; Park, S.; Paek, A.S.; Kweon, I.S. Pixel-level domain transfer. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016, Proceedings, Part VIII 14; Springer: Berlin/Heidelberg, Germany, 2016; pp. 517–532. [Google Scholar]
  57. Li, C.; Wand, M. Precomputed real-time texture synthesis with markovian generative adversarial networks. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016, Proceedings, Part III 14; Springer: Berlin/Heidelberg, Germany, 2016; pp. 702–716. [Google Scholar]
  58. Zhu, J.-Y.; Krähenbühl, P.; Shechtman, E.; Efros, A.A. Generative visual manipulation on the natural image manifold. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016, Proceedings, Part V 14; Springer: Berlin/Heidelberg, Germany, 2016; pp. 597–613. [Google Scholar]
  59. Li, X.; Li, Y.; Shen, C.; Dick, A.; Van Den Hengel, A. Contextual hypergraph modeling for salient object detection. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 3328–3335. [Google Scholar]
  60. Tong, N.; Lu, H.; Ruan, X.; Yang, M.-H. Salient object detection via bootstrap learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1884–1892.
  61. Yuan, Y.; Chao, M.; Lo, Y.-C. Automatic skin lesion segmentation using deep fully convolutional networks with jaccard distance. IEEE Trans. Med. Imaging 2017, 36, 1876–1886. [Google Scholar] [CrossRef]
  62. Al-Masni, M.A.; Al-Antari, M.A.; Choi, M.-T.; Han, S.-M.; Kim, T.-S. Skin lesion segmentation in dermoscopy images via deep full resolution convolutional networks. Comput. Methods Programs Biomed. 2018, 162, 221–231. [Google Scholar] [CrossRef]
  63. Attia, M.; Hossny, M.; Nahavandi, S.; Yazdabadi, A. Skin melanoma segmentation using recurrent and convolutional neural networks. In Proceedings of the 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), Melbourne, VIC, Australia, 18–21 April 2017; pp. 292–296. [Google Scholar]
  64. Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; pp. 234–241. [Google Scholar]
  65. Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein GAN. arXiv 2017, arXiv:1701.07875. [Google Scholar]
  66. Isola, P.; Zhu, J.-Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New York, NY, USA, 15–17 June 1993; pp. 1125–1134. [Google Scholar]
  67. Zhu, J.-Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2223–2232. [Google Scholar]
  68. Karras, T.; Laine, S.; Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 4401–4410. [Google Scholar]
  69. Karras, T.; Aila, T.; Laine, S.; Lehtinen, J. Progressive growing of gans for improved quality, stability, and variation. arXiv 2017, arXiv:1710.10196. [Google Scholar]
  70. Filali, I. Contrast Based Lesion Segmentation on DermIS and DermQuest Datasets. Mendeley Data, V2. 2019. Available online: https://doi.org/10.17632/bfmpbx432k.2 (accessed on 2 January 2023).
  71. Gutman, D.; Codella, N.C.; Celebi, E.; Helba, B.; Marchetti, M.; Mishra, N.; Halpern, A. Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (ISBI) 2016, hosted by the international skin imaging collaboration (ISIC). arXiv 2016, arXiv:1605.01397. [Google Scholar]
  72. Li, C.; Kao, C.-Y.; Gore, J.C.; Ding, Z. Minimization of region-scalable fitting energy for image segmentation. IEEE Trans. Image Process. 2008, 17, 1940–1949. [Google Scholar]
  73. Bozorgtabar, B.; Abedini, M.; Garnavi, R. Sparse coding based skin lesion segmentation using dynamic rule-based refinement. In Machine Learning in Medical Imaging: 7th International Workshop, MLMI 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, 17 October 2016, Proceedings 7; Springer: Berlin/Heidelberg, Germnay, 2016; pp. 254–261. [Google Scholar]
  74. Wei, Y.; Liang, X.; Chen, Y.; Shen, X.; Cheng, M.-M.; Feng, J.; Zhao, Y.; Yan, S. Stc: A simple to complex framework for weakly-supervised semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2016, 39, 2314–2320. [Google Scholar] [CrossRef] [Green Version]
  75. Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. Commun. ACM 2017, 60, 84–90. [Google Scholar] [CrossRef] [Green Version]
  76. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
  77. Yang, W.; Yuan, Y.; Ren, W.; Liu, J.; Scheirer, W.J.; Wang, Z.; Zhang, T.; Zhong, Q.; Xie, D.; Pu, S. Advancing Image Understanding in Poor Visibility Environments: A Collective Benchmark Study. IEEE Trans. Image Process. 2020, 29, 5737–5752. [Google Scholar] [CrossRef]
Sensors 23 03548 g001 550
Figure 1. A Pictorial Overview of Proposed Methodology.
Figure 1. A Pictorial Overview of Proposed Methodology.
Sensors 23 03548 g001
Sensors 23 03548 g002 550
Figure 2. Skin Refinement on ISBI2016 Dataset: Column 1 shows Original Images, Column 2 shows the images after closing morphological operation, and Column 3 shows results after a sharpening kernel is applied.
Figure 2. Skin Refinement on ISBI2016 Dataset: Column 1 shows Original Images, Column 2 shows the images after closing morphological operation, and Column 3 shows results after a sharpening kernel is applied.
Sensors 23 03548 g002
Sensors 23 03548 g003 550
Figure 3. Training of Conditional Generative Adversarial Networks for melanoma lesion segmentation.
Figure 3. Training of Conditional Generative Adversarial Networks for melanoma lesion segmentation.
Sensors 23 03548 g003
Sensors 23 03548 g004 550
Figure 4. Sample images from the datasets: Horizontally Row 1 shows some sample images from the ISCI2016 dataset. Row 2 shows some sample images from the DermQuest dataset and Row 3 shows some sample images from the DermIS dataset.
Figure 4. Sample images from the datasets: Horizontally Row 1 shows some sample images from the ISCI2016 dataset. Row 2 shows some sample images from the DermQuest dataset and Row 3 shows some sample images from the DermIS dataset.
Sensors 23 03548 g004
Sensors 23 03548 g005 550
Figure 5. The augmented samples of the images from the DermQuest dataset.
Figure 5. The augmented samples of the images from the DermQuest dataset.
Sensors 23 03548 g005
Sensors 23 03548 g006 550
Figure 6. Melanoma segmentation results of DermIS dataset from the skin with the respective masks and contour images.
Figure 6. Melanoma segmentation results of DermIS dataset from the skin with the respective masks and contour images.
Sensors 23 03548 g006
Sensors 23 03548 g007 550
Figure 7. Melanoma segmentation results of DermQuest dataset from the skin with the respective masks and contour images.
Figure 7. Melanoma segmentation results of DermQuest dataset from the skin with the respective masks and contour images.
Sensors 23 03548 g007
Sensors 23 03548 g008 550
Figure 8. Sample melanoma segmentation results of datasets in form of contour images.
Figure 8. Sample melanoma segmentation results of datasets in form of contour images.
Sensors 23 03548 g008
Sensors 23 03548 g009 550
Figure 9. Loss of generator (a) and discriminator (b) of CGANs model.
Figure 9. Loss of generator (a) and discriminator (b) of CGANs model.
Sensors 23 03548 g009
Table
Table 1. Types of Data Augmentation.
Table 1. Types of Data Augmentation.
Sr. NoAugmentation StepsParameters
1Rotate90°, 180°, 270°
2Crop from Right45°, 60°, 90°
3Crop from Left45°, 60°, 90°
4Crop from Top 45°, 60°, 90°
5Crop from Bottom45°, 60°, 90°
6Flipping Left Right
7Shiftingshifted by (25, 25) pixels
Table
Table 2. Comparison of state-of-the-art approaches with our proposed model.
Table 2. Comparison of state-of-the-art approaches with our proposed model.
TechniquesAccuracyDice ScoreJaccardSpecificity
Adaptive Thresholding [23]72%56%45%80%
Bootstrap learning [60]78%72%57%75%
Contextual [59]83%75%60%77%
ISO [27]82%68%56%78%
Level set [72]70%58%46%79%
Sparse coding [73]91%80%66%86%
Region growing [31]73%55%43%76%
FCN [74]82%82%86%70%
Proposed (DermQuest)99%95%99%90%
Proposed (DermIS)97%93%97%95%
Proposed (ISCI2016)95%90%95%91%
Table
Table 3. Comparison of proposed model with ISBI2016 challenge winners.
Table 3. Comparison of proposed model with ISBI2016 challenge winners.
TechniqueAccuracyDice ScoreJaccardSensitivitySpecificity
ExB95%91%84%91%96.5%
CUMED94%89.7%82.9%91.1%95.7%
Mahmudur95.2%89.5%82.2%88%96.9%
SFU-mial94.4%88.5%81.1%91.5%95.5%
TMUteam94.6%88.8%81%83.2%98.7%
Uit-Seg93.9%88.1%80.6%86.3%97.4%
IHPC-CS93.8%87.9%79.9%91%94.1%
UNIST94%86.7%79.7%87.6%95.4%
JoseLuis93.4%86.9%79.1%87%97.8%
Marcoromelli93.6%86.4%78.6%88.3%96.2%
Proposed95%90%95%91%90%
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Ali, Z.; Naz, S.; Zaffar, H.; Choi, J.; Kim, Y. An IoMT-Based Melanoma Lesion Segmentation Using Conditional Generative Adversarial Networks. Sensors 2023, 23, 3548. https://doi.org/10.3390/s23073548

AMA Style

Ali Z, Naz S, Zaffar H, Choi J, Kim Y. An IoMT-Based Melanoma Lesion Segmentation Using Conditional Generative Adversarial Networks. Sensors. 2023; 23(7):3548. https://doi.org/10.3390/s23073548

Chicago/Turabian Style

Ali, Zeeshan, Sheneela Naz, Hira Zaffar, Jaeun Choi, and Yongsung Kim. 2023. "An IoMT-Based Melanoma Lesion Segmentation Using Conditional Generative Adversarial Networks" Sensors 23, no. 7: 3548. https://doi.org/10.3390/s23073548

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop