remote sensing
Review
Deep Learning-Based Semantic Segmentation of Urban
Features in Satellite Images: A Review and Meta-Analysis
Bipul Neupane 1 , Teerayut Horanont 2, *
1
2
3
*
Citation: Neupane, B.; Horanont, T.;
Aryal, J. Deep Learning-Based
Semantic Segmentation of Urban
Features in Satellite Images: A
and Jagannath Aryal 3
Advanced Geospatial Technology Research Unit, Sirindhorn International Institute of Technology, 131 Moo 5,
Tiwanon Road, Bangkadi, Mueang Pathumthani, Pathumthani 12000, Thailand; geomat.bipul@siit.tu.ac.th
School of Information, Computer, and Communication Technology (ICT),
Sirindhorn International Institute of Technology, 131 Moo 5, Tiwanon Road, Bangkadi,
Mueang Pathumthani, Pathumthani 12000, Thailand
Department of Infrastructure Engineering, Faculty of Engineering and IT, The University of Melbourne,
Melbourne, VIC 3010, Australia; jagannath.aryal@unimelb.edu.au
Correspondence: teerayut@siit.tu.ac.th
Abstract: Availability of very high-resolution remote sensing images and advancement of deep
learning methods have shifted the paradigm of image classification from pixel-based and objectbased methods to deep learning-based semantic segmentation. This shift demands a structured
analysis and revision of the current status on the research domain of deep learning-based semantic
segmentation. The focus of this paper is on urban remote sensing images. We review and perform a
meta-analysis to juxtapose recent papers in terms of research problems, data source, data preparation
methods including pre-processing and augmentation techniques, training details on architectures,
backbones, frameworks, optimizers, loss functions and other hyper-parameters and performance
comparison. Our detailed review and meta-analysis show that deep learning not only outperforms
traditional methods in terms of accuracy, but also addresses several challenges previously faced.
Further, we provide future directions of research in this domain.
Keywords: deep learning; remote sensing; review; semantic segmentation; urban image classification
Review and Meta-Analysis. Remote
Sens. 2021, 13, 808. https://doi.org/
10.3390/rs13040808
1. Introduction
Academic Editor: Guoqing Li
Received: 18 January 2021
Accepted: 18 February 2021
Published: 23 February 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affiliations.
Copyright: c 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Land-use/land-cover (LULC) maps are often generated from medium resolution
satellite images like Sentinel [1] and Landsat [2]. These images are useful to classify land
cover classes like built area, residential area, vegetation surface, impervious surface, water,
etc. However, to prepare LULC maps for urban areas, objects like cars, individual buildings
and trees, etc. needs to be classified. When extracting urban features or urban land cover
information from aerial images, spatial resolution is considered being more important
than spectral resolution. In other words, a finer-resolution image pixel is more useful than
a greater number of spectral bands or narrower interval of wavelength [3]. This is the
reason why commercial satellite images and unmanned aerial vehicles (UAV) are now more
popular as they aim to increase the visibility of terrestrial objects, especially urban features,
by reducing per-pixel size. With the increase in spatial resolution, more urban objects are
now clearly visible in satellite images, and studies shifted its paradigm from spectral image
classification, pixel-based image analysis (PBIA) and object-based image analysis (OBIA)
to and most recently, pixel-level semantic segmentation. In this paper, we will analyse the
advancement of deep learning-based semantic segmentation for urban LULC.
In early PBIA methods, the pixel size was not fine enough to identify an object in an
image. As reported by Cowen et al. [4], a 4-m wide object needs a minimum of 2 by 2 m
spatial resolution (i.e., minimum of four pixels is required). A 4 m wide object however
does not locate perfectly over four pixels in a 2 m spatial resolution image. As the spatial
resolution increased, the spectral response from different small objects in the urban area
Remote Sens. 2021, 13, 808. https://doi.org/10.3390/rs13040808
https://www.mdpi.com/journal/remotesensing
Remote Sens. 2021, 13, 808
2 of 41
started to show complex patterns. This was because many objects are made of the same or
similar material (e.g., cemented rooftops, cemented parking lots and cemented sidewalks;
grasses and shrubs; etc.), and they emit a similar spectral response [5]. This is where
the traditional classification methods and per-pixel classifiers (e.g., maximum likelihood)
did not perform effectively because they used pixel-level spectral information alone as a
fundamental basis to classify remote sensing images, and ignored spatial and contextual
information [6]. Also, PBIA methods tended to produce “salt and pepper” noise after
classification. To overcome these drawbacks, OBIA methods were studied.
Unlike PBIA, OBIA starts with the generation of segmented objects at multiple levels
of scale as fundamental analysis units [7–9]. An object (aka. segment) in this approach is
a group of contiguous homogeneous pixels with similar texture, spatial and/or spectral
attributes. An image is initially divided into segments, various attributes of the segments
are then computed and some rules are built to classify features. These rules are based on
attributes such as geometry (length, area), size and textures etc., thus helping to differentiate
features in an image. e.g., differentiating lakes and rivers based on length, trees from grass
based on texture, separating building and road based on area, etc. For many years, OBIA
was considered the better approach because it prioritized contextual information of an
object [10]. Supervised and unsupervised learning classifiers methods were produced for
the classification task, and PBIA was considered not as useful as OBIA to classify very
high-resolution (VHR) images, until recently, after deep learning-based methods were
started to be explored for pixel-based semantic segmentation of VHR images. In recent
years, the use of UAVs to collect images and open-source/non-commercial software [11] to
prepare orthomosaic have opened up a promising future towards the increased use of UAVcollected images. Despite the development of a large number of PBIA and OBIA-based
methods proposed in the last two decades, these frameworks had several drawbacks [3]
and complexities due to classification errors and imbalance in classes, which limited
the widespread application. With recent end-to-end deep deep learning-based semantic
segmentation, LULC has seen rapid progress in the classification of VHR images compared
to traditional PBIA and OBIA methods.
Deep learning (DL) [12,13] architectures are modern machine learning methods
that have increased the performance and precision of computation by increasing the
number of “layers” or “depths”. DL allows fast and automatic feature extraction from
an adequately large dataset, iteratively using complex models to reduce classification
errors in regression [14]. In recent years, DL has become a core method in many researches in remote sensing, such as plant recognition [15], plant disease recognition [16,17],
weed detection [18], crop type classification [19,20], crop counting [21], natural hazard
modeling [22–24], land cover classification [25] and also uncertainty modeling [26]. DL
architectures like convolutional neural networks (CNN) [27,28] and Fully Convolutional
Network (FCN) [29] are commonly used methods for the segmentation of urban images.
The most successful state-of-the-art DL architecture is FCN [30,31]. The main idea of
this approach is to use a CNN as a powerful feature extractor while replacing the fully
connected layers with convolution ones to output spatial maps instead of classification
scores. Those maps are up-sampled to produce dense per-pixel output. This method
allows training CNN in the end to end manner for segmentation with input images of
arbitrary sizes. This approach achieved a notable enhancement in segmentation accuracy
over common methods on standard dataset like Pattern Analysis, Statistical Modelling
and Computational Learning (PASCAL) Visual Object Classes (VOC) [32]. However, the
architecture of the FCN network is often changed to solve different challenges faced during
semantic segmentation of urban satellite images.
A meta-analysis is performed in this paper to review the research papers that used
DL-based methods to answer several research questions. A few review and meta-analysis
papers have been recently published on a broader scope of the study, mostly focusing on
a brief summary of recent trends on DL in remote sensing applications [33,34]. However,
these studies do not provide detailed analysis in the domain of deep learning-based
Remote Sens. 2021, 13, 808
3 of 41
semantic segmentation of urban remote sensing images, which has been competitively
and rigorously studied in the last 5 years as shown by our meta-analysis. Several studies
improved image segmentation on urban features by designing efficient DL architectures,
however there is a lack of a complete review on the technical details of the architectures
experimented. In this paper, we present detailed and tabulated analysis and discuss how
the recent papers have tried to address the challenges faced during the shift of paradigm
to semantic segmentation such as: ignorance of spatial/contextual information by CNNs,
boundary pixel classification problems, class imbalance problem, domain-shift problem,
salt-and-pepper noise, structural stereotype, insufficient training and other limitations.
The rest of the paper is structured as follows: Section 2 presents literature on DL-based
semantic segmentation of urban satellite images; Section 3 presents the complete metaanalysis on DL-based semantic segmentation; Section 4 provides discussions; and Section 5
concludes the paper with future directions. Furthermore, Appendices A–C are presented to
display (i) the meta-analysis in structured summary, (ii) a separate performance comparison
of papers that use the most abundant datasets and (iii) a list of the available dataset for
DL-based semantic segmentation of urban images, respectively.
2. Deep Learning-Based Semantic Segmentation of Urban Remote Sensing Images
2.1. Semantic Segmentation in Remote Sensing
Semantic segmentation can be defined as the process of assigning a semantic label
(aka. class) to each coherent region of an image. This coherent region can be a pixel, a subpixel, a super-pixel, or an image patch consisting of several pixels. Per-pixel segmentation
classifies pixels by either assigning a single label to each pixel for high-resolution images,
or assigning class membership on lower-resolution images because the resolution is not
enough to contain an object [35]. Several parametric classifiers such as maximum likelihood
classifiers and non-parametric classifiers such as artificial neural networks (ANN), support
vector machine (SVM), decision tree classifiers, expert systems, etc. have been used in
the past for per-pixel segmentation [10,36–38]. While the majority of ANN-based research
was conducted for per-pixel classification in earlier days [39–41], sub-pixel classification
of impervious surfaces were also highly studied later [42–46]. In sub-pixel classification,
pixels are further classified into a fraction of the pixel size. Soon as the discovery of
ANN, various networks were studied in early 2000s: Multi-layer Perceptron (MLP) [47–50],
Adaptive Resonance Theory (ARTMAP), Self-Organizing Map (SOM) [51–53] and Hopfield
Neural Networks. Superpixel is another coherent region for semantic segmentation of
urban images [54–56]. These regions are first segmented on images using methods like
Simple Linear Iterative Clustering (SLIC) [57] or superpixelization [58] to generate coherent
regions at sub-object level.
In patch-based semantic segmentation, classifiers are trained on image patches as
a single label and make predictions in a similar fashion. A sliding window is used to
extract patches from the input images as bounding boxes around the object, which are
further forwarded to predict label [21,23]. Multi-scale inference and re-current refinements
performed significant gains on their study, and is also supported by other studies for scene
labeling [59,60]. Patch-based segmentation performs well for object detection, but for the
tasks like LULC, per-pixel methods often outperform the former method [55,61]. Some
even consider patch-based methods wasteful because of redundant operations performed
on adjacent patches [31].
As the resolution of satellite images increased, problems like loss of spatial information
and imbalance of class distribution due to many small objects visible in an image resulted
in many studies on pixel-based semantic-segmentation. Most studies in recent years have
used DL-based semantic-segmentation to accurately label each pixel to a specific class like
buildings, roads, etc. [62], which is further detailed in the next subsection.
Remote Sens. 2021, 13, 808
4 of 41
2.2. Convolutional Neural Networks (CNN)
It was in 2006 that deep neural network (DNN) was once again brought to the spotlight, long after CNN was first proposed in [27]. DNN models like deep belief network
(DBN) [63], autoencoders (AEs) [64], deep boltzmann machines (DBMs) [65], and stacked
denoising autoencoders (SDAEs) [66] improved efficiency when training with large-scale
samples. In 2012, AlexNet was proposed and achieved state-of-the-art on ImageNet classification benchmark [67]. After that, the networks started to take a deeper form in ZFNet [68],
VGGNet (VGG16 and VGG19) [69] and GoogleNet [70].
The use of blocks consisting of separate architecture on its own and ensemble of the
outputs using softmax layer is a general framework for CNN. Steps parametric rectified linear unit (PReLU) [71], dropout [72], batch normalization (BN) [73], and network optimizers
like stochastic gradient descent (SGD) [74] and Adam Optimizer [75] are widely used to
accelerate CNN training process. A standard structure of CNN consists of an input layer,
convolutional layers, pooling layers, a fully connected layer and last the output soft-max
layer. The output of each filter for a convolutional layer is calculated as
yln = f (
∑
m ∈ Mn
l −1 l
l
xm
.wm,n + bm
)
(1)
l
l are the weights and bias term of mth filter of lth layer and f ( x ) denotes
where wm,n
and bm
nonlinear activation function. A most commonly used activation function in CNN is
rectified linear unit (ReLU) [71]. The weights and bias of each filter are passed to every
location of the input feature map such that a model learns from the regrets of having
applied the filter overlaid over any location of feature maps. A pooling layer like averagepooling and max-pooling are commonly used to provide shift-invariance by reducing the
resolution of feature maps. The consecutive use of multiple convolutional and pooling
layers produces smaller and abstract 2D feature maps, which are then connected by a
sequence of fully connected layers that transform them into 1D features. The last softmax
layer makes the final predictions.
CNNs are effective for object detection, scene-wise classification and feature extraction.
However for pixel-based semantic segmentation, the use of pooling layers diminishes many
features and the resulting feature maps and predictions cannot achieve the required PBIA.
In the next section, we will talk about how FCN has improved the drawbacks of CNN.
2.3. Fully Convolutional Network (FCN)
When Long et al. [29] first proposed FCN in 2015, it achieved state-of-the-art semantic
segmentation. What made it more efficient than CNN is that fully connected layers in
a network for aerial image classification purposes can be considered as convolutions
with kernels that cover their entire input region. Mou et al. [76] have considered this as
equivalent to evaluating classification network on patches with overlapping regions. As the
computation runs across images by overlapping the regions, FCN achieves better efficiency.
And due to the presence of a pooling layer, feature maps obtained from FCN are then
upsampled. CNN’s pooling layer aggregates information and extracts spatial-invariant
features that are crucial for pixel-level semantic segmentation. A general framework of
FCN therefore consists of two parts: encoders and decoders. Encoders are inspired from [66]
and are similar to CNNs that extracts feature maps, and decoders transform these features
into dense label maps, whose size is the same as the input image.
The first proposed FCNs were FCN-8s, FCN-16s and FCN-32s [29]. Instead of using
simple bilinear interpolation, they used transpose convolutional layers to upsample the
deep feature maps into labeling results. This improved the performance of classification
resulting in finer predictions, but new challenges were born: (i) transpose convolutional
layers were computationally expensive because of its hunger towards memory [77], (ii)
they were difficult to train and (iii) the resulted classification was poor around the object’s
boundary [76]. Several studies were thereafter performed to overcome these drawbacks.
Chen et al. (2017) [78] introduced atrous convolutions in FCN, removing the max pooling
Remote Sens. 2021, 13, 808
5 of 41
layers, to expand the field of view with fewer parameters. To improve the classification
errors around the boundary of label maps, they used conditional random fields (CRF) after
FCNs. Ji et al. [79] introduced atrous convolutions on the first two layers of decoding steps
to enlarge sight-of-view and integrate semantic information of large buildings, which are
aggregated with multi-scale aggregation strategy. The last feature maps of each layer make
predictions that are concatenated in the final prediction. Chen et al. (2019) [80] used FCN
called DeepLabv3 [81] with Resnet backbone and atrous convolution layers, augmented
Atrous Spatial Pyramid Pool (AASPP) with 4 dilated convolutional layers for multi-scale
information and fusion layers to concatenate and merge the feature maps.
A mean-field approximate inference was used for CRF by Zheng et al. [82] with
Gaussian pairwise potentials as recurrent neural fields (RNNs) to train an end-to-end
CRF-as-RNN with unary energy of CRF. Lin et al. [83] later predicted both unary and
pairwise energy using a deep structured model, which achieved significant performance
on PASCAL VOC 2012 dataset. Most recently, Liu et al. (2019) [84] combined two pixellevel predictions—a pretrained FCN-8s fine-tuned and re-trained on 3-band image and
probabilistic classifier called multinomial logistic regression (MLR) trained on LiDAR
data—as unary potential modeled as CRF. They also used segments obtained from a
gradient-based segmentation algorithm (GSEG) into a higher-order CRF called P N Potts’
model to resolve ambiguities by exploiting spatial-contextual information, and finally use
graph-cut to efficiently infer the final semantic labeling for their proposed higher-order CRF
(HCRF) framework. Apart from integrating CRF with FCNs, other network architectures
were also designed for better classification, such as ResNet [85] based FCN [86,87].
Pyramid pooling module was proposed by Zhao et al. [88] in 2017, which was applied
onto ResNet-based architecture to obtain clues on semantic categories distribution by
parsing global information using large kernel pooling layers. Yang et al. [62] proposed
end-to-end DL architecture to perform pixel-level understanding of high spatial resolution
RS images, based on both local and global contextual information. They used the pyramid
pooling module to collect multi-level global information. Low-dimension context representations were then upsampled by bilinear interpolation, thus obtaining representations
with the same size as the original feature map. Finally, they concatenated multi-level
global features and the last-layer convolutional feature map for the pixel-wise prediction. Yu et al. [89] proposed a pyramid scene parsing network (PSPNet) by incorporating
cheaper network building blocks to extract multi-scale features for semantic segmentation.
PSPNet comprised of two parts: (i) a CNN based on ResNet101-v2 to extract features by
encoding input images into feature maps and (ii) a pyramid pooling module inspired
from SPPNet [90] to extract features at multiple scales and upsample the feature maps to
learn global contextual information by concatenating the multi-scale features. The final
segmentation result is achieved by convolution operation on the concatenated feature maps.
Later, Chen et al. (2018) [91] compared the use of ResNet-101 as the base structure for
three baseline methods applied on roof segmentation: (i) feature pyramid network (FPN)
(ii) FPN with multi-scale feature fusion (MSFF) and (iii) PSPNet; to observe the highest
performance from PSPNet.
Different approaches of data fusion such as multi-modal, multi-scale and multikernel data fusion have also been practiced to improve the performance of FCN on RS
images [56,80,91–98]. In a distinctive approach, some studies focused on the symmetry of
the encoder-decoder structure, which is discussed in the next subsection.
2.4. Symmetrical FCNs with Skip Connections
Semantic segmentation requires both contextual information (object-level information) as well as low-level pixel data. Besides the pixel-level information, how to utilize
contextual information is a key to formulate better semantic labeling. Contextual relationships therefore provide valuable information from neighborhood objects. In a symmetrical
encoder-decoder FCN architecture, upper layers of encoders encode object-level information and lower layers capture rich spatial information. This information can become
Remote Sens. 2021, 13, 808
6 of 41
invariant to factors such as pose and appearance. And with skip connections, such networks can use both lower and higher-level information for finer predictions. This concept
also helped the original U-Net architecture of Ronneberger et al. (2015) [99], in which
each layer produces independent predictions. Multiple layers of convolutions, ReLU and
max pooling combine the outputs of lower layers and higher layers to generate the final
output. BN and ReLU are heavily applied to accelerate training and avoid vanishing gradient problems. These layers when trained using model optimizers such as SGD and back
propagation (BP) of error, U-Net learns patterns. Finally, a softmax function generates the
segmentation results. The model soon became widely used for medical image segmentation
and also some studies can be found in urban images segmentation [100–102]. Yi et al. [103]
proposed DeepResUnet for effective urban building segmentation at pixel-scale from VHR
imagery. Their modification to the original U-Net included ResBlocks of two 3 × 3 and
one 1 × 1 convolutional layers and ReLU activation functions. Compared to the original
U-Net, DeepResUnet achieved better performance with fewer parameters, but with longer
inference time. Yue et al. [104] proposed TreeUNet with DeepUNet [105] and Tree-CNN
block to improve the differentiation of easily confused multi-classes. Tree-based CNN was
later also used by Robinson et al. [106] to improve segmentation results with decision trees.
Liu et al. (2020) [107] used two branches of modified U-Net to align feature distributions
in the image domain and wavelet domain simultaneously, which were later merged to
predict the final classification results. Diakogiannis et al. [108] proposed ResUnet-a, which
uses U-Net’s encoder-decoder backbone, in combination with residual connections, atrous
convolutions, pyramid scene parsing pooling and multi-tasking inference. SiameseDenseUNet of Dong et al. (2020) [109] used two similar parallel DenseU-Nets [110] to alleviate
boundary detection and class imbalance problems.
Another widely used symmetrical FCN is SegNet [111], which improved boundary
delineation with minimum parameters by reusing pooling indices. Segnet is widely used
in the domain of urban feature extraction, which can be seen later in Section 3.2.4. Many
modified SegNet or used it as a backbone architecture [92,112–115] and some composed a
new network using multiple FCNs like SegNet and U-Net [116,117]. Some used a network
with multiple encoders of SegNet [113].
Several other symmetrical FCN networks have been proposed for urban image segmentation: DeconvNet [118], gated semantic segmentation network (GSN) [87], a network
with boundaries obtained from edge detection [112] and SharpMask [119]. To boost the
speed of segmentation compared to their original network called DeepMask [120], SharpMask produced features instead of independent predictions at all layers. Kemker et al. [121]
adapted SharpMask and compared the results to RefineNet with ResNet-50 architecture,
to observe similar performance results on urban feature classification. RefineNet [122] is
similar to U-Net, but introduces several residual convolutional blocks inside the encoder
and decoder. DeepLab networks—DeepLabv3 [81] and DeepLabv3+ [123] are other commonly used FCNs [104,124]. Chen et al. (2018) [125] introduced two semantic segmentation
frameworks: Symmetrical normal shortcut fully convolutional networks (SNFCN) and
Symmetrical dense-shortcut fully convolutional networks (SDFCN), both of which contain
deep FCN with shortcut blocks. In order to properly fuse multi-level convolutional feature
maps for semantic segmentation, Mou et al. [76] proposed a novel bidirectional network
called “recurrent network in fully convolutional network” (RiFCN). Li et al. (2019) [126]
used two separate sub-networks to make a Y-shaped network, whose final predicted feature
maps are concatenated and merged using convolutional layers. The first sub-network is an
FCN with skip-connections and the second sub-network is a network of 13 convolutional
layers without downsampling layers to avoid loss of information. Sun et al. [115] proposed
a novel residual architecture called ResegNet to solve the problem of “structural stereotype”
and “insufficient learnings” that encoder-decoder architectures face.
Remote Sens. 2021, 13, 808
7 of 41
2.5. Generative Adversarial Networks (GAN)
Most CNN based methods perform well when they are tested on the images that are
similar to the train images, but fail when the domain shifts to a new domain. To address
this problem and reduce the domain shift impact caused by imaging sensors, resolution and
class representation, Benjdira et al. [127] modified the original GAN [128]. GAN consists
of two models: generator and discriminator. A generator is trained to generate fake data
to fool the discriminator, whereas discriminator is trained to differentiate between fake
and real data. In Benjdira’s GAN, the generator follows a symmetrical encoder-decoder
architecture and encoder consists of 4 convolutional layers for down-sampling and Leaky
ReLU as an activation function. The output features extracted are passed to the decoder
to rebuild the original feature vector. Decoder consists of four convolutional layers for
up-sampling, standard ReLU as activation function and ‘dropout’ to reduce over-fitting.
Instead of batch normalization, they used instance normalization. On the other hand, the
discriminator consists of 5 convolutional layers that encode the generated image into a
feature vector of a size of 256 × 256. Then, the sigmoid activation function in the last
layer converts this feature vector into a binary output. Similar to the generator, Leaky
ReLU activation function and instance normalization are used. The overall method can
be summarized into the following steps: (i) Train segmentation model on source dataset.
(ii) Train GAN to efficiently translate images from the source domain to the target domain.
(iii) Convert the source dataset to the target domain using GAN, producing a new dataset
with conserved structures of the source dataset but mimicking the global characteristics
of the target dataset. (iv) Fine-tune the already trained segmentation model of step 1
with translated dataset associated with the source labels. Lin et al. [129] proposed an
unsupervised model called multiple-layer feature-matching GAN (MARTA GANs) to learn
a representation using only unlabeled data to increase the label dataset and Zhan et al. [130]
designed a semi-supervised framework for hyper-spectral image data based on a 1-D GAN
(HSGAN). Liu et al. (2020) [107] proposed a bispace alignment network called BSANet for
domain adaptation (DA) and for automated labeling. BSANet uses two discriminators of
modified U-Nets to minimize the discrepancy between the source and target domains. The
use of GAN to solve the domain shift problems is limited and is expected to be studied
more in the upcoming research works.
2.6. Transfer Learning
In order to increase the performance of a DL-model with fewer training samples and
less computational power, some studies have used transfer learning [131]. Transfer learning
allows the transfer of knowledge gained from solving one problem to be used for similar
problems. The method has been quite popular in the studies that lack enough training
images and labels [14,132,133]. Panboonyuen et al. [96] proposed to segment urban features
in RS images using a global convolutional network (GCN) with channel attention blocks
and domain specific transfer learning to transfer between learning obtained by training on
VHR image to medium resolution images. Du et al. (2019) [134] performed semantic segmentation of crop (vegetation) area on RGB aerial images of Worldview-2. A DeepLabv3+
model pre-trained on ImageNet dataset was retrained on their dataset with image-GT
(ground truth) label pairs. Compared to modern methods like U-Net, PSPNet, SegNet and
DeepLabv2, and traditional machine learning methods like maximum likelihood (ML),
SVM, and random forest (RF), their re-trained DeepLabv3+ model obtained the highest
performance. Wurm et al. [135] segmented slum areas on RS images, in which they use
transfer learning between models trained on different resolution images. Transfer Learning
was done in two groups: (i) FCN based on VGG19 architecture that was pre-trained on
ImageNet dataset transfers learnings and weights to three FCNs, trained on images collected from QuickBird (FCN-QB), Sentinel-2 (FCN-S2) and TerraSAR-X (FCN-TX). (ii) The
learning of FCN-QB of the first group of the experiment was again transferred to FCN-S2
and FCN-TX. Some transfers produced better performance than others. Some studies use
transfer learning to test their model in different ablations [79].
Remote Sens. 2021, 13, 808
8 of 41
3. Meta-Analysis
3.1. Methods and Data for Review
Our methodology for bibliographical analysis in the domain under study follows
three steps: (a) collection of related works (b) thorough study and (c) detailed metaanalysis. For the collection of related works, title search was performed in IEEE Xplore,
ScienceDirect and Google Scholar using the search query [[“semantic segmentation”]
OR [“pixel-level classification”] AND [“urban feature classification”] AND [“satellite
imagery”]] on 21 December 2020. Also, more papers were downloaded from International Society for Photogrammetry and Remote Sensing (ISPRS)’s 2D Semantic Labeling
Contest’s leaderboard for Vaihingen (Click the http://www2.isprs.org/commissions/
comm2/wg4/vaihingen-2d-semantic-labeling-contest.html (accessed on 21 December
2020) to go to ISPRS’s leaderboard for Vaihingen dataset.) and Potsdam (Click the http:
//www2.isprs.org/commissions/comm2/wg4/potsdam-2d-semantic-labeling.html (accessed on 21 December 2020) to go to ISPRS’s leaderboard for Potsdam dataset). A total of
122 papers related to pixel-based classification of urban features were first downloaded.
Secondly, with thorough study, the papers that adapted object-based methods, traditional machine learning methods, performed vehicle segmentation and the studies that
used terrestrial scene images, multi-view stereo (MVS) and Lidar for data collection were
filtered out. Also, the technical reports that were not peer-reviewed papers were filtered
out to select 71 papers that studied semantic segmentation of VHR satellite images for
urban feature classification using DL-based methods. The 71 papers come from different
countries. The countries of the first author’s affiliation are shown in Figure 1.
Figure 1. Overview of first author’s affiliation grouped by countries and continents.
In the third step, the 71 papers filtered in the previous step were studied carefully
to look for the trends and contributions of the 71 papers on the following five major
research questions:
1.
2.
3.
4.
What are the study targets?
What are the data sources and datasets used?
How is the training and testing data prepared for deep learning? This question looks
for pre-processing, preparations and augmentation methods if used.
What are the training details? This question looks for architecture, backbone, framework, optimizer, loss function and hyper-parameters that are mentioned by the papers?
Remote Sens. 2021, 13, 808
9 of 41
5.
What is the overall performance of the method? This question looks for the performance metrics used, methods used for comparison, highest performance in each area
of study and performance gains over previous methods.
3.2. Dissection and Overview of Research Questions
3.2.1. The Study Targets
The 71 papers can be divided into 4 study targets as building/roof segmentation, road
segmentation, urban feature segmentation and urban land cover classification. Some papers
intend multiple targets. The distribution is shown in Figure 2, which shows fewer studies on
urban land cover [96,100,134,136–139]. With the availability of VHR images, smaller urban
features have been segmented in the majority of papers. The papers that used dataset such
as ISPRS Vaihingen 2D Semantic Labeling dataset, ISPRS Potsdam 2D Semantic Labeling
dataset and IEEE GRSS (Geoscience and Remote Sensing Society) Data Fusion Contest
(of Zeebrugge, Belgium) are dedicated to the improvement of semantic segmentation of
urban features from traditionally interesting “urban area” class to Impervious surfaces,
Building, Low vegetation, Tree, Car, Water, Boats and Clutter/background. Some have
segmented urban features as small as sidewalk, motorcycles, traffic signs, pedestrians,
picnic table, orange pad, buoy, rocks, sports courts, etc. [106,121,140,141]. Some classified
buildings to their utilities [102,113], smaller features inside roads [142] and slum area [135].
Besides the segmentation of features, the studies use different approaches to improve
semantic segmentation of urban features, which are shown in Table 1. The coherent regions
used for segmentation are pixel, patch and super-pixel for 66, 4 and 2 papers respectively,
including [143], which uses super-pixels to enhance pixel-based segmentation.
Figure 2. Evolution of deep learning-based semantic segmentation of urban remote sensing images. (a) The distribution of
number of publications into four study targets. (b) The number of publications by year.
Table 1. Overview of the research problems the papers were motivated to solve in order to improve the results of semantic
segmentation.
Research Problem
Count
Reference
Use better fusion technique
Use contextual information
Use auxiliary data
Use skip connections
Use transfer learning
Minimize pre/post processing
Improve labeling dataset
Solve class imbalance problem
Improve boundary detection
Remove downsampling
Reduce model size
Minimize training parameters
Solve structural stereotype and insufficient training
Improve with decision tree
Minimize domain shift problem
Improve using pre-processing filters
18
8
2
5
4
3
2
5
9
1
1
1
1
2
2
1
[76,79,80,84,87,92,95,97,98,126,144–151]
[62,93,151–156]
[101,140]
[94,103,125,149,157]
[96,134,135,148]
[89,158,159]
[160,161]
[108–110,154,162]
[55,109,112,114,124,138,143,163,164]
[31]
[165]
[155]
[115]
[104,106]
[107,127]
[139]
Remote Sens. 2021, 13, 808
10 of 41
3.2.2. Data Sources
Among the 71 papers, most of them used a publicly available dataset consisting of
a large number of images and labels, and some tested their model on multiple of these
datasets. The most commonly used datasets are ISPRS Vaihingen 2D Semantic Labeling
dataset [166] (38 papers) and ISPRS Potsdam 2D Semantic Labeling dataset [167] (27 papers) and IEEE GRSS Data Fusion Contest [168] (5 papers) collected their images using
UAV. Other than UAV-based images, satellite images were obtained from RADARSAT2 [136], Worldview-2 [102,134,156], Worldview-3 [149], Landsat-8 [96], SPOT [137,155],
Gaofen [138,169], Quickbird, Sentinel-1 and 2 [147], Sentinel-2 and TerraSAR-X [135], PlanetScope (Dove constellations) [164]. A particular research collected data from a plane [140].
Some also used web-services such as Google Maps/Earth [140,161], World [101,140], Bing
Maps [113] and Linz Data Service [103] often along with satellite images to improve
segmentation results. Terrestrial data was also used for similar purposes [113]. For the
comparison purpose or to assist the DL model, some used Lidar data. Lidar data were
mostly included in the widely used public dataset mentioned before, and some used different ones. For VHR images, some prepared their own UAV-based dataset [121]. Some
used Inria Aerial Image Labeling Dataset [76,114,117] for urban LULC. Other than urban LULC, some specific dataset dedicated for building/roof segmentation are Aerial
Imagery for Roof Segmentation [91], SpaceNet building dataset [101], UK-based building
dataset [160], WHU Building dataset [79]; for roads are AerialLanes18 dataset [142]; and
for both roads and buildings is Massachusetts Building and Road Dataset (Mnih, 2013
dataset) [116,126,151–153,158,159]. Other dataset for building and roads are also used
in [152,153]. More datasets that were found along with the details on resolution and websource are provided in Appendix C. The spatial coverage of the 71 papers includes 46 local
domain studies and 25 global domain studies. The local domain study means the dataset
was collected within a single country and global domain study means the dataset was
collected in more than one country.
3.2.3. Data Preparation
Here, we summarize the distribution of pre-processing, preparation and augmentation methods used. Pre-processing methods change the characteristics of images on pixel
or spectral level. Some of these methods include image processing methods, image normalization by mean value subtraction on images [31,96,158,161], k-means clustering [134],
relative radiometric calibration strategy [79], super-pixel segmentation [93], Lidar to digital
surface model (DSM) [163], satellite image correction and pan-sharpening [102,164,169],
normalized DSM (nDSM) and calculation of vegetation index such as normalized difference vegetation index (NDVI). Twenty-two papers performed such pre-processing
methods. The most commonly performed methods calculation of DSM, nDSM, LiDAR
data and NDVI [76,84,92,94,146,150]. Some used a stack of multiple channels of images [95,101,144,160]. Some used filters like unsharp mask filter, median filter, linear
contrast filter [106] and Wiener filter [139]. Most papers that used datasets like ISPRS or
IEEE did not perform pre-processing on images, because the images were already in a
ready-to-use condition, and consisted of annotated labels too.
Data preparation includes methods used to prepare labels and images for train, test
and validation dataset. The ratio of train-test/validation data is commonly 80-20 or 90-10.
Image tiling is the most commonly used method and tile size was mostly 256 × 256 or
512 × 512 pixels. 18 papers overlapped the training tiles to their neighbors, among which
most of them used 50% overlap. Yue et al. [104] used 2D Gaussian function to calculate
the overlap. The distribution of these methods is shown in Figure 3. For the papers that
used datasets like ISPRS and IEEE, the labels were already prepared. Others prepared
labels manually [76,103,134,142] or used some traditional image segmenting methods to
prepare labels [135,160]. The majority of papers (42 papers) have mentioned the use of
image augmentation techniques to increase the number of training dataset or to increase
the performance of overall learning procedure. The distribution of these methods is shown
Remote Sens. 2021, 13, 808
11 of 41
in Figure 3. Some also evaluated the increment in performance due to augmentation [95].
More details on this section can be found in Appendix A.
Figure 3. Distribution of image preparation and augmentation methods used.
3.2.4. Training Details
Training details include network architectures, frameworks, optimizers, loss function,
hyper-parameters like learning rate, weight decay, momentum, epochs and iterations, use
of dropout to handle over-fitting, and the hardware used by the papers. The architectures
used are shown in Figure 4. The most commonly used architectures are CNN, SegNet, FCN,
U-Net and FPN. More details can be found in Appendix A. Among the architectures used,
SegNet, few FCNs, FPN, U-Net, DeepLabv3+, FSN, ResegNet, DeepUNet, ResUNet-a,
DenseU-Net, FuseNet and ReuseNet are encoder-decoder structured architectures, mostly
using skip connections (aka. skip branches). The convolutional backbones employed by the
papers are shown in Figure 5. Among the papers that mentioned the name of the backbone
used, the most commonly used ones are ResNet and Visual Geometry Group (VGG). The
most commonly used ResNet backbone is ResNet-101, which helps in the reduction of
vanishing gradient problem of deep learning. Other than convolutional backbones, some
papers used DL architecture like SegNet and U-Net as a part of their model, making them
the backbone of their new architecture.
Talking about frameworks that are used to run or wrap the network architecture,
the most commonly used were Caffe, Tensorflow, Keras and Torch. The distribution is
shown in Figure 6 and details in Appendix A. Most commonly used network optimizers
that optimize the network and keep off from model over-fitting are SGD including SGD
with momentum or other hyper-parameters like learning rate. Other algorithms were
Adam, Nadam (aka. Nesterov Adam) [76,121] and Adadelta [125]. Some used dropout
functions to stop the over-fitting with common values of 0.5. To initialize the weights
in the network, some used a special algorithm called Glorot Uniform Initializer [76,103]
and Xavier algorithm [107,147]. The use of optimizers, dropout values and initializers are
shown in Figure 7.
Remote Sens. 2021, 13, 808
12 of 41
Figure 4. Overview of the deep learning (DL) architectures employed. The encoder-decoder models like Fully Convolutional
Network (FCN), U-Net, SegNet, DeepLab, Hourglass and others are the most commonly employed ones. Many papers
have employed more than one of these architectures to later fuse the output feature maps.
Figure 5. Overview of the convolutional backbones employed. Out of the papers that mentioned the use of backbones, the
most commonly employed are ResNet and Visual Geometry Group (VGG). As many papers used multiple DL architectures,
more than one backbone is used by those papers.
Remote Sens. 2021, 13, 808
13 of 41
Figure 6. Distribution of frameworks used to wrap the deep learning models.
Figure 7. Distribution of optimizers used to fit the deep learning models.
The loss functions that are used to evaluate how well specific algorithm models the
given data include commonly used cross-entropy (CE) loss function (35 papers). The
variety of CE loss used includes binary CE with semantic encoding loss [154], CE with
median frequency balancing (MFB) [61,94,109,110,162], normalized CE [151], categorical
CE function called logloss [125] and sparse softmax CE [135,148]. Some modified the
CE loss [142,163] and compared the use of CE with MFB and CE with focal loss function
weighted by MFB [110]. Other loss functions include auxiliary loss [62,89], adversarial
loss [127], regularized logistic regression [144], multinomial logistic loss [146,147,161],
validation loss from Keras [121], softmax [104,106,134,138], bidirectional loss [76], moving average [169], negative log likelihood loss (NLLLoss) [164] and Dice coefficient and
Tanimoto coefficient [108]. The distribution of loss function is shown in Figure 8.
Remote Sens. 2021, 13, 808
14 of 41
Figure 8. Overview of loss functions used to evaluate the deep learning models. The most commonly used loss in Cross
Entropy (CE) loss.
Hyper-parameters like learning rate, weight decay, momentum, epoch, batch size,
number of iterations and steps are set up carefully as they depend upon the architecture,
number of layers used and computer hardware specifications. The learning rate is either
fixed or is decreasing with some decay rate of weight or momentum after a certain number
of steps, iterations or epochs; ranging from 10−1 to 10−10 . Most studies used graphics
processing unit (GPU) units to train models faster (Figure 9). Most recent and powerful
GPUs are used by the most recent papers, and all GPUs are from Nvidia.
Figure 9. Overview of GPUs used to run the deep learning models. All GPUs are from Nvidia.
3.2.5. Performance Comparison
A wide range of metrics are used to evaluate the methods/models as seen in Figure 10
For simplicity, we refer to “DL performance” as the DL model’s performance score obtained
from the metrics used in the papers. Most papers use multiple metrics to compare different
combinations of same the model or to compare to other base models. The use of different
experimental dataset and performance metrics makes performance comparison between
different papers extremely difficult and often carries conviction. Therefore, we summarize
the findings in the most meaningful ways by only comparing overall accuracy (OA) and F1
score among the papers that segmented multiple classes, and do not compare other metrics
if these already exist OA. And some used k-fold cross validation in case the train and test
dataset was small [135,137,146].
Remote Sens. 2021, 13, 808
15 of 41
Figure 10. Overview of performance metrics used to evaluate the results of deep learning models.
The papers that used common dataset have used similar metrics. Among the 32 papers
that used ISPRS Vaihingen dataset and OA as the metric, OA ranged from 0.86 to 0.92
for the study target of “urban feature segmentation” with the highest value of 0.92 by
superpixel-enhanced Deep Neural Forest (SDNF) [143]. For the ones that used F1 score, the
highest of 0.92 was obtained by the ScasNet model (CASIA-2 in ISPRS leaderboard) [151].
Among the papers that used pixel accuracy, the highest of 0.87 was obtained. Similarly for
the ISPRS Potsdam dataset, among the 23 papers that used the dataset for “urban feature
segmentation”, OA ranged from 0.85 to 0.93 with the highest of 0.93 from 2 papers [143,147].
For the ones that used F1 score, the highest was 0.93 [98,108,151] and for the papers that
used pixel accuracy, 0.87 was obtained. Potsdam data was also used for other study targets.
The highest OA of 0.96 was obtained for “building/roof segmentation” [160].
For the dataset of IEEE GRSS, the highest OA of 0.90 was for “urban feature segmentation” [155] and for the Massachusetts Building dataset and Road Dataset, the highest
was 0.97 [159] and 0.94 [153] respectively. For the other datasets used, the highest OA for
each study targets are: 0.99 for “building/roof detection” [160], 0.96 for “urban feature
segmentation” [76,156] and 0.99 for “urban land cover classification” [100]. For “Road
segmentation”, 90% of roads were correctly segmented [159] and the highest F1 score of
0.94 was achieved [152].
It has to be noted that the objective of some papers [107,127] was not focused on
producing the highest metric in segmentation, but was centered to improve the segmentation when the train and test datasets are from different domains. Benjdira’s method of
GAN [127] improved OA from 35% to 52% when passing from Potsdam (source domain)
to Vaihingen (target domain). Also, it improved the average segmentation accuracy of
classes inverted due to sensor variation from 14% to 61%.
Out of all 71 papers, 62 compared their DL performance to some base models. Out
of these, most (54 papers) compared to base models using modern DL architectures and
14 papers compared their results to traditional models. The traditional models include
SVM [138], Extended Morphological Profiles (EMP) [156], conventional neural networks
(NN) [152,158,159,170], Stochastic Expectation-Maximization (SEM) [136], Random Forest
(RF) [55,102], Unary Potential Classifier [140], CRF, Simple Linear Iterative Clustering
(SLIC) [137], maximum likelihood (ML) [134], k-nearest neighbor (kNN), Multi-layer
Perceptron (MLP), multi-scale independent component analysis (MICA), stacked convolutional autoencoders (SCAE) [121], etc.
The trend shows that the more recent the research, the less are they being compared to
traditional methods that do not use DL. Out of the 54 papers that compared their method
to the modern DL-based methods, 38 papers used ISPRS’s Vaihingen or Potsdam data for
comparison and 16 papers used other datasets. Details can be found out about the 38 papers
that used the ISPRS dataset in Section 4.3 and Appendix B. Among the papers that compare
Remote Sens. 2021, 13, 808
16 of 41
their results to other datasets, some of the modern base models used for comparison are
Cascade CNN [137]; Saito et al. (2015) [158]’s CNN [152,153]; DeepLab, Unet, FCN-8s,
DeepLabv3 and DeepLabv3+ [142]; U-Net, PSPNet, SegNet and DeepLabv2 [134]; FCN-8s,
SegNet, DeconvNet, U-Net, ResUNet and DeepUNet [103]; C-Unet, U-net, FCN-8s and
2-scale FCN [145]’s 2-scale FCN [79]. The recent papers tend to compare their method to
recent other DL-based methods for state-of-the-art comparison.
Out of all, most papers (65 papers) obtained less than 20% DL performance, including
55 papers less than 10%, 49 papers less than 6%, 36 papers less than 3% and 25 papers
less than 1%. Among the 54 papers that compared their method to modern DL-based
methods, majority (41 papers) show improvement in DL performance by less than 6%,
and 8 papers shows more than 6% improvement [62,84,101,114,163]. Similarly, among
the 14 papers that compared to traditional methods, 7 papers observed less than 6% DL
performance and the remaining observed more than 6%. [55,92,136,137,140,156] shows 6 to
20% increment in performance over methods like HOG+SVM, Discriminatively trained
Model Mixture (DTMM), SLIC with feature extractor called BIC and SVM with Radial
Basis Kernel (RBF-SVM), Cascade CNN, diversity-based fusion framework (DFF), SVM,
EMP, pixel-based CRF, RF, SEM and conventional NNs and classifiers. Some papers show
20 to 50% improvement over traditional methods ML, SVM, RF, kNN, MLP, MICA, SCAE
and multi-resolution segmentation (MRS) + SCAE [121,134]. These numbers are the DL
performance of the overall method, and comparison between each class is not shown.
However, they can help us understand how significant DL-based methods are when
compared to traditional methods.
The readers are suggested to go through Appendix A, a tabulated analysis that is
ordered by the year of publication. Now in the next section, we will discuss how traditional
methods helped DL-based methods, improvements shown by DL, improvement boosted
by dataset challenges and research problems addressed by DL on semantic segmentation.
4. Discussion
Our meta-analysis shows that DL-based methods have shown significant performance
gains in the majority of literature reviewed. Among all papers, 89% papers compared to
some traditional, modern or both type of base model and 93% compared to a different
version/ablations or combination of their own model including some comparison among
different datasets. Some recent papers of 2019 and 2020 include an ablation study to discuss
the performance of their method. Most works are motivated to improve segmentation with
methods for better fusion, contextual information, auxiliary data, skip connections and
transfer learning. Some wanted to minimize pre-processing and post-processing, improve
labeling dataset, solve the class imbalance problem and improve the boundary prediction. Compared to the traditional methods, all related works have shown improvement
regarding these problems. Thanks to labeling contests, 71% of papers have used a common
dataset of ISPRS 2D semantic labeling dataset including 38 that used Vaihingen and 27 that
used Potsdam. Also other data sources were mentioned in Section 3.2.2. If the papers did
not use these common datasets, it would have been nearly impossible to compare their
work. Further, we will now discuss the improvements achieved with the help of traditional
methods (Section 4.1), improvements shown by DL (Section 4.2) and dataset challenges
(Section 4.3) and research problems addressed by DL (Section 4.4).
4.1. Helping Hands for DL Models
DL models offered superior performance compared to SVM, conventional NNs, RF,
CRF, HOG, ML, SLIC and other supervised and unsupervised classification methods.
However, some DL-based methods have also included a few of these traditional methods to support their model either on segmentation before training, or as pre-processing
or as a post-processing method. CRF has been commonly used with CNN and FCN to
exploit contextual information. Many papers applied CRF as a post-processing on their
model [31,55,144,169]. Some compared their model to the methods using CRF [92,145]. In
Remote Sens. 2021, 13, 808
17 of 41
2015, Paisitkriangkrai et al. [144] used CRF on the combined probabilities of a CNN and an
RF classifier. Saito et al. (2015) [158] compared the conventional NN’s of Mnih et al. [170],
out of which, NN with CRF had shown better segmentation of building. Saito et al.
(2016) [159] in their next study omitted the use of data augmentation that Mnih et al.’s networks with CRF needed. Sherrah et al. [31] observed a slight improvement in results when
they used RF and CRF on their FCN without downsampling. Marmanis et al. (2016) [157]
observed a slight gain in OA while using CRF with his FCN. To quantify the influence of
post-processing, they used class likelihoods predicted by their ensemble of FCN, as input
to a fully connected CRF (FCRF). Later, Zhao et al. [156] used CRF to capture contextual
information about semantic segments and refine classification maps. When comparing
pixel-based CRF vs pixel-based CNN, they pointed out that CRF was better than CNN because CRF overcomes the “salt-and-pepper” noise effects. However, CRF generally requires
a substantial amount of calculation time and overlooks the contextual information between
different objects. Therefore they combined CNN with CRF to observe up to 3% better
OA. Liu et al. (2017) [150] used higher-order CRF to combine two predictions from a FCN
trained on infrared-red-green (IR-R-G) images of Potsdam and a linear classifier trained on
LiDAR. Their fusion using higher order CRF helped them resolve fusion ambiguities and
observed 1 to 3% better OA.
As most methods combined CNN with strategies for spatial regularization such
as CRF, Volpi et al. (2018) [163] proposed a method to learn evidence in the form of
semantic class likelihoods, semantic boundaries across classes and shallow-to-deep visual
features, each one modeled by a multi-task CNN. They used CRF to extract boundaries
with base parameters of sensitivity to color contrast, flat graph and segmentation tree,
producing better OA. Pan et al. (2018) [148] used fully connected CRFs as unary potential
on the outputs of softmax layer (heat maps) and as pairwise potential on CIR images,
to observe a slight improvement in results on their encoder-decoder architecture called
Fine segmentation Network (FSN). Liu et al. (2019) [84] proposed a decision-level multisensor fusion technique for semantic labeling of the VHR RGB imagery and LiDAR data.
In their study, they fused segmented outputs obtained from multiple classifiers such
as FCN, probabilistic classifier and unary potential modeled as CRF, each trained on a
different multi-modal data such as 3-band images, LiDAR and NDVI, using HCRF. When
tested on Zeebruges dataset, their method produced 3 to 19% better OA than methods
like SVM, AlexNet and FCN-8s. Du et al. (2020) [124] obtained two initial probabilistic
labeling predictions using a DeepLabv3+ network on spectral image and an RF classifier
on hand-crafted features, which they integrated by Dempster-Shafer (D-S) evidence theory
to be fed into an object-constrained higher-order CRF framework to estimate the final
semantic labeling results with the consideration of the spatial contextual information.
Some other post-processing methods also include denoise and smoothing [100] and Otsu’s
thresholding [139]. Li et al. (2019) [101] adjusted probability threshold of building pixel
and adjusted possible threshold of minimum polygon size of buildings to minimize error
and noise. Mi et al. (2020) [143] obtained one of the highest OA on ISPRS Vaihingen
dataset by enhancing the output of pixel-based semantic segmentation obtained from
Deep Neural Forest (DNF) with superpixel-enhanced Region Module (SRM). While some
used post-processing methods to enhance the segmented outputs, some omitted this step
as well [104,106,115,143,145,149,151,165].
4.2. Improvements from Deep Learning
Besides the improvement from the use of traditional methods on DL-based methods,
various improvements have been produced from the rise in the use of CNNs and FCNs.
Among the various architectures used (Figure 4), encoder-decoder architectures like SegNet,
few FCNs, FPN, U-Net, Hourglass and DeepLabv3+ mostly have a symmetrical architecture
with skip connections. These architectures minimized the problem of boundary pixel
classification by using both lower-level and higher-level information coming from skip
connections and previous down-sampling layers. Unlike CNN, these architectures do not
Remote Sens. 2021, 13, 808
18 of 41
rely only on the feature maps produced by the pooling layers. Azimi et al. [142] have
shown the impact of symmetrical FCN over non-symmetrical FCN like FCN-8s in terms of
DL performance. Also from Table 1, 8 papers exclusively exploit contextual information to
improve the accuracy of pixel-based semantic segmentation.
Talking about pixel-based and patch-based semantic segmentation, 68 papers performed pixel-based and 4 performed patch-based [55,61,158,159]. Volpi et al. (2016) [55]
compared between patch-based, sub-patch-based and full-patch-based CNNs, where the
third one performed better. Kampffmeyer et al. [61] compared pixel-based vs patch-based
FCN while improving segmentation of small objects like cars. Pixel-based FCN performed
better than patch-based methods in their comparison. Other papers performed pixel-based
segmentation. Sherrah et al. have also argued that patch-based methods produce output
in lower resolution, do not fulfill the task of semantic segmentation and also perform
redundant operations on adjacent patches.
As most of the papers used the dataset from some semantic labeling contest, in the next
section, we compare those papers in more detail and point out how the DL performance
was improved by the contests.
4.3. Improvement Boosted by Dataset Challenges
In 2014, Gerke et al. [171] used Stair Vision Library (abbr. SVL) on ISPRS Vaihingen
2D labeling dataset for the first time, and in 2015, Paisitkriangkrai et al. [144] first used
multi-resolution CNN with RF and CRF (abbr. DSTO in ISPRS leaderboard) on the Potsdam
dataset. Since then, several competitive CNN and FCN-based architectures have included
their contribution in improving the accuracy of DL-based methods using the datasets of
Vaihingen and Potsdam. The comparison of their methods to each other and traditional
methods is summarized in Appendix B.
For the study target of “urban feature segmentation”, 32 papers that used Vaihingen
dataset achieved OA from 0.86 to the highest of 0.92 [143], 23 papers that used the Potsdam
dataset obtained OA of 0.85 to the highest 0.93 [143,147], among the 6 papers that used
IEEE GRSS the highest OA was 0.90 [155] and for Massachusetts Building dataset and Road
Dataset, the highest of 0.97 [159] and 0.94 [153] was achieved. Also for the other datasets,
the study target of “building/roof detection”, “urban feature segmentation”, “urban land
cover classification” and “road segmentation” achieved up to 0.99, 0.96, 0.99 and 0.90 DL
performance respectively. It can be seen that the performance metric of over 90% has been
achieved in all four study targets using high-resolution images, which was not possible
using traditional methods.
4.4. Problems Addressed by Deep Learning
Sections 3.2.5, 4.2 and 4.3 have shown that DL-based semantic segmentation shows
contrasting improvement compared to traditional methods. All of the four study targets
have achieved over 90% DL performance, even in challenging datasets like ISPRS 2D
labeling dataset and IEEE GRSS dataset. Some of the previously faced challenges of
pixel-level semantic segmentation have been addressed by several of the recent DL-based
methods, as listed below.
1.
Complete ignorance of spatial information: In most of the traditional methods for
pixel-level segmentation, spatial information was completely ignored. To solve this
problem, several DL-based studies figured out the use of contextual information from
lower and higher layers/levels of encoder (or downsampling) block, using skip connection and symmetrical networks. The features maps obtained from these lower to
higher levels of encoders are concatenated to the feature maps of decoder (or upsampling) layers in symmetrical networks. As this problem also entails incorrect segmentation caused by similar features of similar categories, concatenation/aggregation or
better fusion techniques are sought to merge feature maps of different levels. Table 1
shows 18 papers were motivated to use better fusion techniques. The details of these
papers are already presented in Sections 2 and 3.2.1.
Remote Sens. 2021, 13, 808
19 of 41
2.
3.
4.
5.
6.
Boundary pixel classification (aka boundary blur) problem: Several studies were
performed to address this problem. Chen et al. (2017) [78] removed max pooling layers
and used CRF for post-processing, Badrinarayanan et al. [111] reused pooling indices,
and Marmanis et al. (2018) [112] combined semantic segmentation with semantically
informed edge detection to make boundary explicit. Other papers [55,163] were also
motivated to address this problem. Specifically, encoder-decoder architectures with
symmetrical network and skip connections have minimized this problem significantly.
Mi et al. [143] alleviated the problem by using superpixel segmentation with region
loss to emphasize on homogeneity within and the heterogeneity between superpixels.
Class imbalance problem: Several studies proposed the use of contextual information to address this problem. Li et al. (2017) [162] used multi-skip network,
Liu et al. (2017) [94] proposed a novel Hourglass-Shaped CNN (HSN) to perform
multi-scale inference and Li et al. (2018) [154] proposed Contextual Hourglass Network (CxtHGNet). Dong et al. (2019) [110] and Dong et al. (2020) [109] proposed
DenseU-Net and SiameseDenseU-Net architecture inspired from [172] to solve this
problem. Some used MFB with focal loss to minimize this problem [61,94,109,110,162].
Salt-and-pepper noise: To minimize the noises produced by pixel-based semantic
segmentation, Guo et al. [100] denoised and smoothened the segmentation results by
implementing kernel-based morphological methods, and Zhao et al. [156] used CRF
to capture contextual information of the semantic segments and refine the classification map.
Structural stereotype and insufficient training: As highlighted by Sun et al. [115] in
2019, the problem of structural stereotype causes unfair learning and inhomogeneous
reasoning in encoder-decoder architectures. They alleviate this problem by random
sampling and ensemble inference strategy. They also proposed a novel encoderdecoder architecture called ResegNet to solve insufficient training.
Domain-shift problem: Another drawback of DL-based method is that the DL performance decreases when the study domains are shifted. To address this problem, and
reduce the domain-shift impact caused by imaging sensors, resolution and class representation, Benjdira et al. [127] used GAN consisting of generator and discriminant as
explained in Section 2.5. Liu et al. (2020) [107] used two discriminators to minimize
the discrepancy between the source and target domains. Also, many others performed
multi-modal, multi-scale and multi-resolution training of DL-models [56,80,91–98],
for sufficient training of DL model.
The major drawback of DL-based methods is considered the lack of training dataset.
Depending on the complexity of the problem and number of classes, any DL-model
would require a large set of training images. Also with the added complexity of remote
sensing-based data collection and expense, the use of DL is even more challenging. Several
augmentation techniques (Figure 3) are often used to increase the number and variation
of the dataset. However, ISPRS’s 2D labeling dataset and IEEE’s GRRS dataset have tried
to address the insufficiency of data by providing VHR images collected from UAVs, of
up to 5 cm resolution. Also, several studies used transfer learning to transfer the learning
from a model trained on a different domain to the model to be trained. Therefore, with
the availability of public datasets and the use of transfer learning, several studies have
therefore also conducted multi-modal and multi-resolution training of DL-models.
Another disadvantage of DL-based semantic segmentation is the requirement of a
high amount of label dataset that often needs manual annotation. This problem has also
been addressed to some extent by the public datasets by providing annotations as well, but
is still persistent if we want to our own datasets. Some studies used traditional methods
to produce the annotations [135,160]. In a similar way, one can also take advantage of
existing pre-trained models to create their label dataset. Some have also tried to improve
the labeling dataset [160,161]. Also, the labels can be collected from web services like open
street maps (OSM) [164] for the segmentation of some urban features.
Remote Sens. 2021, 13, 808
20 of 41
The other cons of DL-based methods are the necessity of GPU-based computation and
heavier size of models in terms of storage volume. This challenge is addressed by a few
cloud computing services like Google Colaboratory [173] recently, providing free usage
until several hours. From Figure 9, 62% of papers mentioned the GPU specification they
used, and it has to be noted that most of them were able to use costly GPU’s like Nvidia’s
Titan-X. This could also be because of the rapid introduction of the latest powerful GPUs
with higher capabilities and price reduction of the older ones. In a dissimilar note, some
have also attempted to produce a smaller DL-model of less than a megabyte [165] and
some reduced the training parameters [155] to create a faster model. Although DL-models
take longer time to train compared to traditional methods, the time to test the model is
significantly faster.
5. Conclusions and Future Directions
The results of the meta-analysis found that DL offers superior performance and
outperforms traditional methods. Several drawbacks of DL-based methods have been
addressed and minimized in recent years (Section 4.4), further increasing the performance.
However, more challenges are to be expected with the recent trend to classify smaller
features such as type of vehicle. Similar to the ISPRS and IEEE’s dataset for 2D labeling
challenge, availability of more public datasets including VHR imagery as well as coarser
resolution imagery could help in the improvement of urban feature classification, by
allowing multi-modal and multi-resolution training of DL-model. Also, more training
labels are expected in the future. Auxiliary data sources like OSM are needed to be updated
for the most recent labels. More digital image processing methods and vegetation indices
can be explored to make the labeling/annotation task easier. GANs can help increase
the number of training datasets by translating images, and also provide synthetically
produced ultra-high resolution images. Moreover, we propose the surge of studies that
focus on minimization of domain-shift problem, the number of training dataset, the training
parameters, size and the time required to train the DL models with optimized architectures,
GANs and transfer learning.
Author Contributions: Conceptualization, methodology, investigation, data curation, formal analysis, and writing-original draft preparation B.N.; supervision, writing—review and editing J.A. and
T.H. All the authors read, edited, and critiqued the manuscript and approved the final version. All
authors have read and agreed to the published version of the manuscript.
Funding: This study was financially supported by Thammasat University Research Fund under the
TU Research Scholar, Contract No. 6/2562.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: All the necessary data are included in the manuscript and the tables in
the Appendixs A–C . There no other data presented in the study.
Acknowledgments: This work was partially supported by Center of Excellence in Intelligent Informatics, Speech and Language Technology and Service Innovation (CILS), Thammasat University
and Advance Geospatial Technology Research Unit, SIIT, Thammasat University. We would like to
thank the supporters and also the reviewers, whose valuable suggestions and comments increased
the overall quality of this review paper.
Conflicts of Interest: The authors declare no conflict of interest.
Remote Sens. 2021, 13, 808
21 of 41
Appendix A
Table A1. Deep Learning-Based Semantic Segmentation for Urban LULC and Methods Used.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
DL Model
Framework
Metric
Highest Value
Ref.
1
UL. Cl.
RADARSAT-2
PolSAR data
No
10 land cover classes
DBN
N/A
Conf. matrix, OA, K
OA: 0.81
[136]
2
Rd. Seg.
No
Sky, Build, Road, Sidewalk, Vegetation, Car
MRF using S-SVM
C++
P-R Curve
and IoU
IoU: 78.71
[140]
3
UF. Seg.
Images
from
camera
mounted
on
a plane, Aerial
KITTI, Google
Earth Pro
ISPRS Vaihingen
Multi-spectral images were orthorectified using DEM, and converted to Pauli RGB Image using a
neighboring window of some size.
No
N/A
ISPRS: Vaihingen
CNN+RF+CRF (aka.
DSTO)
MatConvNet
CNN toolbox
and
MATLAB
OA and F1
OA: 0.87
[144]
4
B/R. Seg.
Massachusetts
Buildings and
Roads Dataset
Random rotation.
buildings, roads, and
others
CNN
Caffe
P, R
P-R: 0.84–0.86
[158]
5
B/R. Seg.;
Rd. Seg.
CNN
N/A
P-R Curve
P-R: 0.9–0.97
[159]
UF. Seg.
Rotation with a random
angle and random horizontal flip.
Flip and Rotation of 90
and 10 degree.
building, road, background
6
Massachusetts
Buildings and
Roads Dataset
ISPRS
Vaihingen
and
Potsdam
nDSMs are generated using lasground tool. 7500 patches are randomly extracted for each class. Image patches of 16 × 16, 32 × 32
and 64 × 64 pixels are used.
Mean value substraction over each
patch and division by standard deviation computed over the entire
dataset.
64 × 64 sized RGB image patch.
ISPRS Vaihingen and
Potsdam
Caffe
OA and F1
OA: 0.89–0.90
[31]
7
UF. Seg.
Random sampling of training data
Random flip and rotation, and noise injection
ISPRS Vaihingen and
Potsdam
FCN with downsampling (DS); FCN
without DS (DST_1);
DS+RF+CRF (DST_2)
Encoder-decoder
CNN.
OA, K, AA,
and F1
OA: 0.89–0.90
[55]
8
UL. Cl.
Image patch of 7 × 7 and 25 × 25
were used to train the model.
No
AGRICULTURE: coffee
crop and non-coffee. URBAN: unclassified, road,
trees, red roof, grey roof,
concrete roof, vegetation, bare soil
DAGNN
from MatConvNet
Torch
OA and K
OA: 0.91
[137]
ISPRS
Vaihingen
and
Potsdam
IEEE GRSS; Coffee Dataset
Mean subtraction;
Vahingen:
128 × 128; Potsdam: 256 × 265;
no overlapping
CNN
Remote Sens. 2021, 13, 808
22 of 41
Table A1. Cont.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
DL Model
Framework
Metric
Highest Value
Ref.
9
UF. Seg.
ISPRS
gen
Vaihin-
Patch-based: First extracted a
patch with every car at center.
Pixel-based: Image patches with
50% overlap.
ISPRS Vaihingen
CNN and FCN
Caffe
OA and F1
OA: 0.87
[61]
10
UF. Seg.
FCN
N/A
[157]
Random flip (vertical,
horizontal orboth) and
transposition.
ISPRS Vaihingen and
Potsdam
FCN-based
(INR)
Caffe
Mean acc.
and OA
OA and F1
OA: 0.88
UF. Seg.
OA: 0.87–0.89
[145]
12
UF. Seg.
ISPRS
gen
No
ISPRS Vaihingen
SegNet
N/A
P, R and F1
OA: 0.9
[92]
13
B/R. Seg.
No
Residential Area, Buildings and Roads
CNN: VGGNet based
on Alexnet;
and
ResNet.
Caffe
F1
F1: 0.94
[152]
14
B/R. Seg.
Local Patches: 64 × 64 and Global
patches: 256 × 256
No
Residential Area, Buildings and Roads
CNN: VGGNet based
on Alexnet;
and
ResNet.
Caffe
F1
F1: 0.94
[153]
15
UF. Seg.
No
No
ISPRS Vaihingen; Scenes:
sky, tree, road, grass, water, building, mountain,
and foreground object
CNN (aka. ETH_C)
Python and
Matlab
Pixel acc.
and Class
acc.
Pixel acc: 0.85
[165]
16
UF. Seg.
Massachusetts
Buildings
Dataset, European Buildings
Dataset,
Romanian Roads,
Satu
mare
Dataset
Massachusetts
Buildings
Dataset, European Buildings
Dataset,
Romanian Roads,
Satu
mare
Dataset
ISPRS
Vaihingen
and
Stanford Background dataset
(Scene data)
ISPRS Potsdam;
OSM; Google
Earth
Randomly sampled 12,000 patches
of 259 × 259 px for training.
Random sampling to select
patches of 256 × 256 for Vaihingen
and 512 × 512 for Potsdam, with
overlap.
Prepare a composite of DSM,
nDSM and NDVI. Patches of
256 × 256 px are used for training,
which are overlapped for testing.
Local Patches: 64 × 64 and Global
patches: 256 × 256
ISPRS Vaihingen
11
ISPRS Vaihingen
ISPRS
Vaihingen
and
Potsdam
1. Patch-based: Rotation.
2. Pixel-based: Flip (horizontal and vertical) and
rotation at 90 degree intervals.
No
Mean value substraction on every
500 × 500 px patch.
No
Building, road, and background
FCN based on VGG16
N/A
F1
OA: 0.88
[161]
Vaihin-
MLP
Remote Sens. 2021, 13, 808
23 of 41
Table A1. Cont.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
DL Model
Framework
Metric
Highest Value
Ref.
17
UF. Seg.
ISPRS Vaihingen and Beijing
dataset using
Worldview-2
satellite
18 × 18 image patches used for
training
No
CNN
N/A
OA and K
OA: 0.86–0.96
[156]
18
UF. Seg.
ISPRS
gen
Vaihin-
Patches of 600 × 600 with 50%
overlap
CNN (named GSN)
with ResNet-101
Caffe
OA and F1
OA: 0.89
[87]
19
UF. Seg.
ISPRS
gen
Vaihin-
256 × 256 patches with 50% overlap
Rotation of 90 and 180
degrees. Random mirror,
rotate between −10 and
10 degrees, resize by factor between 0.5 and 1.5,
and gaussian blur.
Rotate by step of 90 degrees and flip.
ISPRS Vaihingen; Beijing
dataset:
Commercial
buildings,
Residential buildings, Roads,
Parking lots, Shadows,
Impervious
surfaces,
Bare soils
ISPRS Vaihingen
Caffe
OA and F1
OA: 0.86
[162]
20
UF. Seg.
ISPRS Potsdam
CNN with multiple
skip
connections
(named MSN)
FCN
Caffe
OA and F1
OA: 0.88
[150]
21
UF. Seg.
ISPRS
hingen
Potsdam
Vaiand
22
UF. Seg.
ISPRS
hingen;
GRSS
VaiIEEE
23
UF. Seg.
ISPRS
hingen
Potsdam
Vaiand
nDSM, NDVI were produced from
Lidar. 36,000 images and GTs of
224 × 224 px used for training
and 50% overlap used on test data.
Chose extra data for car category
for data balancing.
Training: NIR-R-G-B and the
nDSMs of 256 × 256 px with 50%
of overlap. Testing: 0%, 25%, 50%
and 75% overlaps.
Super-pixel segmentation by
multi-scale semantic segmentation. Image patches of 32 × 32,
64 × 64 and 128 × 128 are extracted around the superpixel
centroid. Superpixel is classified
first. Then the multi-scale patches
are resized to 228 × 228 and fed to
pre-trained AlexNet.
A composite image of stacked
NDVI, DSM and nDSM are first
prepared. Patch size = 128 × 128;
stride 32 px and 64 px for two
dataset.
ISPRS Vaihingen
Random crop
ISPRS Potsdam
Flip (horizontal and vertical)
ISPRS Vaihingen and
Potsdam
CNN (named HSN)
Caffe
OA and F1
OA: 0.89
[94]
No
Impervious
surfaces,
Building, Low vegetation, Tree, Car
AlexNet CNN and
SegNet with VGG-16
encoder
N/A
OA
OA: 0.89
[93]
No
ISPRS Vaihingen and
Potsdam
SegNet-RC,
VFuseNet,
ResNet34-RC, FusResNet.
Caffe
P,R,
OA
and F1
OA: 0.90–0.91
[146]
Remote Sens. 2021, 13, 808
24 of 41
Table A1. Cont.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
DL Model
Framework
Metric
Highest Value
Ref.
24
B/R. Seg.;
UF. Seg.
400 × 400 patches with the overlap
of 100 pixels
Flip (horizontal and vertical) and rotate counterclockwise at the step of
90 degrees.
ISPRS Vaihingen and
Potsdam; Massachussets: buildings
ScasNet (aka. CASIA)
Caffe
IoU and F1
F1: 0.92–0.93
[151]
25
UF. Seg.;
UL. Cl.
Caffe
OA and F1
OA: 0.90–0.93
[147]
UF. Seg.
ISPRS
Vaihingen
and Potsdam; IEEE
GRSS; Sentinel:
water,farmland, forest and
urban area.
ISPRS Vaihingen
Sevral fusion techniques: CoFsn, LaFsn
and
LnFsn
(aka.
RIT_3 ... RIT_7)
26
ISPRS
Vaihingen
and
Potsdam; Massachussets
Buildings
dataset
ISPRS
Vaihingen
and
Potsdam; IEEE
GRSS; Sentinel1; Sentinel-2
ISPRS Vaihingen
N/A
OA, P, R
and F1
OA: 0.85–0.90
[112]
27
UF. Seg.
RIT-18 dataset
Ensemble of SegNet
and two variants of
FCN initialized with
(i) Pascal and (ii)
VGG-16
SharpMask and RefineNet
Theano/
Keras
Per-class
acc.,
AA
and OA
AA: 0.60
[121]
28
UF. Seg.
P, R, OA,
K, F1 and
mIoU
Pixel acc.
and mIoU
OA: 0.88–0.89
[125]
UF. Seg.
FCN (named SDFCN
and SNFCN, aka.
CVEO)
Cotextual Hourglass
Network (named CxtHGNet)
Keras and
Tensorflow.
29
ISPRS
hingen
Potsdam
ISPRS
hingen
Potsdam
Pixel acc: 0.87
[154]
30
Rd. Seg.
FCN-32s
(named
Aerial LaneNet)
Tensorflow
Pixel acc.,
AA, IoU,
Dice Sim.
Coef.,
P
and R
AA: 0.70
[142]
Vaiand
Vaiand
AerialLanes18
dataset
ISPRS and IEEE data: Tiling into
224 × 224 without overlap, and
random sampling. For Sentinel
images: labels were created from
OSM.
256 × 256 pixels tiles with strides
of 150, 200 and 220 px.
Pre-trained ResNet-50 FCN and
520 thousand 80 × 80 MSI are
randomly shuffled for training.
Then again trained the model with
16 × 160 patches of their training
dataset for semantic segmentation.
Patches of 128 × 128 × 3 with
stride of 48 px (62.5) and 128 px
(100)
Patches of size 256 × 256 from the
original images without overlap,
and pad 0 s if needed. Further, the
training images are split into train
and valid sets in the ratio of 9:1.
Training data are randomly shuffled.
1024 × 1024 patches are cropped
bt 800 px step in horizontal and
vertical directions. Manual annotation of road lane features are prepared.
Random: scaling, rotation, linear shear, translation and flips (vertical
and horizontal axis).
Random horizontal and
vertical flips
18 urban feature classes
Random scaling, translation and flips (horizontal
and vertical)
Random flip (horizontal and vertical), scale
and crop image into fix
size with padding 0 s if
needed.
ISPRS Vaihingen and
Potsdam
Random flip
5 classes including road
signs and lane lines
ISPRS Vaihingen and
Potsdam
Tensorflow
Remote Sens. 2021, 13, 808
25 of 41
Table A1. Cont.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
DL Model
Framework
Metric
Highest Value
Ref.
31
UF. Seg.
ISPRS
gen
Random horizontal flips
and resize with five
scales, 0.5, 0.75, 1.0, 1.25,
and 1.5
ISPRS Vaihingen
PSPNet with pretrained ResNet101-v2
Caffe
P, R, and F1
OA: 0.88
[89]
32
UF. Seg.
ISPRS
hingen
Potsdam
Method 1. Crop each entire image
into 500 × 500 patches, use vertical/horizontal overlap of 100 px.
Method 2. Randomly crop these
patches again into 473 × 473 to
train the network.
Potsdam: 512 × 512, Vaihingen:
256 × 256, both with 50% overlap.
Flip (horizontal and vertical) and rotation by
steps of 90 degree.
ISPRS Vaihingen and
Potsdam
N/A
OA and F1
OA: 0.89–0.9
[148]
33
B/R. Seg.
Aerial Imagery
for Roof Segmentation
(AIRS) dataset
Random horizontal flip,
random scaling, crop
into 401 × 401 pixel
patch and random rotation of 0, 90, 180, 270.
Roof
TensorFlow
IoU, F1, P
and R
F1: 0.95
[91]
34
UF. Seg.
ISPRS
gen
Random scaling, horizontal flip and rotation
of step of 90 degrees.
ISPRS Vaihingen
CNN
MXNet
deep learning
OA, mIoU
and F1
OA: 0.86
[95]
35
UF. Seg.
Flip (horizontal and vertical) done on 3 quarters
of image patches.
Impervious
surfaces,
Building, Low vegetation, Tree, Car, Clutter
RiFCN with forward
stream inspired by
VGG-16.
TensorFlow
F1, P, R, OA
and IoU
OA: 0.88–0.96
[76]
36
UF. Seg.
ISPRS Potsdam;
Inria
Aerial
Image Labeling
Data Set
ISPRS
Vaihingen;
IEEE
GRSS
Significant misalignment between
building footprint and detected
roof were corrected and refined.
GT dataset was then created containing 226,342 buildings for roof
segmentation.
Stack of NIR-R-G image, DSM,
nDSM, normalization to NIR-R-G
to nNIR, nR, nG, NDVI, GNDVI
are used to train the model. Image
patches of 992 × 992 px with step
size of 812 px used for patch-wise
prediction.
IRRG images with prepared nDSM
are used to train model. Manually
annotated labels for Inria dataset.
FSN
(aka.
CASDE2/CASRS1)
and FSN-noL (aka.
CASDE1/CASRS2)
FPN, FPN + MSFF
and PSPNet
No
Pylon
model
OA,
AA
and F1
OA: 0.86
[163]
UL. Cl.
Random rotation and
stretch.
Impervious
surfaces,
Building,Low vegetation, Tree, Car, water
and boats
11 classes that are not
mentioned.
CNN based on modified VGG-16 and CRF
37
Vaihingen: Images of 256 × 256
px and nDSM. IEEE GRSS: Lidar
transformed into DSM an images
of 500 × 500 px.
No
U-Net
N/A
IoU
OA
OA: 0.99; IoU:
0.94
[100]
38
UF. Seg.
Vahingen: 128 × 128; Potsdam:
256 × 265; no overlapping
No
ISPRS Vaihingen
FuseNet
ReuseNet
ITCB)
N/A
OA, K, AA
and F1
OA: 0.88
[149]
Vaihin-
Vaiand
Vaihin-
RGB
Urban
planning maps
of
Shibuya,
Tokyo.
ISPRS Vaihingen; Worldview3
with
(aka.
and
Remote Sens. 2021, 13, 808
26 of 41
Table A1. Cont.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
DL Model
Framework
Metric
Highest Value
Ref.
39
UF. Seg.
Bing Maps and
Google Street
Views
No
11 classes of building
types based on usage
PlacesCNN and SegNet
Pytorch
Pixel acc,
mIoU and
K
OA: 0.78
[113]
40
UL. Cl.;
UF. Seg.
ISPRS Vaihingen; Landsat-8
Aerial images, corresponding land
use maps, and sparsely sampled
street views are collected and prepared to train.
Mean subtraction done on the images, with patch size 512 × 512.
Random horizontal flip
CNN (named GCN)
Tensorflow
AA, F1 and
mIoU
F1: 0.79
[96]
41
B/R. Seg.
SpaceNet building
dataset;
Google Maps;
OSM;
Map
World
Rotation of step of 90 degrees.
U-Net
Keras
P, R and F1
OA: 0.70
[101]
42
UF. Seg.
ISPRS
Vaihingen
and
Potsdam; IEEE
GRSS; Coffee
dataset (SPOT)
A stack of (R,G,B, red edge,
coastal) of WorldView-3 and RGB
map images collected from auxiliary sources made total 8-channel
input for the U-Net architecture.
Each 650 × 650 px image was
rescaled into 256 × 256 px, or
sliced into 3 × 3 sub-images of 256
× 256.
No
ISPRS Vaihingen; Landsat: agriculture, forest, miscellaneous, urban, and water.
Buildings
CNN:
Dilated6,
DenseDilated6,
Dilated6Pooling,
Dilated8Pooling (aka.
UFMG 1 to 5)
Tensorflow
OA, AA, K
and F1
OA: 0.88–0.90
[155]
43
UF. Seg.
ISPRS
hingen
Potsdam
Vaiand
Patches of 500 × 500 px with overlap of 200 px between neighboring
patches
Random horizontal flip,
rescale and random
crop.
Caffe
P, R, F1 and
OA
OA: 0.89–0.90
[62]
44
UF. Seg.
ISPRS
Vaihingen
and
Potsdam; IEEE
GRSS
No
Impervious
surface,
buildings, low vegetation, trees, cars, boats
and water
Caffe
OA and F1
OA: 0.88
[84]
45
UL. Cl.
WorldView-2
FCN-8s: Images only. MLR: Images and LiDAR data (height,
height variations, surface norm),
and NDVI. Image patches of 224 ×
224 and 1000 pixels are randomly
selected, and 50 cars and boats
were randomly chosen. Overlap
of 50% used on image patches.
Used K-means clustering to check
the abundance of classes in the
study image. Then, they manually
labeled eight slices (2048 × 2048)
of whole area at pixel-level. Then
128 × 128 patches with 32 px stride
were prepared.
Combination
of
ResNet-101-v2 and
pyramid
pooling
module.
FCN
(named
DNN_HCRF)
Rotation with steps of 90
degree and flip.
Crop Area (CA) and
Non-CA
DeepLabv3+
Tensorflow
OA, F1 and
K
OA: 0.95
[134]
No
Coffee: Coffee and noncoffee; IEEE GRSS: trees,
vegetation, road, bare
soil, red roof, gray roof,
and concrete roof; ISPRS
Vaihingen and Potsdam
ISPRS Vaihingen and
Potsdam
Remote Sens. 2021, 13, 808
27 of 41
Table A1. Cont.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
DL Model
Framework
Metric
Highest Value
Ref.
46
UF. Seg.
QuickBird:
BGR-NIR at 2 m
res, Sentinel-2:
BGR-NIR
at
10 m res, and
TerraSAR-X:
SAR
images
with 6 m res.
No
urban, vegetation, water
and slums
FCN of VGG19 backbone
Tensorflow
K, PPV, OA
and IoU
OA: 0.91
[135]
47
B/R. Seg.
ISPRS Potsdam;
UK-based building
dataset;
OSM
No
Buildings
MaskR-CNN
RetinaNet
Tensorflow
mAP, OA
and F1
OA: 0.96–0.99
[160]
48
B/R. Seg.
No
Buildings
DeepResUnet
Keras and
Tensorflow.
P, R, F1, K
and OA
OA: 0.97
[103]
49
UF. Seg.
Linz Data Service:
VHR
satellite images
with
spatial
resolution
of
0.075 m for
Christchurch
and
Waimakariri,
New Zealand
ISPRS
Vaihingen
and
Potsdam
Patches of 224 × 224 with overlap of 28 px used for training. Labeling of reference data is based
on a multi-step image analysis
procedure through a combination
of hierarchical, knowledge-based
and object-based classification, machine learning and visual image
interpretation with 93% OA.
RGB, RG-DE, RG-DN, RGB-DE
and RGB-DN, where N = normalized pixel values, E = edgemagnitude data for depth channel
were used. Any areas missing lidar data and tiles with no building
were deleted. Labels for buildings
were created using image border
extraction algorithm.
Patches of 256 × 256 px with stride
of 128 during train and stride of 64
during test. Building labels were
prepared as polygon shapefiles of
building outines.
512 × 512 training patches.
No
ISPRS Vaihingen and
Potsdam
BiseNet
with
ResNet101
feature
extractor and GAN.
AA, P, R, F1
and IoU
F1: 0.49
[127]
50
UF. Seg.
Images of 256 × 256 px with 50%
overlap.
Rotated at 0, 90, 180 and
270 degrees and then
horizontally flipped.
ISPRS Vaihingen
DenseU-Net
Semantic
Seg. Suite
framework;
Keras and
tensorflow.
N/A
P, R, F1 and
OA
OA: 0.86
[110]
ISPRS
gen
Vaihin-
and
Remote Sens. 2021, 13, 808
28 of 41
Table A1. Cont.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
DL Model
Framework
Metric
Highest Value
Ref.
51
B/R. Seg.
WHU Building
dataset
Relative radiometric calibration
strategy and tiles of 512 × 512.
Buildings
SR-FCN with VGG-16
and Atrous conv layers
Keras with
TensorFlow
IoU, P, R
IoU: 0.64
[79]
52
Rd. Seg.
Mnih
dataset
Tiling: 500 × 500 pixels
Random
resampling
of images using one
of linear stretching,
histogram equalization,
gaussian blur,
and
salt-and-pepper noise
N/A
Road
Y-Net
Tensorflow
53
UF. Seg.
Tensorflow
54
B/R. Seg.
55
UF. Seg.
56
B/R. Seg.
CCF
Satellite
Image AI Classification and
Recognition
Competition
Inria
Aerial
Image Labeling
Dataset
ISPRS
Vaihingen
and
Potsdam
Gaofen-2 VHR
satellite
imagery
Mean acc,
Mean Accuracy:
0.83
pixel acc,
mIoU,
fwIoU,
dice
coeff.,
and
Matthew
correlation coeff.
(MCC)
Classification Acc: 0.77
acc
and
mean IoU
57
UF. Seg.
ISPRS
hingen
Potsdam
58
B/R. Seg.
Massachusetts
building dataset
2013
Vaiand
Random sampling of image tiles
of 512 × 512
N/A
Background, vegetation,
road, building and water
body
DeepLabv3 + ASPP +
FC-Fusion Path
Crop 24 patches of 384 × 384 px
Random horizontal and
vertical flips
Buildings
buildings
SegNet
Tiling: 128 × 128 and random position sampling
N/A
ISPRS Vaihingen and
Potsdam
ResegNet
Georectification, registration and
pan-sharpening. The images are
normalized. Labels are delineated
manually for segments of connected building roofs. Tiling: 512
× 512 and random sampling
Tiling: 600 × 600 with overlaps alculated using 2D Gaussian function.
N/A
Buildings
buildings
Tiling: 384 × 384
Transformation,
ping and rotation
N/A
clip-
and
and
non-
non-
ISPRS Vaihingen and
Potsdam
Buildings
[126]
[80]
IoU
and
pixel acc
IoU: 0.74
[114]
N/A
OA and F1
OA: 0.92
[115]
FCN with VGG-16
Tensorflow
OA, IoU
and mIoU
OA: 0.95
[169]
TreeUNet
with
DeepLabv3+
and
TreeUNet with DeepUNet
Seg-Unet
MXNET
deep learning
OA, F1, P,
R
OA: 0.90
[104]
Keras with
TensorFlow
OA, F1, P,
R
OA: 0.93
[116]
Remote Sens. 2021, 13, 808
29 of 41
Table A1. Cont.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
59
UF. Seg.
Tiling: 512 × 512
Randomly flipped and
rotated
Ground, grass,
building, car
60
B/R. Seg.
Buildings
DSFE
with
DenseNet
B/R. Seg.
62
B/R. Seg.
Inria
Band normalization, coregistration, refinement, and a truncated
signed distance map (TSDM).
Tiling: 256 × 256.
Pansharpening of 2 m MS imagery
into 0.5 m by nearest neighbor diffusion (NNDiffuse) pan sharpening algorithm. Tiling: 256 × 256
Tiling: 224 × 224
TSDM for the mediumresolution images
61
ISPRS
Vaihingen
and
Potsdam
ISPRS
Potsdam
and
PlanetScope
Dove
Worldview-2
Random crop; rotating,
mirroring, brightness enhancement and adding
noise points
N/A
Types of buildings: Old
house, Old factory, Iron
roof building and New
building.
Buildings
63
UF. Seg.
ISPRS
hingen
Potsdam
ISPRS Vaihingen and
Potsdam
64
UF. Seg.;
UL. Cl.
Zurich dataset,
Gaofen
Image
Dataset
(GID), and Data
Fountain 2017
Tiling: 256 × 256 with overlap 80%
Noise injection (salt
pepper noise or Gaussian
noise),
color
interference
(change
saturation, brightness
or contrast), random
non-proportional scaling from 0.8 to 1.2 times,
random rotation 0 to 360
degree and random flip
of 90, 180, or 270.
Random rotation and
random scaling
65
UF. Seg.
Zurich dataset
and ISPRS Vaihingen and Potsdam
Tiling 512 × 512 pixels. Overlap:
100 px for ISPRS and 256 px for
Zurich.
66
UF. Seg.
ISPRS
hingen
Potsdam
Tiling 512 × 512 pixels
Vaiand
Vaiand
Tiling: 500 × 500
Horizontal and vertical
flip and random scale rotation. Random changes
in brightness, saturation, and contrast were
adopted in color.
Random-flipping and
random-cropping.
tree,
DL Model
Framework
Metric
Highest Value
Ref.
BSANet
Pytorch
Pixel acc, K,
F1, mIoU
F1: 0.53
[107]
Pytorch
OA, F1 and
IoU
OA: 0.93
[164]
U-Net
Keras with
TensorFlow
OA, F1 and
IoU
OA: 0.87
[102]
SegNet with VGG-16
and FCN
DeepLabv3+
N/A
Val
acc
(IoU)
OA and K
0.90
[117]
OA: 0.91
[124]
FC-
N/A
Zurich dataset: Road,
buildings, trees, bare
soil, water, grass, rails
and pools.
Gaofen
dataset:
land-cover
classes
ISPRS Vaihingen; Zurich
dataset: Road, buildings,
trees, bare soil, water,
grass, rails and pools.
Mask-R-FCN model:
Mask-RCNN
with
RPN and FCN8s with
VGG-16
N/A
OA, AA,
F1, P, R
AA: 0.82
[138]
MANet
Pytorch
OA, F1, P,
R
OA: 0.88–0.89
[97]
ISPRS Vaihingen and
Potsdam
HRNet
Pytorch
OA, F1, P,
R
Vaihingen OA:
0.90; Potsdam
OA: 0.92
[98]
Remote Sens. 2021, 13, 808
30 of 41
Table A1. Cont.
SN
Area
Data Summary
Pre-Processing/Preparation
Data Augmentation
Classes
DL Model
Framework
Metric
Highest Value
Ref.
67
UF. Seg.
ISPRS Potsdam
Tiling: 256 × 256 with stride of
64 px and 128 px
ISPRS Potsdam
ResUNet-a
MXNET
deep learning library
OA, F1, P,
R, MCC
OA: 0.91
[108]
68
UF. Seg.
ISPRS
gen
Vaihin-
Tiling: 256 × 256 with 50% overlap
ISPRS Vaihingen
SiameseDenseU-Net
N/A
OA, F1, P,
R
OA: 0.86–0.89
[109]
69
UF. Seg.
ISPRS
hingen
Potsdam
Vaiand
Tiling: Train: Cropped the image to patches of 256 × 256 with
random upper left corners. Test:
256 × 256 with 50% overlap
ISPRS Vaihingen and
Potsdam
SDNF+SRM
backbones
ResNet101
ASPP
N/A
OA, F1, P,
R
OA: 0.92–0.93
[143]
70
UF. Seg.
NWPU-VHR-10
dataset
Cars, buildings, ships,
trees, tennis courts, basket ball courts, ground
track fields, harbors,
bridges and airplanes
Tree-based CNN with
Decision tree and
Atrous conv layers
MatLab
Accuracy,
F1, P, R
F1: 0.94
[106]
71
UL. Cl.
N/A
An unsharp mask filter followed
by a median filter, resized, the median values are obtained from histogram and the images image are
pre-filtered. Also, linear contrast
filter and a histogram-equalized
with a spatial filter.
Wiener Filter.
Crop image
into 50 × 50 px and enlarge to
500 × 500 px.
Geometric data sugmentation: Rotation at random angle, with random centre and zoomed
in/out according to random scale factor. Reflect
padding.
Rotated at four angles (0,
90, 180, and 270), and
each rotated image was
horizontally mirrored.
Random enhancement
of images. Rotating with
a random angle; Random Gamma transformation; Gaussian blur;
Adding Gaussian noise.
No
N/A
Water, road, residential
area and natural vegetation
GoogLeNet
N/A
F1, P, R
F1: 0.68
[139]
with
of
and
Remote Sens. 2021, 13, 808
31 of 41
Appendix B
Table A2. Comparison of Papers Using ISPRS 2D Semantic Labeling Dataset.
SN
Ref.
Year
Their Model
1
[144]
2015
2
3
[55]
[31]
2016
2016
CNN + RF + CRF (abbr.
DSTO)
CNN
FCN + RF + CRF (abbr. DST)
4
5
[157]
[145]
2016
2016
FCN (DLR)
MLP (INR)
6
[92]
2016
Multi-kernel SegNet
7
[165]
2017
CNN (ETH_C)
8
9
[161]
[87]
2017
2017
FCN
CNN called GSN
10
11
12
13
[162]
[150]
[94]
[151]
2017
2017
2017
2018
14
[146]
2018
15
[112]
2018
16
[125]
2018
CNN
FCN
CNN called HSN
VGG-Scasnet (CASIA-1) and
ResNet-ScasNet (CASIA-2)
SegNet-RC,
V-FuseNet,
ResNet-34-RC and FusResNet
Ensemble of SegNet, CNN
and FCN (DLR)
SDFCN-139 (CVEO)
17
[89]
2018
PSPNet
Models Compared for Vaihingen
Better by
Models Compared for Potsdam
Better by
UT_Mev, SVL 3, HUST, ONE_5, ADL_3,
UZ_1 [55], DLR_1, DLR_2
7%
SVL_1
12%
CNN + RF and CNN + RF + CRF of [144], Deconvolution [55], Dilation and Dilation + CRF
of [31]
SVL_3, RF + CRF (HUST), CNN ensemble
(ONE_5), FCN (UZ_1), FCN (UOA), CNN + RF
+ CRF (ADL_ 3), FCN (DLR_2), FCN + RF +
CRF (DST_2)
CNN [144] and CNN_PC [55]
1 to 2%
Dilation, VGG pretr. and VGG + Dilation of
(sherrah2016fully)
Reference [31]’s VGG + Dilation was better
UPB, ETH_C, UOA, ADL_3, RIT_2, DST_2,
ONE_7, DLR_9
1% to 5%
SVL_6, UZ_1 [55], ADL_3 [174], DST_2 [31],
DLR_8 [157], ONE_7
FCN of [31,157]
3 to 13%
SVL_3, GU, UZ_1, AZ_1, RIT_2, DST_2
1 to 2%
FCN of [157] performed
better by 0.3%
FCN of [31,150]
Their V-FuseNet was better
compared to [31,150] by 0.3%
and 1.2% OA.
AMA_1, CASIA2, AZ3, RIT6, BUCT_1,
WuhZ, KLab_2, UZ_1, GU
AMA_1 (91.2), CASIA2
(91.1), AZ3 (90.7), RIT6 (90.2)
and BUCT_1 (90) were better
than CVEO (89) by 1 to 2%.
1% to 5%
Not better, but smaller
model
DST_2, INR, ONE_6 and ONE_7
1%
(CASIA, HUSTW4, ADL_3, WUH_C4, RIT_L8,
RIT_4, CONC_2, HUST, UPB and Ucal)
CASIA (90.6) and
HUSTW4 (89.5) were
better than their CVEO
(88.3) by 1 and 2%
up to 6%
SVL_3, UT_Mev, ETH_C. UPB, UZ_1 and
CAS_L1
Remote Sens. 2021, 13, 808
32 of 41
Table A2. Cont.
SN
Ref.
Year
Their Model
Models Compared for Vaihingen
Better by
18
[163]
2018
CNN
All of the compared
models were better
19
[147]
2018
20
[148]
2018
21
[149]
2018
22
[96]
2019
CoFsn, LaFsn and LnFsn
(RIT_3 to RIT_7)
FSN
(CASDE2/CASRS1)
and
FSN-noL
(CASDE1/CASRS2)
FuseNet with ReuseNet
(ITCB)
CNN called GCN
Reference [112]’s single scale, ensemble and full
model, [55]’s Segnet, [146]’s SegNet, WUH_W3
(ResNet-101, à-trous conv), CAS_L1 (PSPNet)
and HUSTW5 (ensemble of deconv. Net and
U-Net)
DST_2, DLR_10, structured RF and NLPR_3.
23
[62]
2019
FPN
24
[84]
2019
DNN_HCRF
25
[110]
2019
U-Net
26
[155]
2019
DLR_9, GSN_3, ONE_7, INR, DST_2, UFMG_2,
ADL_3, RIT_2, RIT_L8, UZ_1.
27
[115]
2019
Dilated6,
DenseDilated6,
Dilated6Pooling,
Dilated8Pooling (UFMG 1
to 5)
ResegNet (HUSTW)
28
[104]
2019
TreeUNet
SVL_3, DST_2, UZ_1, RIT_L7, ONE_7, ADL_3,
DLR_10, CASIA2, BKHN10
29
30
[107]
[164]
2020
2020
BSANet
DSFE-GGCN
Models Compared for Potsdam
Better by
NLPR3 was better.
FCN-8s, DST_5 and CASIA2.
CASIA2 was better.
ONE_7, UZ_1, DST_2
ONE_7 was better
Ensemble of 5 FCN-8s (BKHN_2) and
RITL_7.
BKHN_2 was better
FCNs of [31], CNN-FPL [55] and AllConvNet.
Their better in OA by
1%.
Their GCN with Res152
backbone had better F1
score than DCED by
2.5%
1 to 4%
SP-SVL, DCNN [121], CNN-FPL and SegNetp.
SVL_1, UZ_1, KLab_3 and DST_6
4 to 13%
DST_5, RIT_L7, Klab_2, UZ_1
DST_5 and RIT_L7 are better
by up to 2%.
15 model from leaderboard including AMA
and CASIA
HUSTW better than AMA
and CASIA by 0.5 and 0.4%
resp.
SWJ_2 and CASIA2 are better by less than 1%.
DCED of [111]
SP-SVL [171], CNN_HAW [165], CNN-FPL [55]
and SegNet-p [112].
SVL_3, ADL_3, ONE_7, UZ_1, DLR_10 and
UOA
3 hourglass-based models of [94]
20 model from leaderboard including NLPR
and CASIA
DLR_10 was better by
2.5%
Their mode better by
1%.
DLR_9, GSN_3, ONE_7
and INR are better by
up to 1%.
HUSTW better than
NLPR and CASIA by
0.5 and 0.4% resp
BKHN10 and CASIA2
were better than TreeUNet (with DeepUNet)
in OA by around 0.6%.
SVL_1, DST_5, UZ_1, RIT_L7, SWJ_2, CASIA2.
FCN-32s, SegNet, FCN16s, U-Net, FCN-8s,
ResNet-DUC, CWGAN-GP, FC-DenseNet,
GCN, GraphSAGE, and GGNN
DST_6 was better by 1.8%
Better than DSFE-GCN [175]
by 1%.
Remote Sens. 2021, 13, 808
33 of 41
Table A2. Cont.
SN
Ref.
Year
Their Model
Models Compared for Vaihingen
Better by
Models Compared for Potsdam
Better by
31
[124]
2020
DeepLabv3+
Their method better
than BKHN by 1%.
[150,157] and U-Net
Better than U-net by 1.2%
32
[97]
2020
MANet
33
[98]
2020
HRNet
SVL_3, ADL_3, UT_Mev, HUST, ONE_7, DST_2,
UZ_1, DLR_9, IVFL, ETH_C, UCal6, CASIA3,
RIT_7, HUSTW3, WUH_C3, CASRS1 and
BKHN10
FCN, U-net, UZ1, Light-weight RefineNet,
DeepLabv3+ and APPD
FCN, PSPNet, DeepLabv3+, SENet, CBAM,
GloRe and DANet.
MANet was better than
APPD by 1%.
HRNet was better than
DeepLabv3+ in OA by
1.16%
FCN, U-net, UZ1, Light-weight RefineNet,
DeepLabv3+ and APPD
FCN, PSPNet, DeepLabv3+, SENet, CBAM,
GloRe and DANet.
MANet was better than
APPD by 1%.
HRNet was better than
DeepLabv3+ in OA by 1.74%
34
[108]
2020
ResUnet-a
UZ_1 , RIT_L7, RIT_4, DST_5, CAS_Y3, CASIA2, DPN_MFFL and HSN + OI + WBP.
Their method better than
CASIA2 [151] in OA by 0.4%.
35
[109]
2020
SiameseDenseU- Net
HSN [94], U-Net, DenseU-Net (with and without CE loss and MFB Focal loss)
36
[143]
2020
SDNF+SRM
UPB, UZ_1, RIT_L8, ADL_3, CVEO, ITC_B2,
DST_2, UFMG_4, INR, MMDN, RIT_7, VFuseNet, TreeUNet, DLR_9, BKHN11, CASIA2,
NLPR3, HUSTW5
UZ_1, Klab_3, UFMG_4, RIT_L7, CVEO,
DST_5, RIT6, V-FuseNet, TreeUNet, CASIA3,
BKHN_3, AMA_1, HUSTW4, SWJ_2
Their method better than
SWJ_2 in OA by 1%.
Their SiameseDenseUNet + MFB_Focalloss
was better than DenseUNet+ MFB_Focalloss by
1%
Their method better
than HUSTW5 [115] in
OA by less than 1%.
Remote Sens. 2021, 13, 808
34 of 41
Appendix C
Table A3. Datasets for Urban LULC Segmentation.
S.N.
Dataset/ Challenge
Description
Resolution
Tags
Link
1
2
Linz Data Service
Massachusetts
Bldg./Road
Dataset
ISPRS Vaihingen 2D Semantic Labeling
ISPRS Potsdam 2D Semantic Labeling
IEEE GRSS Data Fusion
RGB images of New Zealand
Train and test images with vector data
10 m
1m
Land Cover
Building and Road
https://data.linz.govt.nz/
https://www.cs.toronto.edu/~vmnih/data/
33 patches of true orthophoto (TOP) with IR-R-G bands,
DSM, labelled ground truth (GT)
38 patches of TOP with different bands (IRRG, RGB, RGBIR), DSM, labelled GT
Lidar and image collected by airborne platform
9 cm
Imp. surface, bldg., low veg.,
tree, car
Imp. surface, bldg., low veg.,
tree, car, clutter/ background
Urban land cover
3
6
NZAM/ONERA Christchurch
dataset
1785 x ortho-rectified RGB GeoTIFF images
10 cm
background, building, vegetation and vehicle
7
RIT-18 dataset UAS-based spectral dataset
AerialLanes18 dataset
UAS-collected dataset with 18 labeled object classes
4.7 cm
18 urban feature classes [121]
http://www2.isprs.org/commissions/comm3/wg4/2
d-sem-label-vaihingen.html
http://www2.isprs.org/commissions/comm3/wg4/2
d-sem-label-potsdam.html
http://www.grss-ieee.org/community/technicalcommittees/data-fusion/2015-ieee-grss-data-fusioncontest/
https://www.linz.govt.nz/land/maps/linztopographic-maps/imagery-orthophotos/christchurchearthquake-imagery
https://github.com/rmkemker/RIT-18
RGB images of size 5616 × 3744 pixels with flight height
of 1000 m
Collected from Linz Data Service for Christchurch City;
RGB bands
360 ortho-rectified aerial RGB images of 5000 × 5000 px
13 cm
Aerial road lanes and vehicles
[142]
7.5 cm
https://www.airs-dataset.com/
30 cm
contains
over
220,000
buildings [91]
Building segmentation dataset
30 cm to 50 cm
Building footprints and roads
https://spacenetchallenge.github.io/
https://www.iarpa.gov/challenges/fmow.html
[160]
8m
0.075 m
For multiple area
24,556 images with 169,835 buildings
[138]
Building segmentation dataset
https://www.datafountain.cn/competitions/270
http://study.rsgis.whu.edu.cn/pages/download/
sub-meter
Urban features
[80]
8 cm to 2 m
Urban features
0.6
m
pansharpened
8 LULC classes
https://ifpwww.ifp.uni-stuttgart.de/dgpf/DKEP-Allg.
html
[97,138]
For multiple area
https://www.sipeo.bgu.tum.de/downloads
4
5
8
9
11
Aerial Imagery for Roof Segmentation (AIRS)
Inria Aerial Image Labeling Data
Set
SpaceNet building dataset
12
13
fMoW Challenge
UK-based bldg. dataset
14
15
DATAFOUNTAIN 2017
WHU Building dataset
16
17
CCF Satellite Image AI Classification and Recognition Competition
NWPU VHR-10
18
Zurich Dataset
Images from Google Earth (0.5–2 m) and 0.08 m infrared
images from ISPRS Vaihingen dataset
Quickbird’s multi-spectral images
19
Some more datasets from Signal
Processing in Earth Observation
Several Remote Sensing-based datasets for different purpose
10
Images collected from WorldView-2 and 3 for several
locations; 8 band multispectral
4-band and 8-band multispectral imagery
RGB image, DSMs, and OSM shapefile
Comes from Linz Data Service with training samples for
buildings
RGB images
5 cm
Lidar: 10 cm,
Image: 5 cm
25 cm
https://project.inria.fr/aerialimagelabeling/
Remote Sens. 2021, 13, 808
35 of 41
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
Drusch, M.; Del Bello, U.; Carlier, S.; Colin, O.; Fernandez, V.; Gascon, F.; Hoersch, B.; Isola, C.; Laberinti, P.; Martimort, P.; et al.
Sentinel-2: ESA’s optical high-resolution mission for GMES operational services. Remote Sens. Environ. 2012, 120, 25–36.
[CrossRef]
Irons, J.R.; Dwyer, J.L.; Barsi, J.A. The next Landsat satellite: The Landsat data continuity mission. Remote Sens. Environ. 2012,
122, 11–21. [CrossRef]
Myint, S.W.; Gober, P.; Brazel, A.; Grossman-Clarke, S.; Weng, Q. Per-pixel vs. object-based classification of urban land cover
extraction using high spatial resolution imagery. Remote Sens. Environ. 2011, 115, 1145–1161. [CrossRef]
Cowen, D.J.; Jensen, J.R.; Bresnahan, P.J.; Ehler, G.B.; Graves, D. The design and implementation of an integrated geographic
information system for environmental applications. Photogramm. Eng. Remote Sens. 1995, 61, 1393–1404.
Campbell, J.B.; Wynne, R.H. Introduction to Remote Sensing; Guilford Press: lNew York, NY, USA, 2011.
Walter, V. Object-based classification of remote sensing data for change detection. ISPRS J. Photogramm. Remote Sens. 2004,
58, 225–238. [CrossRef]
Myint, S.W.; Giri, C.P.; Wang, L.; Zhu, Z.; Gillette, S.C. Identifying mangrove species and their surrounding land use and land
cover classes using an object-oriented approach with a lacunarity spatial measure. GIScience Remote Sens. 2008, 45, 188–208.
[CrossRef]
Navulur, K. Multispectral Image Analysis Using the Object-Oriented Paradigm; CRC Press: Boca Raton, FL, USA, 2006.
Stow, D.; Lopez, A.; Lippitt, C.; Hinton, S.; Weeks, J. Object-based classification of residential land use within Accra, Ghana based
on QuickBird satellite data. Int. J. Remote Sens. 2007, 28, 5167–5173. [CrossRef] [PubMed]
Lu, D.; Weng, Q. A survey of image classification methods and techniques for improving classification performance. Int. J.
Remote Sens. 2007, 28, 823–870. [CrossRef]
Neupane, B.; Horanont, T.; Duy, H.N.; Suebvong, S.; Mahattanawutakorn, T. An Open-Source UAV Image Processing Web Service
for Crop Health Monitoring. In Proceedings of the 2019 8th IEEE International Congress on Advanced Applied Informatics
(IIAI-AAI), lToyama, Japan, 7–12 July 2019; pp. 11–16.
LeCun, Y.; Bengio, Y. Convolutional networks for images, speech, and time series. Handb. Brain Theory Neural Netw. 1995,
3361, 1995.
LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436. [CrossRef] [PubMed]
Pan, S.J.; Yang, Q. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 2009, 22, 1345–1359. [CrossRef]
Reyes, A.K.; Caicedo, J.C.; Camargo, J.E. Fine-tuning Deep Convolutional Networks for Plant Recognition. CLEF (Work. Notes)
2015, 1391, 467–475.
Amara, J.; Bouaziz, B.; Algergawy, A. A Deep Learning-based Approach for Banana Leaf Diseases Classification. In Datenbanksysteme für Business, Technologie und Web (BTW 2017)—Workshopband; Gesellschaft für Informatik e.V.: Bonn, Germany, 2017;
pp. 79–88.
Mohanty, S.P.; Hughes, D.P.; Salathé, M. Using deep learning for image-based plant disease detection. Front. Plant Sci. 2016,
7, 1419. [CrossRef] [PubMed]
Huang, H.; Deng, J.; Lan, Y.; Yang, A.; Deng, X.; Zhang, L. A fully convolutional network for weed mapping of unmanned aerial
vehicle (UAV) imagery. PLoS ONE 2018, 13, e0196302.
Kussul, N.; Lavreniuk, M.; Skakun, S.; Shelestov, A. Deep learning classification of land cover and crop types using remote
sensing data. IEEE Geosci. Remote Sens. Lett. 2017, 14, 778–782. [CrossRef]
Mortensen, A.K.; Dyrmann, M.; Karstoft, H.; Jørgensen, R.N.; Gislum, R. Semantic segmentation of mixed crops using deep
convolutional neural network. In Proceedings of the CIGR-AgEng Conference, Abstracts and Full Papers, Aarhus, Denmark,
26–29 June 2016; Organising Committee, CIGR: London, UK, 2016; pp. 1–6.
Neupane, B.; Horanont, T.; Hung, N.D. Deep learning based banana plant detection and counting using high-resolution
red-green-blue (RGB) images collected from unmanned aerial vehicle (UAV). PLoS ONE 2019, 14, e0223906. [CrossRef]
Dutta, R.; Aryal, J.; Das, A.; Kirkpatrick, J.B. Deep cognitive imaging systems enable estimation of continental-scale fire incidence
from climate data. Sci. Rep. 2013, 3, 1–4. [CrossRef] [PubMed]
Ghorbanzadeh, O.; Blaschke, T.; Gholamnia, K.; Meena, S.R.; Tiede, D.; Aryal, J. Evaluation of different machine learning methods
and deep-learning convolutional neural networks for landslide detection. Remote Sens. 2019, 11, 196. [CrossRef]
Ghorbanzadeh, O.; Meena, S.R.; Blaschke, T.; Aryal, J. UAV-based slope failure detection using deep-learning convolutional
neural networks. Remote Sens. 2019, 11, 2046. [CrossRef]
Zhu, X.X.; Tuia, D.; Mou, L.; Xia, G.S.; Zhang, L.; Xu, F.; Fraundorfer, F. Deep learning in remote sensing: A comprehensive
review and list of resources. IEEE Geosci. Remote Sens. Mag. 2017, 5, 8–36. [CrossRef]
Shadman Roodposhti, M.; Aryal, J.; Lucieer, A.; Bryan, B.A. Uncertainty assessment of hyperspectral image classification: Deep
learning vs. random forest. Entropy 2019, 21, 78. [CrossRef]
LeCun, Y.; Boser, B.; Denker, J.S.; Henderson, D.; Howard, R.E.; Hubbard, W.; Jackel, L.D. Backpropagation applied to handwritten
zip code recognition. Neural Comput. 1989, 1, 541–551. [CrossRef]
Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf.
Process. Syst. 2012, 25, 1097–1105. [CrossRef]
Remote Sens. 2021, 13, 808
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
36 of 41
Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3431–3440.
Fu, G.; Liu, C.; Zhou, R.; Sun, T.; Zhang, Q. Classification for high resolution remote sensing imagery using a fully convolutional
network. Remote Sens. 2017, 9, 498. [CrossRef]
Sherrah, J. Fully convolutional networks for dense semantic labelling of high-resolution aerial imagery.
arXiv 2016,
arXiv:1606.02585.
Everingham, M.; Van Gool, L.; Williams, C.K.; Winn, J.; Zisserman, A. The pascal visual object classes (voc) challenge. Int. J.
Comput. Vis. 2010, 88, 303–338. [CrossRef]
Li, Y.; Zhang, H.; Xue, X.; Jiang, Y.; Shen, Q. Deep learning for remote sensing image classification: A survey. Wiley Interdiscip.
Rev. Data Min. Knowl. Discov. 2018, 8, e1264. [CrossRef]
Ma, L.; Liu, Y.; Zhang, X.; Ye, Y.; Yin, G.; Johnson, B.A. Deep learning in remote sensing applications: A meta-analysis and review.
ISPRS J. Photogramm. Remote Sens. 2019, 152, 166–177. [CrossRef]
Wang, F. Fuzzy supervised classification of remote sensing images. IEEE Trans. Geosci. Remote Sens. 1990, 28, 194–201. [CrossRef]
Anees, A.; Aryal, J. Near-real time detection of beetle infestation in pine forests using MODIS data. IEEE J. Sel. Top. Appl. Earth
Obs. Remote Sens. 2014, 7, 3713–3723. [CrossRef]
Anees, A.; Aryal, J. A statistical framework for near-real time detection of beetle infestation in pine forests using MODIS data.
IEEE Geosci. Remote Sens. Lett. 2014, 11, 1717–1721. [CrossRef]
Anees, A.; Aryal, J.; O’Reilly, M.M.; Gale, T.J. A relative density ratio-based framework for detection of land cover changes in
MODIS NDVI time series. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2015, 9, 3359–3371. [CrossRef]
Civco, D.L. Artificial neural networks for land-cover classification and mapping. Int. J. Geogr. Inf. Sci. 1993, 7, 173–186. [CrossRef]
Chen, K.; Tzeng, Y.; Chen, C.; Kao, W.; Ni, C. Classification of multispectral imagery using dynamic learning neural network. In
Proceedings of the IGARSS’93-IEEE International Geoscience and Remote Sensing Symposium, Tokyo, Japan, 18–21 August 1993;
pp. 896–898.
Foody, G.M. Image classification with a neural network: From completely-crisp to fully-fuzzy situations. Adv. Remote Sens. GIS
Anal. 1999, 17–37.
Flanagan, M.; Civco, D.L. Subpixel impervious surface mapping. In Proceedings of the 2001 ASPRS Annual Convention,
Bethesda, MD, USA, 23–27 April 2001; American Society for Photogrammetry & Remote Sensing: St. Louis, MO, USA, 2001;
Volume 23.
Yang, L.; Xian, G.; Klaver, J.M.; Deal, B. Urban land-cover change detection through sub-pixel imperviousness mapping using
remotely sensed data. Photogramm. Eng. Remote Sens. 2003, 69, 1003–1010. [CrossRef]
Powell, R.L.; Roberts, D.A.; Dennison, P.E.; Hess, L.L. Sub-pixel mapping of urban land cover using multiple endmember spectral
mixture analysis: Manaus, Brazil. Remote Sens. Environ. 2007, 106, 253–267. [CrossRef]
Walton, J.T. Subpixel urban land cover estimation. Photogramm. Eng. Remote Sens. 2008, 74, 1213–1222. [CrossRef]
Deng, C.; Wu, C. A spatially adaptive spectral mixture analysis for mapping subpixel urban impervious surface distribution.
Remote Sens. Environ. 2013, 133, 62–70. [CrossRef]
Kavzoglu, T.; Mather, P. The use of backpropagating artificial neural networks in land cover classification. Int. J. Remote Sens.
2003, 24, 4907–4938. [CrossRef]
Chormanski, J.; Van de Voorde, T.; De Roeck, T.; Batelaan, O.; Canters, F. Improving distributed runoff prediction in urbanized
catchments with remote sensing based estimates of impervious surface cover. Sensors 2008, 8, 910–932. [CrossRef] [PubMed]
Mohapatra, R.P.; Wu, C. Subpixel imperviousness estimation with IKONOS imagery: An artificial neural network approach.
Remote Sens. Impervious Surfaces 2008, 21–37.
Weng, Q.; Hu, X. Medium spatial resolution satellite imagery for estimating and mapping urban impervious surfaces using
LSMA and ANN. IEEE Trans. Geosci. Remote Sens. 2008, 46, 2397–2406. [CrossRef]
Hu, X.; Weng, Q. Estimating impervious surfaces from medium spatial resolution imagery using the self-organizing map and
multi-layer perceptron neural networks. Remote Sens. Environ. 2009, 113, 2089–2102. [CrossRef]
Ji, C.Y. Land-use classification of remotely sensed data using Kohonen self-organizing feature map neural networks. Photogramm.
Eng. Remote Sens. 2000, 66, 1451–1460.
Li, Z.; Eastman, J.R. Commitment and typicality measurements for fuzzy ARTMAP neural network. In Proceedings of the
Geoinformatics 2006: Geospatial Information Science, International Society for Optics and Photonics, Wuhan, China, 28–29
October 2006; Volume 6420, p. 64201I.
Volpi, M.; Ferrari, V. Semantic segmentation of urban scenes by learning local class interactions. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition Workshops, Boston, MA, USA, 7–12 June 2015; pp. 1–9.
Volpi, M.; Tuia, D. Dense semantic labeling of subdecimeter resolution images with convolutional neural networks. IEEE Trans.
Geosci. Remote Sens. 2016, 55, 881–893. [CrossRef]
Audebert, N.; Le Saux, B.; Lefèvre, S. Segment-before-detect: Vehicle detection and classification through semantic segmentation
of aerial images. Remote Sens. 2017, 9, 368. [CrossRef]
Achanta, R.; Shaji, A.; Smith, K.; Lucchi, A.; Fua, P.; Süsstrunk, S. SLIC superpixels compared to state-of-the-art superpixel
methods. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 34, 2274–2282. [CrossRef] [PubMed]
Felzenszwalb, P.F.; Huttenlocher, D.P. Efficient graph-based image segmentation. Int. J. Comput. Vis. 2004, 59, 167–181. [CrossRef]
Remote Sens. 2021, 13, 808
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
82.
83.
84.
85.
86.
37 of 41
Farabet, C.; Couprie, C.; Najman, L.; LeCun, Y. Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach.
Intell. 2012, 35, 1915–1929. [CrossRef] [PubMed]
Pinherio, R.C.P.H.; Pedro, H. Recurrent convolutional neural networks for scene parsing. In Proceedings of the International
Conference of Machine Learning, Bejing, China, 22–24 June 2014.
Kampffmeyer, M.; Salberg, A.B.; Jenssen, R. Semantic segmentation of small objects and modeling of uncertainty in urban remote
sensing images using deep convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition Workshops, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 1–9.
Yang, H.; Yu, B.; Luo, J.; Chen, F. Semantic segmentation of high spatial resolution images with deep neural networks. GIScience
Remote Sens. 2019, 56, 749–768. [CrossRef]
Hinton, G.E.; Osindero, S.; Teh, Y.W. A fast learning algorithm for deep belief nets. Neural Comput. 2006, 18, 1527–1554. [CrossRef]
[PubMed]
Hinton, G.E.; Salakhutdinov, R.R. Reducing the dimensionality of data with neural networks. Science 2006, 313, 504–507.
[CrossRef]
Salakhutdinov, R.; Hinton, G. Deep boltzmann machines. In Proceedings of the Artificial Intelligence and Statistics, Clearwater
Beach, FL, USA, 16–18 April 2009; pp. 448–455.
Vincent, P.; Larochelle, H.; Lajoie, I.; Bengio, Y.; Manzagol, P.A.; Bottou, L. Stacked denoising autoencoders: Learning useful
representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 2010, 11, 3371–3408.
Deng, J.; Dong, W.; Socher, R.; Li, L.J.; Li, K.; Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of
the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255.
Zeiler, M.D.; Fergus, R. Visualizing and understanding convolutional networks. In European Conference on Computer Vision;
Springer: Berlin/Heidelberg, Germany, 2014; pp. 818–833.
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556.
Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with
convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June
2015; pp. 1–9.
He, K.; Zhang, X.; Ren, S.; Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.
In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1026–1034.
Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: A simple way to prevent neural networks
from overfitting. J. Mach. Learn. Res. 2014, 15, 1929–1958.
Ioffe, S.; Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv 2015,
arXiv:1502.03167.
Bottou, L. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010; Springer:
Berlin/Heidelberg, Germany, 2010; pp. 177–186.
Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980.
Mou, L.; Zhu, X.X. RiFCN: Recurrent network in fully convolutional network for semantic segmentation of high resolution
remote sensing images. arXiv 2018, arXiv:1805.02091.
Noh, H.; Hong, S.; Han, B. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE International
Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1520–1528.
Chen, L.C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L. Deeplab: Semantic image segmentation with deep convolutional
nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 40, 834–848. [CrossRef]
Ji, S.; Wei, S.; Lu, M. A scale robust convolutional neural network for automatic building extraction from aerial and satellite
imagery. Int. J. Remote Sens. 2019, 40, 3308–3322. [CrossRef]
Chen, G.; Li, C.; Wei, W.; Jing, W.; Woźniak, M.; Blažauskas, T.; Damaševičius, R. Fully convolutional neural network with
augmented atrous spatial pyramid pool and fully connected fusion path for high resolution remote sensing image segmentation.
Appl. Sci. 2019, 9, 1816. [CrossRef]
Chen, L.C.; Papandreou, G.; Schroff, F.; Adam, H. Rethinking atrous convolution for semantic image segmentation. arXiv 2017,
arXiv:1706.05587.
Zheng, S.; Jayasumana, S.; Romera-Paredes, B.; Vineet, V.; Su, Z.; Du, D.; Huang, C.; Torr, P.H. Conditional random fields as
recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13
December 2015; pp. 1529–1537.
Lin, G.; Shen, C.; Van Den Hengel, A.; Reid, I. Efficient piecewise training of deep structured models for semantic segmentation.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp.
3194–3203.
Liu, Y.; Piramanayagam, S.; Monteiro, S.T.; Saber, E. Semantic segmentation of multisensor remote sensing imagery with deep
ConvNets and higher-order conditional random fields. J. Appl. Remote Sens. 2019, 13, 016501. [CrossRef]
He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27-30 June 2016; pp. 770–778.
Wu, Z.; Shen, C.; Hengel, A.V.D. High-performance semantic segmentation using very deep fully convolutional networks. arXiv
2016, arXiv:1604.04339.
Remote Sens. 2021, 13, 808
87.
88.
89.
90.
91.
92.
93.
94.
95.
96.
97.
98.
99.
100.
101.
102.
103.
104.
105.
106.
107.
108.
109.
110.
111.
112.
113.
38 of 41
Wang, H.; Wang, Y.; Zhang, Q.; Xiang, S.; Pan, C. Gated convolutional neural network for semantic segmentation in highresolution images. Remote Sens. 2017, 9, 446. [CrossRef]
Zhao, H.; Shi, J.; Qi, X.; Wang, X.; Jia, J. Pyramid scene parsing network. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2881–2890.
Yu, B.; Yang, L.; Chen, F. Semantic segmentation for high spatial resolution remote sensing images based on convolution neural
network and pyramid pooling module. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2018, 11, 3252–3261. [CrossRef]
He, K.; Zhang, X.; Ren, S.; Sun, J. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans.
Pattern Anal. Mach. Intell. 2015, 37, 1904–1916. [CrossRef]
Chen, Q.; Wang, L.; Wu, Y.; Wu, G.; Guo, Z.; Waslander, S.L. Aerial imagery for roof segmentation: A large-scale dataset towards
automatic mapping of buildings. arXiv 2018, arXiv:1807.09532.
Audebert, N.; Le Saux, B.; Lefèvre, S. Semantic segmentation of earth observation data using multimodal and multi-scale deep
networks. In Asian Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2016; pp. 180–196.
Audebert, N.; Boulch, A.; Randrianarivo, H.; Le Saux, B.; Ferecatu, M.; Lefèvre, S.; Marlet, R. Deep learning for urban remote
sensing. In Proceedings of the 2017 IEEE Joint Urban Remote Sensing Event (JURSE), Dubai, United Arab Emirates, 6–8 March
2017; pp. 1–4.
Liu, Y.; Minh Nguyen, D.; Deligiannis, N.; Ding, W.; Munteanu, A. Hourglass-shapenetwork based semantic segmentation for
high resolution aerial imagery. Remote Sens. 2017, 9, 522. [CrossRef]
Chen, K.; Weinmann, M.; Sun, X.; Yan, M.; Hinz, S.; Jutzi, B.; Weinmann, M. Semantic Segmentation of Aerial Imagery Via
Multi-Scale Shuffling Convolutional Neural Networks with Deep Supervision. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf.
Sci. 2018, 4, 29–36. [CrossRef]
Panboonyuen, T.; Jitkajornwanich, K.; Lawawirojwong, S.; Srestasathiern, P.; Vateekul, P. Semantic segmentation on remotely
sensed images using an enhanced global convolutional network with channel attention and domain specific transfer learning.
Remote Sens. 2019, 11, 83. [CrossRef]
Shang, R.; Zhang, J.; Jiao, L.; Li, Y.; Marturi, N.; Stolkin, R. Multi-scale Adaptive Feature Fusion Network for Semantic
Segmentation in Remote Sensing Images. Remote Sens. 2020, 12, 872. [CrossRef]
Zhang, J.; Lin, S.; Ding, L.; Bruzzone, L. Multi-scale context aggregation for semantic segmentation of remote sensing images.
Remote Sens. 2020, 12, 701. [CrossRef]
Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International
Conference on Medical Image Computing and Computer-Assisted Intervention; Springer: Berlin/Heidelberg, Germany, 2015; pp.
234–241.
Guo, Z.; Shengoku, H.; Wu, G.; Chen, Q.; Yuan, W.; Shi, X.; Shao, X.; Xu, Y.; Shibasaki, R. Semantic segmentation for urban
planning maps based on U-Net. In Proceedings of the IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing
Symposium, Valencia, Spain, 22–27 July 2018; pp. 6187–6190.
Li, W.; He, C.; Fang, J.; Zheng, J.; Fu, H.; Yu, L. Semantic segmentation-based building footprint extraction using very highresolution satellite images and multi-source GIS data. Remote Sens. 2019, 11, 403. [CrossRef]
Pan, Z.; Xu, J.; Guo, Y.; Hu, Y.; Wang, G. Deep Learning Segmentation and Classification for Urban Village Using a Worldview
Satellite Image Based on U-Net. Remote Sens. 2020, 12, 1574. [CrossRef]
Yi, Y.; Zhang, Z.; Zhang, W.; Zhang, C.; Li, W.; Zhao, T. Semantic segmentation of urban buildings from vhr remote sensing
imagery using a deep convolutional neural network. Remote Sens. 2019, 11, 1774. [CrossRef]
Yue, K.; Yang, L.; Li, R.; Hu, W.; Zhang, F.; Li, W. TreeUNet: Adaptive Tree convolutional neural networks for subdecimeter aerial
image segmentation. ISPRS J. Photogramm. Remote Sens. 2019, 156, 1–13. [CrossRef]
Li, R.; Liu, W.; Yang, L.; Sun, S.; Hu, W.; Zhang, F.; Li, W. Deepunet: A deep fully convolutional network for pixel-level sea-land
segmentation. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2018, 11, 3954–3962. [CrossRef]
Robinson, Y.H.; Vimal, S.; Khari, M.; Hernández, F.C.L.; Crespo, R.G. Tree-based convolutional neural networks for object
classification in segmented satellite images. Int. J. High Perform. Comput. Appl. 2020. [CrossRef]
Liu, W.; Su, F.; Jin, X.; Li, H.; Qin, R. Bispace Domain Adaptation Network for Remotely Sensed Semantic Segmentation. IEEE
Trans. Geosci. Remote Sens. 2020. [CrossRef]
Diakogiannis, F.I.; Waldner, F.; Caccetta, P.; Wu, C. Resunet-a: A deep learning framework for semantic segmentation of remotely
sensed data. ISPRS J. Photogramm. Remote Sens. 2020, 162, 94–114. [CrossRef]
Dong, R.; Bai, L.; Li, F. SiameseDenseU-Net-based Semantic Segmentation of Urban Remote Sensing Images. Math. Probl. Eng.
2020, 2020. [CrossRef]
Dong, R.; Pan, X.; Li, F. DenseU-net-based semantic segmentation of small objects in urban remote sensing images. IEEE Access
2019, 7, 65347–65356. [CrossRef]
Badrinarayanan, V.; Kendall, A.; Cipolla, R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation.
IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 2481–2495. [CrossRef]
Marmanis, D.; Schindler, K.; Wegner, J.D.; Galliani, S.; Datcu, M.; Stilla, U. Classification with an edge: Improving semantic image
segmentation with boundary detection. ISPRS J. Photogramm. Remote Sens. 2018, 135, 158–172. [CrossRef]
Cao, R.; Qiu, G. Urban land use classification based on aerial and ground images. In Proceedings of the 2018 IEEE International
Conference on Content-Based Multimedia Indexing (CBMI), Bordeaux, France, 25–27 June 2018; pp. 1–6.
Remote Sens. 2021, 13, 808
39 of 41
114. Bischke, B.; Helber, P.; Folz, J.; Borth, D.; Dengel, A. Multi-task learning for segmentation of building footprints with deep neural
networks. In Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 22–29 September
2019; pp. 1480–1484.
115. Sun, Y.; Tian, Y.; Xu, Y. Problems of encoder-decoder frameworks for high-resolution remote sensing image segmentation:
Structural stereotype and insufficient learning. Neurocomputing 2019, 330, 297–304. [CrossRef]
116. Abdollahi, A.; Pradhan, B.; Alamri, A.M. An Ensemble Architecture of Deep Convolutional Segnet and Unet Networks for
Building Semantic Segmentation from High-resolution Aerial Images. Geocarto Int. 2020, 1–13. [CrossRef]
117. Sariturk, B.; Bayram, B.; Duran, Z.; Seker, D.Z. Feature Extraction from Satellite Images Using Segnet and Fully Convolutional
Networks (FCN). Int. J. Eng. Geosci. 2020, 5, 138–143. [CrossRef]
118. Hong, S.; Noh, H.; Han, B. Decoupled deep neural network for semi-supervised semantic segmentation. Advances in neural
information processing systems. arXiv 2015, arXiv:1506.04924.
119. Pinheiro, P.O.; Lin, T.Y.; Collobert, R.; Dollár, P. Learning to refine object segments. In European Conference on Computer Vision;
Springer: Berlin/Heidelberg, Germany, 2016; pp. 75–91.
120. Pinheiro, P.O.; Collobert, R.; Dollár, P. Learning to segment object candidates. Advances in Neural Information Processing
Systems. arXiv 2015, arXiv:1506.06204.
121. Kemker, R.; Salvaggio, C.; Kanan, C. Algorithms for semantic segmentation of multispectral remote sensing imagery using deep
learning. ISPRS J. Photogramm. Remote Sens. 2018, 145, 60–77. [CrossRef]
122. Lin, G.; Milan, A.; Shen, C.; Reid, I. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp.
1925–1934.
123. Chen, L.C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-decoder with atrous separable convolution for semantic image
segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018;
pp. 801–818.
124. Du, S.; Du, S.; Liu, B.; Zhang, X. Incorporating DeepLabv3+ and object-based image analysis for semantic segmentation of very
high resolution remote sensing images. Int. J. Digit. Earth 2020, 1–22. [CrossRef]
125. Chen, G.; Zhang, X.; Wang, Q.; Dai, F.; Gong, Y.; Zhu, K. Symmetrical dense-shortcut deep fully convolutional networks for
semantic segmentation of very-high-resolution remote sensing images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2018,
11, 1633–1644. [CrossRef]
126. Li, Y.; Xu, L.; Rao, J.; Guo, L.; Yan, Z.; Jin, S. A Y-Net deep learning method for road segmentation using high-resolution visible
remote sensing images. Remote Sens. Lett. 2019, 10, 381–390. [CrossRef]
127. Benjdira, B.; Bazi, Y.; Koubaa, A.; Ouni, K. Unsupervised domain adaptation using generative adversarial networks for semantic
segmentation of aerial images. Remote Sens. 2019, 11, 1369. [CrossRef]
128. Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial
nets. In Advances in Neural Information Processing Systems; ACM: New York, NY, USA, 2014; pp. 2672–2680.
129. Lin, D.; Fu, K.; Wang, Y.; Xu, G.; Sun, X. MARTA GANs: Unsupervised representation learning for remote sensing image
classification. IEEE Geosci. Remote Sens. Lett. 2017, 14, 2092–2096. [CrossRef]
130. Zhan, Y.; Hu, D.; Wang, Y.; Yu, X. Semisupervised hyperspectral image classification based on generative adversarial networks.
IEEE Geosci. Remote Sens. Lett. 2017, 15, 212–216. [CrossRef]
131. Torrey, L.; Shavlik, J. Transfer learning. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods,
and Techniques; IGI Global: Hershey, PA, USA, 2010; pp. 242–264.
132. Weiss, K.; Khoshgoftaar, T.M.; Wang, D. A survey of transfer learning. J. Big Data 2016, 3, 9. [CrossRef]
133. Tan, C.; Sun, F.; Kong, T.; Zhang, W.; Yang, C.; Liu, C. A survey on deep transfer learning. In International Conference on Artificial
Neural Networks; Springer: Berlin/Heidelberg, Germany, 2018; pp. 270–279.
134. Du, Z.; Yang, J.; Ou, C.; Zhang, T. Smallholder crop area mapped with a semantic segmentation deep learning method. Remote
Sens. 2019, 11, 888. [CrossRef]
135. Wurm, M.; Stark, T.; Zhu, X.X.; Weigand, M.; Taubenböck, H. Semantic segmentation of slums in satellite images using transfer
learning on fully convolutional neural networks. ISPRS J. Photogramm. Remote Sens. 2019, 150, 59–69. [CrossRef]
136. Lv, Q.; Dou, Y.; Niu, X.; Xu, J.; Xu, J.; Xia, F. Urban land use and land cover classification using remotely sensed SAR data through
deep belief networks. J. Sens. 2015, 2015. [CrossRef]
137. Nogueira, K.; Dalla Mura, M.; Chanussot, J.; Schwartz, W.R.; dos Santos, J.A. Learning to semantically segment high-resolution
remote sensing images. In Proceedings of the 2016 23rd IEEE International Conference on Pattern Recognition (ICPR), Cancun,
Mexico, 4–8 December 2016; pp. 3566–3571.
138. Zhang, Y.; Chi, M. Mask-R-FCN: A Deep Fusion Network for Semantic Segmentation. IEEE Access 2020, 8, 155753–155765.
[CrossRef]
139. Poomani, M.; Sutha, J.; Soundar, K.R. Wiener filter based deep convolutional network approach for classification of satellite
images. J. Ambient. Intell. Humaniz. Comput. 2020. [CrossRef]
140. Mattyus, G.; Wang, S.; Fidler, S.; Urtasun, R. Enhancing road maps by parsing aerial images around the world. In Proceedings of
the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1689–1697.
Remote Sens. 2021, 13, 808
40 of 41
141. Hackel, T.; Wegner, J.D.; Schindler, K. Fast semantic segmentation of 3D point clouds with strongly varying density. ISPRS Ann.
Photogramm. Remote Sens. Spat. Inf. Sci. 2016, 3, 177–184. [CrossRef]
142. Azimi, S.M.; Fischer, P.; Körner, M.; Reinartz, P. Aerial LaneNet: Lane-marking semantic segmentation in aerial imagery
using wavelet-enhanced cost-sensitive symmetric fully convolutional neural networks. IEEE Trans. Geosci. Remote Sens. 2018,
57, 2920–2938. [CrossRef]
143. Mi, L.; Chen, Z. Superpixel-enhanced deep neural forest for remote sensing image semantic segmentation. ISPRS J. Photogramm.
Remote Sens. 2020, 159, 140–152. [CrossRef]
144. Paisitkriangkrai, S.; Sherrah, J.; Janney, P.; Hengel, V.D. Effective semantic pixel labelling with convolutional networks and
conditional random fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops,
Boston, MA, USA, 7–12 June 2015; pp. 36–43.
145. Maggiori, E.; Tarabalka, Y.; Charpiat, G.; Alliez, P. High-resolution semantic labeling with convolutional neural networks. arXiv
2016, arXiv:1611.01962.
146. Audebert, N.; Le Saux, B.; Lefèvre, S. Beyond RGB: Very high resolution urban remote sensing with multimodal deep networks.
ISPRS J. Photogramm. Remote Sens. 2018, 140, 20–32. [CrossRef]
147. Piramanayagam, S.; Saber, E.; Schwartzkopf, W.; Koehler, F.W. Supervised classification of multisensor remotely sensed images
using a deep learning framework. Remote Sens. 2018, 10, 1429. [CrossRef]
148. Pan, X.; Gao, L.; Marinoni, A.; Zhang, B.; Yang, F.; Gamba, P. Semantic labeling of high resolution aerial imagery and LiDAR data
with fine segmentation network. Remote Sens. 2018, 10, 743. [CrossRef]
149. Bergado, J.R.; Persello, C.; Stein, A. Recurrent multiresolution convolutional networks for VHR image classification. IEEE Trans.
Geosci. Remote Sens. 2018, 56, 6361–6374. [CrossRef]
150. Liu, Y.; Piramanayagam, S.; Monteiro, S.T.; Saber, E. Dense semantic labeling of very-high-resolution aerial imagery and lidar
with fully-convolutional neural networks and higher-order CRFs. In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition Workshops, Honolulu, HI, USA, 21–26 July 2017; pp. 76–85.
151. Liu, Y.; Fan, B.; Wang, L.; Bai, J.; Xiang, S.; Pan, C. Semantic labeling in very high resolution images via a self-cascaded
convolutional neural network. ISPRS J. Photogramm. Remote Sens. 2018, 145, 78–95. [CrossRef]
152. Marcu, A.; Leordeanu, M. Dual local-global contextual pathways for recognition in aerial imagery. arXiv 2016, arXiv:1605.05462.
153. Marcu, A.E.; Leordeanu, M. Object contra context: Dual local-global semantic segmentation in aerial images. In Proceedings of
the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017.
154. Li, P.; Lin, Y.; Schultz-Fellenz, E. Contextual Hourglass Network for Semantic Segmentation of High Resolution Aerial Imagery.
arXiv 2018, arXiv:1810.12813.
155. Nogueira, K.; Dalla Mura, M.; Chanussot, J.; Schwartz, W.R.; dos Santos, J.A. Dynamic multicontext segmentation of remote
sensing images based on convolutional networks. IEEE Trans. Geosci. Remote Sens. 2019, 57, 7503–7520. [CrossRef]
156. Zhao, W.; Du, S.; Wang, Q.; Emery, W.J. Contextually guided very-high-resolution imagery classification with semantic segments.
ISPRS J. Photogramm. Remote Sens. 2017, 132, 48–60. [CrossRef]
157. Marmanis, D.; Wegner, J.D.; Galliani, S.; Schindler, K.; Datcu, M.; Stilla, U. Semantic segmentation of aerial images with an
ensemble of CNSS. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2016 2016, 3, 473–480. [CrossRef]
158. Saito, S.; Aoki, Y. Building and road detection from large aerial imagery. In Image Processing: Machine Vision Applications VIII;
International Society for Optics and Photonics: Bellingham, WA, USA, 2015; Volume 9405, p. 94050K.
159. Saito, S.; Yamashita, T.; Aoki, Y. Multiple object extraction from aerial imagery with convolutional neural networks. Electron.
Imaging 2016, 2016, 1–9. [CrossRef]
160. Griffiths, D.; Boehm, J. Improving public data for building segmentation from Convolutional Neural Networks (CNNs) for fused
airborne lidar and image data using active contours. ISPRS J. Photogramm. Remote Sens. 2019, 154, 70–83. [CrossRef]
161. Kaiser, P.; Wegner, J.D.; Lucchi, A.; Jaggi, M.; Hofmann, T.; Schindler, K. Learning aerial image segmentation from online maps.
IEEE Trans. Geosci. Remote Sens. 2017, 55, 6054–6068. [CrossRef]
162. Li, J.; Ding, W.; Li, H.; Liu, C. Semantic segmentation for high-resolution aerial imagery using multi-skip network and Markov
random fields. In Proceedings of the 2017 IEEE International Conference on Unmanned Systems (ICUS), Beijing, China, 27–29
October 2017; pp. 12–17.
163. Volpi, M.; Tuia, D. Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images. ISPRS J.
Photogramm. Remote Sens. 2018, 144, 48–60. [CrossRef]
164. Shi, Y.; Li, Q.; Zhu, X.X. Building segmentation through a gated graph convolutional neural network with deep structured feature
embedding. ISPRS J. Photogramm. Remote Sens. 2020, 159, 184–197. [CrossRef]
165. Tschannen, M.; Cavigelli, L.; Mentzer, F.; Wiatowski, T.; Benini, L. Deep structured features for semantic segmentation. In
Proceedings of the 2017 25th European Signal Processing Conference (EUSIPCO), Kos, Greece, 28 August–2 September 2017;
pp. 61–65.
166. ISPRS Vaihingen 2D Semantic Labeling Dataset. Available online: http://www2.isprs.org/commissions/comm3/wg4/2d-semlabel-vaihingen.html (accessed on 21 December 2020).
167. ISPRS Potsdam 2D Semantic Labeling Dataset. Available online: http://www2.isprs.org/commissions/comm3/wg4/2d-semlabel-potsdam.html (accessed on 21 December 2020).
Remote Sens. 2021, 13, 808
41 of 41
168. Moser, G.; Tuia, D.; Shimoni, M. 2015 IEEE GRSS data fusion contest: Extremely high resolution LidAR and optical data [technical
committees]. IEEE Geosci. Remote Sens. Mag. 2015, 3, 40–41. [CrossRef]
169. Qin, Y.; Wu, Y.; Li, B.; Gao, S.; Liu, M.; Zhan, Y. Semantic segmentation of building roof in dense urban environment with deep
convolutional neural network: A case study using GF2 VHR imagery in China. Sensors 2019, 19, 1164. [CrossRef] [PubMed]
170. Mnih, V. Machine Learning for Aerial Image Labeling; Citeseer: Forest Grove, OR, USA, 2013.
171. Gerke, M. Use of the Stair Vision Library within the ISPRS 2D Semantic Labeling Benchmark (Vaihingen). 2014. Available online:
https://doi.org/10.13140/2.1.5015.9683 (accessed on 21 December 2020).
172. Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4700–4708.
173. Bisong, E. Google colaboratory. In Building Machine Learning and Deep Learning Models on Google Cloud Platform; Springer:
lBerlin/Heidelberg, Germany, 2019; pp. 59–64.
174. Paisitkriangkrai, S.; Sherrah, J.; Janney, P.; Van Den Hengel, A. Semantic labeling of aerial and satellite imagery. IEEE J. Sel. Top.
Appl. Earth Obs. Remote Sens. 2016, 9, 2868–2881. [CrossRef]
175. Kipf, T.N.; Welling, M. Semi-supervised classification with graph convolutional networks. arXiv 2016, arXiv:1609.02907.