Evaluation of Model Validation Techniques in Land Cover Dynamics
"> Figure 1
<p>Location of Khulna City in Bangladesh. Source: Banglapedia, National Encyclopedia of Bangladesh, 2012.</p> "> Figure 2
<p>Location of the study area (areas of Khulna City Corporation (KCC) and adjoining fringe areas) on Landsat satellite images. (Image source: US Geological Survey (USGS), 2012 and Shapefile source: Khulna City Corporation, 2012).</p> "> Figure 3
<p>Landcover maps of the study area.</p> "> Figure 4
<p>Percentages of presence of land cover types (1989–2009).</p> "> Figure 5
<p>Simulated land cover maps of Khulna City (2009).</p> "> Figure 6
<p>Per category comparison method (Base Map (2009) <span class="html-italic">vs.</span> St_Markov (2009)).</p> "> Figure 7
<p>Per category comparison method (Base Map (2009) <span class="html-italic">vs.</span> CA_Markov (2009)).</p> "> Figure 8
<p>Per category comparison method (Base Map (2009) <span class="html-italic">vs.</span> MLP_Markov (2009)).</p> "> Figure 9
<p>Maps of the components of agreement and disagreement.</p> "> Figure 10
<p>Fuzzy similarity comparison maps.</p> ">
Abstract
:1. Introduction
2. Materials and Methods
2.1. Study Area
2.2. Remote Sensing Data
Land Cover Type | Description |
---|---|
Built-up Area | All residential, commercial and industrial areas, villages, settlements and transportation infrastructure. |
Water Body | River, sea, permanent open water, lakes, ponds, canals and reservoirs. |
Vegetation | Trees, shrub lands and semi natural vegetation: deciduous, coniferous, and mixed forest, palms, orchard, herbs, climbers, gardens, inner-city recreational areas, parks and playgrounds, grassland and vegetable lands. |
Low Land | Permanent and seasonal wetlands, low-lying areas, marshy land, rills and gully, swamps, mudflats, all cultivated areas including urban agriculture; crop fields and rice-paddies. |
Fallow Land | Fallow land, earth and sand land in-fillings, construction sites, developed land, excavation sites, solid waste landfills, open space, bare soils. |
2.3. Base Map Preparation
2.4. Accuracy Assessment
2.5. Simulating Land Cover Maps
3. Results and Discussion
3.1. Per Category Method
Land Cover Type | In both Maps | In none of the Maps | Only in Map 1 | Only in Map 2 |
---|---|---|---|---|
Builtup Area | 60,574 | 386,747 | 154,005 | 86,098 |
Water Body | 12,799 | 655,963 | 8,240 | 10,422 |
Vegetation | 87,432 | 497,671 | 43,091 | 59,230 |
Low Land | 103,479 | 336,886 | 98,116 | 148,943 |
Fallow Land | 60,265 | 509,554 | 59,423 | 58,182 |
Land Cover Type | In both Maps | In none of the Maps | Only in Map 1 | Only in Map 2 |
---|---|---|---|---|
Builtup Area | 137,182 | 463,419 | 77,397 | 9,426 |
Water Body | 17,637 | 660,736 | 3,402 | 5,649 |
Vegetation | 126,870 | 537,014 | 3,653 | 19,887 |
Low Land | 192,860 | 426,412 | 8,735 | 59,417 |
Fallow Land | 107,069 | 556,309 | 12,619 | 11,427 |
Land Cover Type | In both Maps | In none of the Maps | Only in Map 1 | Only in Map 2 |
---|---|---|---|---|
Builtup Area | 217,071 | 445,337 | 13,508 | 11,508 |
Water Body | 20,018 | 664,326 | 2,071 | 1,009 |
Vegetation | 127,789 | 552,167 | 4,738 | 2,730 |
Low Land | 194,399 | 477,533 | 9,296 | 6,196 |
Fallow Land | 118,568 | 565,267 | 2,287 | 1,302 |
3.2. Location and Quantity Accuracies Using Kappa Statistics
Kappa Indices | St_Markov | CA_Markov | MLP_Markov |
---|---|---|---|
Kstandard | 0.3001 | 0.7959 | 0.9320 |
Kno | 0.3402 | 0.8076 | 0.9363 |
Klocation | 0.3462 | 0.9184 | 0.9457 |
Kquantity | 0.8672 | 0.9021 | 0.9744 |
3.3. Errors Due to Quantity and Allocation
Name of Component | St_Markov | CA_Markov | MLP_Markov |
---|---|---|---|
Disagreement due to Quantity | 0.1006 | 0.1006 | 0.0074 |
Disagreement at Grid Cell Level | 0.4273 | 0.0533 | 0.0428 |
Agreement at Grid Cell Level | 0.2263 | 0.6003 | 0.6991 |
Agreement due to Quantity | 0.0458 | 0.0458 | 0.0507 |
Agreement due to Chance | 0.2000 | 0.2000 | 0.2000 |
3.4. Comparison of Three Maps
3.5. Fuzzy Set Theory
Name of Component | St_Markov (%) | CA_Markov (%) | MLP_Markov (%) |
---|---|---|---|
Persistence Simulated Correctly | 19.88002 | 22.44001 | 94.9062 |
Change Simulated Correctly | 12.52473 | 1.371206 | 2.187733 |
Total Agreement | 32.404745 | 23.8112 | 97.09393 |
Change Simulated As Persistence | 38.98642 | 46.72851 | 0.359167 |
Persistence Simulated As Change | 19.38833 | 9.371218 | 2.187733 |
Change Simulated As Change to Wrong Category | 9.22051 | 20.08906 | 0.359167 |
Total Disagreement | 67.5952 | 76.1887 | 2.90606 |
Modeling Method | | Average Similarity |
---|---|---|
St_Markov | 0.304 | 0.701 |
CA_Markov | 0.862 | 0.924 |
MLP_Markov | 0.953 | 0.974 |
4. Conclusions
5. Future Research
Acknowledgments
Conflict of Interest
References
- Pontius, R.G., Jr.; Boersma, W.; Castella, J.-C.; Clarke, K.; de Nijs, T.; Dietzel, C.; Duan, Z.; Fotsing, E.; Goldstein, N.; Kok, K.; et al. Comparing the input, output, and validation maps for several models of land change. Ann. Reg. Sci. 2008, 42, 11–47. [Google Scholar] [CrossRef]
- Chen, H.; Pontius, R.G., Jr. Diagnostic tools to evaluate a spatial land change projection along a gradient of an explanatory variable. Landsc. Ecol. 2010, 25, 1319–1331. [Google Scholar] [CrossRef]
- Visser, H.; de Nijs, T. The map comparison kit. Environ. Model. Softw. 2006, 21, 346–358. [Google Scholar] [CrossRef]
- Pontius, R.G., Jr.; Walker, R.; Yao-Kumah, R.; Arima, E.; Aldrich, S.; Caldas, M.; Vergara, D. Accuracy assessment for a simulation model of Amazonian deforestation. Ann. Assn. Amer. Geogr. 2007, 97, 677–695. [Google Scholar] [CrossRef]
- Hagen, A. Fuzzy set approach to assessing similarity of categorical maps. Int. J. Geogr. Inf. Sci. 2003, 17, 235–249. [Google Scholar] [CrossRef]
- Metternicht, G. Change detection assessment using fuzzy sets and remotely sensed data: An application of topographic map revision. ISPRS J. Photogramm. 1999, 54, 221–233. [Google Scholar] [CrossRef]
- Boots, B.; Csillag, F. Categorical maps, comparisons, and confidence. J. Geograph. Syst. 2006, 8, 109–118. [Google Scholar] [CrossRef]
- Hagen-Zanker, A. An improved Fuzzy Kappa statistic that accounts for spatial autocorrelation. Int. J. Geogr. Inf. Sci. 2009, 23, 61–73. [Google Scholar] [CrossRef]
- Richter, O.; Söndgerath, D. Parameter Estimation in Ecology: The Link between Data and Models; VCH Publishers: New York, NY, USA, 1990. [Google Scholar]
- Gardner, R.; Urban, D. Model Validation and Testing: Past Lessons, Present Concerns, Future Prospects. In Models in Ecosystem Science; Canham, C.D., Cole, J.J., Lauenroth, W.K., Eds.; Princeton University Press: Princeton, NJ, USA, 2004; pp. 184–203. [Google Scholar]
- Pontius, R.G., Jr.; Silvia, H.P. Assessing a predictive model of land change using uncertain data. Environ. Model. Softw. 2010, 25, 299–309. [Google Scholar] [CrossRef]
- Van Rompaey, A.J.J.; Govers, C. Data quality and model complexity for regional scale soil erosion prediction. Int. J. Geogr. Inf. Sci. 2002, 16, 663–680. [Google Scholar] [CrossRef]
- Foody, G.M. The impact of imperfect ground reference data on the accuracy of land cover change estimation. Int. J. Remote Sens. 2009, 30, 3275–3281. [Google Scholar] [CrossRef]
- Rykiel, E.J. Testing ecological models: The meaning of validation. Ecol. Model. 1996, 90, 229–244. [Google Scholar] [CrossRef]
- Vliet, J.V.; Bregt, A.K.; Hagen-Zanker, A. Revisiting Kappa to account for change in the accuracy assessment of land-use change models. Ecol. Model. 2011, 222, 1367–1375. [Google Scholar] [CrossRef]
- Hagen-Zanker, A.; Martens, P. Map Comparison Methods for Comprehensive Assessment of Geosimulation Models. In Proceedings of the International Conference on Computational Science and Its Applications, Perugia, Italy, 30 June–3 July 2008; Gervasi, O., Murgante, B., Lagana, A., Taniar, D., Mun, Y., Gavrilova, M., Eds.; Springer: Berlin, Germany; pp. 194–209.
- Jensen, J.R.; Ramsey, E.W.; Halkard, E.M.; Christensen, E.J.; Sharitz, R.R. Inland wetland change detection using aircraft MSS data. Photogramm. Eng. Remote Sensing 1987, 53, 521–529. [Google Scholar]
- Power, C.; Simms, A.; White, R. Hierarchical fuzzy pattern matching for the regional comparison of land use maps. Int. J. Geogr. Inf. Sci. 2001, 15, 77–100. [Google Scholar] [CrossRef]
- White, R.; Engelen, G.; Injee, I. The use of constrained cellular automata for high-resolution modelling of urban land use dynamics. Environ. Plan. 1997, 24, 323–343. [Google Scholar]
- Hagen-Zanker, A.; Lajoie, G. Neutral models of landscape change as benchmarks in the assessment of model performance. Landscape Urban Plan. 2008, 86, 284–296. [Google Scholar] [CrossRef]
- Fewster, R.M.; Buckland, S.T. Similarity indices for spatial ecological data. Biometrics 2001, 57, 495–501. [Google Scholar] [CrossRef]
- Ebert, E.E.; McBride, J.L. Verification of precipitation in weather systems: Determination of systematic errors. J. Hydrol. 2000, 239, 179–202. [Google Scholar] [CrossRef]
- Pijanowski, B.C.; Brown, D.G.; Shellito, B.A.; Manik, G.A. Using neural networks and GIS to forecast land use changes: A land transformation model. Comput. Environ. Urban Syst. 2002, 26, 553–575. [Google Scholar] [CrossRef]
- Briggs, W.M.; Levine, R.A. Wavelets and field forecast verification. Mon. Weather Rev. 1997, 125, 1329–1341. [Google Scholar] [CrossRef]
- Barredo, J.I.; Demicheli, L. Urban sustainability in developing countries’ megacities: Modelling and predicting future urban growth in Lagos. Cities 2003, 20, 297–310. [Google Scholar] [CrossRef]
- Turner, M.G.; Costanza, R.; Sklar, F.H. Methods to evaluate the performance of spatial simulation-models. Ecol. Model 1989, 48, 1–18. [Google Scholar] [CrossRef]
- Urban Strategy. In Structure Plan, Master Plan and Detailed Area Plan (2001–2020) for Khulna City; Aqua-Sheltech Consortium, Khulna Development Authority: Khulna, Bangladesh, 2002; Volume I.
- Ahmed, B.; Ahmed, R. Modeling urban land cover growth dynamics using multi-temporal satellite images: A case study of Dhaka, Bangladesh. ISPRS Int. J. Geo-Inf. 2012, 1, 3–31. [Google Scholar] [CrossRef]
- Liu, H.; Zhou, Q. Accuracy analysis of remote sensing change detection by rule-based rationality evaluation with post-classification comparison. Int. J. Remote Sens. 2004, 5, 1037–1050. [Google Scholar] [CrossRef]
- Eastman, J.R. IDRISI Taiga Guide to GIS and Image Processing, Manual Version 16.02 (Software); Clark Labs: Worcester, MA, USA, 2009. [Google Scholar]
- Basharin, G.P.; Langville, A.N.; Naumov, V.A. The life and work of A.A. Markov. Linear Algebr. Appl. 2004, 386, 3–26. [Google Scholar] [CrossRef]
- Weng, Q. Land use change analysis in the Zhujiang Delta of China using satellite remote sensing, GIS and stochastic modelling. J. Environ. Manage. 2002, 64, 273–284. [Google Scholar] [CrossRef]
- Maerivoet, S.; Moor, B.D. Cellular automata models of road traffic. Phys. Rep. 2005, 419, 1–64. [Google Scholar] [CrossRef]
- Malczewski, J. GIS-based land-use suitability analysis: A critical overview. Prog. Plan. 2004, 62, 3–65. [Google Scholar] [CrossRef]
- Karul, C.; Soyupak, S. A Comparison between Neural Network Based and Multiple Regression Models for Chlorophyll-A Estimation. In Ecological Informatics; Recknagel, F., Ed.; Springer: Berlin, Germany, 2006; pp. 309–323. [Google Scholar]
- Atkinson, P.M.; Tatnall, A.R.L. Introduction neural networks in remote sensing. Int. J. Remote Sens. 1997, 18, 699–709. [Google Scholar] [CrossRef]
- Vliet, J.V. Map Comparison Kit 3 User Manual, Manual Version 3.2 (Software); Research Institute for Knowledge Systems BV: Maastricht, The Netherlands, 2009. [Google Scholar]
- Pontius, R.G., Jr. Quantification error versus location error in the comparison of categorical maps. Photogramm. Eng. Remote Sensing 2000, 66, 1011–1016. [Google Scholar]
- Pontius, R.G., Jr. Statistical methods to partition effects of quantity and location during comparison of categorical maps at multiple resolutions. Photogramm. Eng. Remote Sensing 2002, 68, 1041–1049. [Google Scholar]
- Cohen, J. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 1960, 20, 37–46. [Google Scholar] [CrossRef]
- Kitada, K.; Fukuyama, K. Land-use and land-cover mapping using a gradable classification method. Remote Sens. 2012, 4, 1544–1558. [Google Scholar] [CrossRef]
- Ahmed, B. Modelling spatio-temporal urban land cover growth dynamics using remote sensing and GIS techniques: A case study of Khulna City. J. Bangladesh Instit. Plan. 2011, 4, 16–33. [Google Scholar]
- Long, J.B.; Giri, C. Mapping the Philippines’ mangrove forests using landsat imagery. Sensors 2011, 11, 2972–2981. [Google Scholar] [CrossRef]
- Pontius, R.G., Jr.; Millones, M. Death to Kappa: Birth of quantity disagreement and allocation disagreement for accuracy assessment. Int. J. Remote Sens. 2011, 32, 4407–4429. [Google Scholar] [CrossRef]
- Pontius, R.G., Jr.; Millones, M. Problems and Solutions for Kappa-Based Indices of Agreement. In Proceedings of the Conference Studying, Modeling and Sense Making of Planet Earth, Mytilene, Greece, 1–6 June 2008.
- Pontius, R.G., Jr.; Peethambaram, S.; Castella, J.C. Comparison of three maps at multiple resolutions: A case study of land change simulation in Cho Don District, Vietnam. Ann. Assn. Amer. Geogr. 2011, 101, 45–62. [Google Scholar] [CrossRef]
- Singh, A. Digital change detection using remote-sensing data. Int. J. Remote Sens. 1989, 10, 989–1003. [Google Scholar] [CrossRef]
- Mas, J.F. Monitoring land-cover changes: A comparison of change detection techniques. Int. J. Remote Sens. 1999, 20, 139–152. [Google Scholar] [CrossRef]
- Jensen, J.R. Urban change detection mapping using Landsat digital data. Amer. Cartographer 1981, 8, 127–147. [Google Scholar] [CrossRef]
- Macleod, R.D.; Congalton, R.G. A quantitative comparison of change detection algorithms for monitoring eelgrass from remotely sensed data. Photogramm. Eng. Remote Sensing 1998, 64, 207–216. [Google Scholar]
- Zadeh, L.A. Fuzzy sets. Inf. Control 1965, 8, 338–353. [Google Scholar] [CrossRef]
- Bandemer, H.; Gottwald, S. Fuzzy Sets, Fuzzy Logic, Fuzzy Methods with Applications; Wiley: New York, NY, USA, 1995. [Google Scholar]
- Cheng, T.; Molenaar, M.; Lin, H. Formalizing fuzzy objects from uncertain classification results. Int. J. Geogr. Inf. Sci. 2001, 15, 27–42. [Google Scholar] [CrossRef]
- Lewis, H.G.; Brown, M. A generalized confusion matrix for assessing area estimates from remotely sensed data. Int. J. Remote Sens. 2001, 22, 3223–3235. [Google Scholar] [CrossRef]
- Hagen-Zanker, A. Comparing Continuous Valued Raster Data: A Cross Disciplinary Literature Scan; Research Institute for Knowledge Systems BV: Maastricht, The Netherlands, 2006. [Google Scholar]
© 2013 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).
Share and Cite
Ahmed, B.; Ahmed, R.; Zhu, X. Evaluation of Model Validation Techniques in Land Cover Dynamics. ISPRS Int. J. Geo-Inf. 2013, 2, 577-597. https://doi.org/10.3390/ijgi2030577
Ahmed B, Ahmed R, Zhu X. Evaluation of Model Validation Techniques in Land Cover Dynamics. ISPRS International Journal of Geo-Information. 2013; 2(3):577-597. https://doi.org/10.3390/ijgi2030577
Chicago/Turabian StyleAhmed, Bayes, Raquib Ahmed, and Xuan Zhu. 2013. "Evaluation of Model Validation Techniques in Land Cover Dynamics" ISPRS International Journal of Geo-Information 2, no. 3: 577-597. https://doi.org/10.3390/ijgi2030577