Abstract
Dutch water authorities are responsible for, among others, the management of water levels in waterways. To perform their task properly, it is important that data is of high quality. We compare several univariate and multivariate methods for real time outlier detection in time series data of water sensors from Dutch water authority “Aa en Maas”. Their performance is assessed by measuring how well they detect simulated spike, jump and drift outliers. This approach allowed us to uncover the outlier parameter values (i.e. drift or jump magnitude) at which certain detection thresholds are reached. The experiments show that the outliers are best detected by multivariate (as opposed to univariate) models, and that a multi-layer perceptron quantile regression (QR-MLP) model is best able to capture these multivariate relations. In addition to simulated outliers, the QR-MLP model is able to detect real outliers as well. Moreover, specific rules for each outlier category are not needed. In sum, QR-MLP models are well-suited to detect outliers without supervision.
Made possible by Ynformed and Waterschap Aa en Maas.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
This data is not used in our main experiments.
References
Aggarwal, C.C.: Data Mining: The Textbook. Springer, New York (2015). https://doi.org/10.1007/978-3-319-14142-8
Chatfield, C.: The Analysis of Time Series: An Introduction. Chapman and Hall/CRC, London (2003)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)
Ding, Z., Fei, M.: An anomaly detection approach based on isolation forest algorithm for streaming data using sliding window. IFAC Proc. Vol. 46(20), 12–17 (2013)
García, S., Fernández, A., Luengo, J., Herrera, F.: Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. Inf. Sci. 180(10), 2044–2064 (2010)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Junger, W., de Leon, A.P.: mtsdi: Multivariate time series data imputation (2012). https://cran.r-project.org/web/packages/mtsdi/index.html. R package version 0.3.5
Koenker, R., Hallock, K.F.: Quantile regression. J. Econ. Perspect. 15(4), 143–156 (2001)
Leigh, C., et al.: A framework for automated anomaly detection in high frequency water-quality data from in situ sensors. Sci. Total. Environ. 664, 885–898 (2019)
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: a novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. 18(1), 6765–6816 (2017)
Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forest. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 413–422. IEEE (2008)
Meinshausen, N.: Quantile regression forests. J. Mach. Learn. Res. 7, 983–999 (2006)
Perelman, L., Arad, J., Housh, M., Ostfeld, A.: Event detection in water distribution systems from multivariate water quality time series. Environ. Sci. Technol. 46(15), 8212–8219 (2012)
Rodrigues, F., Pereira, F.C.: Beyond expectation: deep joint mean and quantile regression for spatio-temporal problems. arXiv preprint arXiv:1808.08798 (2018)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B (Methodol.) 58(1), 267–288 (1996)
Van Buuren, S., Groothuis-Oudshoorn, K.: MICE: multivariate imputation by chained equations in R. J. Stat. Softw. 45(3), 1–68 (2011)
Versteeg, R., de Graaff, B.: Valdidatieplan Waterkwantiteitsmetingen. STOWA 2009–20 (2009). (in Dutch)
Western Electric Company: Statistical Quality Control Handbook. Western Electric Company, Rossville (1956)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
van de Wiel, L., van Es, D.M., Feelders, A.J. (2020). Real-Time Outlier Detection in Time Series Data of Water Sensors. In: Lemaire, V., Malinowski, S., Bagnall, A., Guyet, T., Tavenard, R., Ifrim, G. (eds) Advanced Analytics and Learning on Temporal Data. AALTD 2020. Lecture Notes in Computer Science(), vol 12588. Springer, Cham. https://doi.org/10.1007/978-3-030-65742-0_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-65742-0_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-65741-3
Online ISBN: 978-3-030-65742-0
eBook Packages: Computer ScienceComputer Science (R0)
