CN112036075A - Abnormal data judgment method based on environmental monitoring data association relation - Google Patents

Abstract

The invention relates to a method for determining abnormal data based on the association relationship of environmental monitoring data, which mainly includes: firstly, the monitoring data is divided into training data, verification data and test data; After the model is constructed and debugged, it is tested with test data and embedded in the environmental monitoring platform. Based on the real-time monitoring data and the model, the predicted value at the next moment is given on the monitoring platform, and the absolute value bia of the predicted result and the actual value is calculated, and the measured value is judged whether the measured value is abnormal or not after comparing the bia with the true value of MAE±30%. The present invention fully considers the influence of meteorological conditions on monitoring data, as well as the time continuity and variation characteristics of monitoring data, finally solves the problem of lack of automatic quality control means for multi-source monitoring data, and realizes automatic and intelligent screening of suspicious data and judgment function, which ensures the quality of data and provides strong support for later data use and environmental forecasting and early warning.

Description

An abnormal data determination method based on the association relationship of environmental monitoring data

technical field

The invention relates to the technical field of data quality control for real-time environmental monitoring, and is mainly used for automatic judgment of abnormal values of real-time monitoring data of particulate matter and gaseous pollutants.

Background technique

For the control and monitoring of atmospheric environmental data quality, most of the data screening methods currently used are manual, that is, by drawing daily average graphs and monthly average graphs to judge the abnormal fluctuation and outlier degree of each monitoring indicator. This method increases a lot of human resources, and in the face of massive monitoring data, manual review often has omissions. In view of the fact that the concentration indicators of the monitored substances output by environmental monitoring instruments are generally measured in minutes or hours, there is a certain lag in manual manual review of data, and real-time data quality control can be achieved through an automated review mechanism.

In view of the lack of automatic quality control means for atmospheric monitoring data, the algorithm is now designed according to the technical scheme of data monitoring and composite adoption of the environmental monitoring station, to realize the automatic intelligent quality control technology of atmospheric environment monitoring data, and solve the problem of lack of automatic quality control means for multi-source monitoring data. Therefore, the quality control of atmospheric monitoring equipment should follow the same method system, and the development of remote automatic quality control technology of monitoring equipment should be promoted.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for judging abnormal data based on the association relationship of environmental monitoring data, so as to solve the problem of lack of automatic quality control means for multi-source monitoring data.

A method for determining abnormal data based on the association relationship of environmental monitoring data, comprising the following steps:

S1. Preprocessing the historical data and the environmental monitoring data to be analyzed: the monitoring data of the historical data and the environmental monitoring data to be analyzed is used to judge the missing values and abnormal values with data acquisition software, and then replace the missing values and abnormal values;

S2. Divide the data into training data and validation data, and convert the training data and validation data into sequence data required by the model; the training set and validation set data both include normal data and data that are manually identified as abnormal, and the reason for the abnormality Including sudden rise/drop of data, no circadian change, persistent low value, etc., abnormal data judgment is related to the continuity of monitoring data before and after; the ratio of training set and validation set data can be 7:3;

S3. Use the training data to build a model, and then use the validation data to select the best parameters of the model according to the mean absolute error MAE; after the model is built and debugged, it is embedded in the environmental monitoring platform, and the environmental monitoring data to be analyzed at time t-1 c _{t -true is} used as input data, and the predicted value c _t-pre at time t is continuously obtained;

S4. Compare the predicted value c _t-pre with the environmental monitoring data c _t-true to be analyzed at time t, obtain the absolute value bia, and compare it with the empirical error of MAE±30% true value to determine the abnormality. The data in the range is marked as abnormal data. In this process, the MAE changes with the input data, so the threshold is a dynamic threshold.

where bia=|c _t-pre -c _t-true |.

Preferably, the method for replacing missing values and outliers described in step S1 includes:

S11. The interpolation function of linear interpolation is a first-order polynomial. First, it is assumed that the known function y=f(x) has (n+1) mutually different points x _i (i=0,1,2, 3...,n) are respectively y _i , and the polynomial is calculated:

to satisfy

From analytic geometry we know:

where x ₀ , x ₁ , y ₀ , y ₁ — known statistics

x - any data between x ₀ and x ₁

y - interpolation data corresponding to x;

S12. Use the data at two moments before and after the missing or abnormal value, and perform linear interpolation according to the above formula.

Preferably, the environmental monitoring data includes five meteorological parameters, air pressure, temperature, humidity, wind direction, wind speed and pollutant concentration.

Preferably, the historical data in step S1 and the environmental monitoring data to be analyzed are grouped according to seasons and regions, and a two-dimensional table corresponding to each pollutant is constructed respectively.

The existing monitoring data is hourly monitoring data. Due to the strong regional and seasonal characteristics of environmental monitoring data, the national data are divided into regions (Northeast, Northwest, North, Central, South, Southeast, Southwest), seasons (spring, summer) The outliers and missing values judged by the data mining software in the hourly monitoring data are automatically replaced by the linear interpolation method, and finally a two-dimensional table corresponding to each pollutant in different regions and different seasons is constructed for use in Build the respective models.

作为输入，t+1时刻的某污染物浓度和气象条件

作为输出，使用训练集数据

经过学习后得到t+1时刻的污染物浓度和气象条件；Preferably, building a model from the training data in step S3 includes the following steps: during the modeling process, use a certain pollutant concentration and meteorological conditions at time t

As input, a pollutant concentration and meteorological conditions at time t+1

As output, use the training set data

After learning, the pollutant concentration and meteorological conditions at time t+1 are obtained;

When the model is constructed, it goes through the input layer, the hidden layer and the output layer. The hidden layer can choose to retain the information at time t ₁ , t ₂ ...... t, and use it as input information at time t+1; each unit The calculation steps and methods are as follows:

Input gate:

Forgotten Gate:

Output gate:

The hidden layer candidate memory unit value at the current moment:

The state value of the hidden layer memory unit at the current moment:

The output value of the hidden layer at the current moment:

表示输入-输入门权重矩阵，

表示上一时刻隐藏层单元-输入门权重矩阵，

表示输入门-隐藏层记忆单元权重矩阵，

是输入层-遗忘门的权重矩阵，

是上一时刻隐藏层单元-遗忘门的权重矩阵，

是隐藏层记忆单元-遗忘门的权重矩阵，

是输入层-输出门权重矩阵，

是上一时刻隐藏层单元-输出门的权重矩阵，

是隐藏层记忆单元-输出门的权重矩阵，

是输入层-隐藏层记忆单元的权重矩阵，

是上一时刻隐藏层单元到隐藏层记忆单元的权重矩阵，

分别是输入门、遗忘门、输出门和隐藏层记忆单元的偏置。i, φ and ω are the input gate, forgetting gate and output gate respectively, h is the output of the hidden layer, c is the value of the memory unit of the hidden layer, θ and σ respectively represent the nonlinear activation functions of several gates, θ generally takes the tanh function, σ is generally a logistic sigmoid function,

represents the input-input gate weight matrix,

Represents the hidden layer unit-input gate weight matrix at the previous moment,

represents the input gate-hidden layer memory unit weight matrix,

is the weight matrix of the input layer-forget gate,

is the weight matrix of the hidden layer unit-forgetting gate at the previous moment,

is the weight matrix of the hidden layer memory unit-forgetting gate,

is the input layer-output gate weight matrix,

is the weight matrix of the hidden layer unit-output gate at the previous moment,

is the weight matrix of the hidden layer memory unit-output gate,

is the weight matrix of the input layer-hidden layer memory unit,

is the weight matrix from the hidden layer unit to the hidden layer memory unit at the previous moment,

are the biases of the input gate, forget gate, output gate, and hidden layer memory unit, respectively.

Considering the strong temporal correlation between the data in the sample, the RNN-LSTM model is selected among several common machine learning algorithms. This model can use the temporal information of the input sequence to improve the prediction accuracy of time series data. On the basis of the ordinary multi-layer BP neural network, the horizontal connection between the units in the hidden layer is added. Through a weight matrix, the output of the neural unit of the previous time series can be used as the input of the current neural unit, so that the neural network has the memory function.

The preferred training set data is used as the model input to complete the model construction; in step S3, the method of using the verification data to select the best parameters of the model according to the mean absolute error MAE is as follows: the verification data is used as the model input, and the model predicted value c _pre and the real The value c _ture is compared, and the mean absolute error (MAE) is used for evaluation, and the optimal model parameters are obtained when the MAE is the smallest;

Among them, n represents the number of samples, c _true represents the true value, and c _pre represents the predicted value.

Compared with the existing abnormal data detection methods, the present invention uses the manually reviewed historical data as training data to construct a neural network model, and then uses the verification data to select the best parameters of the model according to MAE; after the model is constructed and debugged, it is embedded in the environment. The monitoring platform gives the predicted value at the next moment according to the real-time monitoring data and the model to judge whether the monitoring value at the next moment is abnormal. The invention applies the methods of machine learning and data mining to the field of environmental monitoring, realizes the integration and innovation of computer data science and environmental monitoring interdisciplinary, and establishes an analysis system through learning for monitoring data with large scale and high complexity of internal rules. In addition, when building the model, the influence of meteorological conditions on the monitoring data, as well as the time continuity and variation characteristics of the monitoring data were fully considered. Ultimately solve the problem of lack of automatic quality control methods for multi-source monitoring data, realize automatic and intelligent screening and judgment functions for suspicious data, make the quality control of atmospheric monitoring equipment follow the same method system, ensure the quality of monitoring data, and promote monitoring The development of equipment remote automatic quality control technology provides strong support for later data use and environmental forecasting and early warning.

Description of drawings

Fig. 1 is the flow chart of the abnormal data determination method based on the environmental monitoring data association relationship of the present invention;

Figure 2 is a schematic diagram of the construction of the RNN-LSTM model;

Figure 3 is a prediction effect diagram.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

Example 1

In this embodiment, anomaly detection is performed on the monitoring data in November based on the monitoring data of air pollutants in a certain place from September to October 2019 (see Table 1 for some pollutant data and meteorological data).

The model execution steps are as follows:

Step 1: Preprocess the air pollutant data and the corresponding meteorological data, including interpolation processing of missing values and abnormal values (caused by insufficient sampling time) automatically identified by the data acquisition software, and manually review them as abnormal data marked, see Table 2 for details.

Among them, the automatic replacement of missing values and outliers includes:

S11. First, assume that the known function y=f(x) is on (n+1) mutually different points x _i (i=0,1,2,3...,n) on the interval [a,b] The values are y _i , and find the polynomial:

to satisfy

From analytic geometry we know:

where x ₀ , x ₁ , y ₀ , y ₁ — known statistics

x - any data between x ₀ and x ₁

y - the interpolated data corresponding to x.

Step 2: Use the above September data as training data, October data as validation data, and November data as test data, and convert the training and validation data into the sequence data format required by the model.

Step 3: Build a neural network based on RNN-LSTM, input the training data into the model, and the model will automatically adjust the network parameters (including the weights and thresholds of network nodes) according to the prediction error of the training data until the end of the training process. During training, a standard 3-layer network is used, 1 input layer (6 input nodes) + 1 hidden layer + 1 output layer (1 output node); at the same time, the model hyperparameters (including the hidden layer nodes of the model) are adjusted according to the effect of the verification data. The number of hidden layer nodes of the PM ₁₀ model is 500, the number of hidden layer nodes of the PM _2.5 model is 300, the number of hidden layer nodes of the CO model is 200, the number of hidden layer nodes of the NO ₂ model is 100, and the number of hidden layer nodes of the O ₃ model is 100. The layer nodes are 100 and the SO ₂ model hidden layer nodes are 100. Figure 2 The flow of model building.

The model formula is:

Input gate:

Forgotten Gate:

Output gate:

The hidden layer candidate memory unit value at the current moment:

The state value of the hidden layer memory unit at the current moment:

The output value of the hidden layer at the current moment:

Step 4: Input the November test data into the trained model, and compare the predicted data with the measured value. The effect of some test data is shown in Figure 3.

Using the mean absolute error evaluation, CO is about 0.08, NO ₂ is about 6.85, O ₃ is about 10.45, PM ₁₀ is about 7.80, PM _2.5 is about 4.97, and SO ₂ is about 3.38. The size of the MAE is related to the scale of the test data, and the test results for the November data are generally better.

Step 5: Judging data abnormality according to the absolute value of the difference between the predicted value and the measured value. The data exceeding the model prediction error MAE±30% of the measured value will be judged as abnormal data by the model. Some predicted values, bia and MAE values output by the model are shown in Table 3, and some results of model abnormality judgment are shown in Table 4.

Table 1 Partial pollutant data and meteorological data of a certain place

Note: NA means missing data, RM means manual review as abnormal data

Table 2 Data before and after processing

Connect to Table 2

time SO₂_MARK NO₂_MARK O₃_MARK CO_MARK PM₁₀_MARK PM_2.5_MARK 2019-09-13 01:00 0 0 0 0 0 0 2019-09-13 02:00 0 0 0 0 0 0 2019-09-13 03:00 0 0 0 0 0 0 2019-09-13 04:00 0 0 0 0 0 0 2019-09-13 05:00 0 0 0 0 0 0 2019-09-13 06:00 0 0 0 0 0 0 2019-09-13 07:00 0 0 0 0 0 0 2019-09-13 08:00 0 0 0 0 0 0 2019-09-13 09:00 0 0 0 0 0 0 2019-09-13 10:00 0 0 0 0 0 0 2019-09-13 11:00 0 0 0 0 0 0 2019-09-13 12:00 0 0 0 0 0 0 2019-09-13 13:00 0 0 0 0 0 0 2019-09-13 14:00 0 0 0 0 0 0 2019-09-13 15:00 0 0 0 0 0 0 2019-09-13 16:00 0 0 0 0 0 0 2019-09-13 17:00 0 0 1 0 0 0 2019-09-13 18:00 0 0 0 0 0 0 2019-09-13 19:00 0 0 0 0 0 0 2019-09-13 20:00 0 0 0 0 0 0 2019-09-13 21:00 0 0 0 0 0 0 2019-09-13 22:00 0 0 0 0 0 0 2019-09-13 23:00 0 0 0 0 0 0 2019-09-14 00:00 0 0 0 0 0 0

Table 3 The NO ₂ part of the time prediction data, measured data, MAE and bia value judged by the model

Note: 1 in the NO ₂ _Mark column indicates that the model is judged to be abnormal data

Table 4 Some abnormal pollutant data finally judged by the model

Claims (6)

1. a method for judging abnormal data based on the association relationship of environmental monitoring data, is characterized in that, comprises the following steps: S1. Preprocess historical data and environmental monitoring data to be analyzed: use data acquisition software to determine missing values and abnormal values for historical data and environmental monitoring data to be analyzed, and then replace missing values and abnormal values; S2. Divide the processed historical data into training data and verification data, and convert the training data and verification data into sequence data required by the model; S3. Use the training data to build a model, and then use the verification data to select the best parameters of the model according to the MAE; after the model is built and debugged, it is embedded in the environmental monitoring platform, and the environmental monitoring data to be analyzed at time t-1 c _{t-1- true} is used as input data, and the predicted value c _t-pre at time t is continuously obtained; S4. Compare the predicted value c _t-pre with the environmental monitoring data c _t-true to be analyzed at time t, and find out the absolute value bia compared with the empirical error of the MAE±30% true value to determine the abnormality. Data is marked as abnormal data where bia=|c _t-pre -c _t-true |. 2. The method for determining abnormal data based on the association relationship of environmental monitoring data according to claim 1, wherein the method for replacing missing values and abnormal values in step S1 comprises: S11. First, assume that the known function y=f(x) is on (n+1) mutually different points x _i (i=0,1,2,3...,n) on the interval [a,b] The values are y _i , and find the polynomial:

to satisfy

From analytic geometry we know:

where x ₀ , x ₁ , y ₀ , y ₁ — known statistics x - any data between x ₀ and x ₁ y - interpolation data corresponding to x; S12. Use the data at two moments before and after the missing or abnormal value, and perform interpolation according to the above formula. 3 . The abnormal data determination method based on the association relationship of environmental monitoring data according to claim 1 , wherein the environmental monitoring data includes five meteorological parameters, air pressure, temperature, humidity, wind direction, wind speed and pollutant concentration. 4 . 4. The abnormal data determination method based on the association relationship of environmental monitoring data according to claim 3, is characterized in that, the historical data in step S1 and the environmental monitoring data to be analyzed are grouped according to seasons and regions, and each pollutant is constructed respectively corresponding to 2D table of

5. the abnormal data determination method based on environmental monitoring data association according to claim 4, is characterized in that, in step S3, training data builds model, comprises the following steps: use certain pollutant concentration and meteorological condition at time t

As input, a pollutant concentration and meteorological conditions at time t+1

As output, use the training set data

After learning, the pollutant concentration and meteorological conditions at time t+1 are obtained; When the model is constructed, it goes through the input layer, the hidden layer and the output layer. The hidden layer can choose to retain the information at time t ₁ , t ₂ ...... t, and use it as input information at time t+1; each unit The calculation steps and methods are as follows: Input gate:

Forgotten Gate:

Output gate:

The hidden layer candidate memory unit value at the current moment:

The state value of the hidden layer memory unit at the current moment:

The output value of the hidden layer at the current moment:

i, φ and ω are the input gate, forgetting gate and output gate respectively, h is the output of the hidden layer, c is the value of the memory unit of the hidden layer, θ and σ respectively represent the nonlinear activation functions of several gates, θ generally takes the tanh function, σ is generally a lOgistic sigmOid function,

represents the input-input gate weight matrix,

Represents the hidden layer unit-input gate weight matrix at the previous moment,

represents the input gate-hidden layer memory unit weight matrix,

is the weight matrix of the input layer-forget gate,

is the weight matrix of the hidden layer unit-forgetting gate at the previous moment,

is the weight matrix of the hidden layer memory unit-forgetting gate,

is the input layer-output gate weight matrix,

is the weight matrix of the hidden layer unit-output gate at the previous moment,

is the weight matrix of the hidden layer memory unit-output gate,

is the weight matrix of the input layer-hidden layer memory unit,

is the weight matrix from the hidden layer unit to the hidden layer memory unit at the previous moment,

are the biases of the input gate, forget gate, output gate, and hidden layer memory unit, respectively. 6. according to claim 4, it is characterized in that, in step S3, the method for the optimal parameter MAE of the verification data selection model is: verification data is used as model input, and the model predicted value c is _Compared with the real value c _ture , MAE is used to evaluate, and the optimal model parameters are obtained when the MAE is the smallest;

Among them, n represents the number of samples, c _true represents the true value, and c _pre represents the predicted value.

Images