CN106874651A - Room air data preprocessing method based on local weighted recurrence - Google Patents

Abstract

The present invention relates to the preprocessing of air parameter data that changes with time. It uses a simple but effective method to preprocess the data of indoor air that changes with time, including the filling of short-term data gaps and the removal of data abnormal jump values. At the same time, it is ensured that large changes in data related to changes in human behavior are not identified as outliers, and finally the zero offset is corrected. The technical scheme adopted by the present invention is, based on the indoor air data preprocessing method of local weighted regression, first fill in the short-term data gaps, ensure that there are no gaps of 0 in the entire data, and then remove the data abnormal jump value , when it is ensured that there is no needle-shaped data jump point, the zero offset correction is performed, that is, the processed data is substituted into the calibration curve. The invention is mainly applied to the preprocessing of air parameter data that changes with time.

Description

Indoor Air Data Preprocessing Method Based on Local Weighted Regression

technical field

This algorithm can fill in the data gaps in air parameters (temperature, humidity, formaldehyde concentration, PM2.5 concentration, carbon dioxide concentration, etc.) that change with time, and can remove abnormal jump values in the data, and can The data is corrected for zero offset. Belongs to the field of specific data preprocessing. Specifically, it involves a preprocessing method for indoor air data based on locally weighted regression.

Background technique

At present, the technical methods for data preprocessing are difficult and easy, but simple preprocessing methods are difficult to be effective, and effective preprocessing methods are often more complicated [1]. The data object preprocessed by this technology is indoor air data: first, this kind of data has the characteristics of slow change over time as a whole but has the characteristics of different degrees of random noise at every moment (as shown in Figure 1); second, due to the hardware The system has a data vacancy alarm function, so it can ensure that the duration of the data vacancy is very short; third, it already has a calibration curve for correcting the zero offset. Therefore, compared to filling data gaps and zero offsets, the core of this technology is to remove the jump outliers in the data, and it can ensure that large changes in data related to changes in human behavior will not be identified as outliers. remove.

In the method of eliminating data outliers, the most common method is to directly use the C4.5 decision tree to classify and judge the data [2]. The variable values are classified as outliers together; secondly, the CD (Curve Description) method is also used to classify outliers [3]. As far as the problem to be solved in this patent is concerned, it has similar defects to the decision tree method, and it is also more complicated than the decision tree method in program implementation; foreign countries also use noise data filtering method (Filters) to identify and eliminate outliers, which is more typical It is Ensemble Filter (EF) [4] and Iterative-Partitioning Filter (IPF) [5]. These two methods are relatively well-known, but they are relatively complicated, and multiple additional parameters must be set for them [1]. Faced with the problem is not necessary.

Contents of the invention

In order to overcome the deficiencies in the prior art, the present invention aims to use a simple but effective method to preprocess the data of indoor air changes over time, including filling short-term data gaps and removing data abnormal jump values, while ensuring Large changes in data related to changes in human behavior are not identified as outliers, and finally the zero offset is corrected. The technical scheme adopted by the present invention is, based on the indoor air data preprocessing method of local weighted regression, first fill in the short-term data gaps, ensure that there are no gaps of 0 in the entire data, and then remove the data abnormal jump value , when it is ensured that there is no needle-shaped data jump point, the zero offset correction is performed, that is, the processed data is substituted into the calibration curve.

The specific steps to remove the abnormal jump value of the data are to use the fitting curve to fit the meaningful information, and at the same time not to fit the needle-shaped data jump and all the high-frequency noise, specifically choose the local weighted regression (LocalWeightRegression ) to fit the useful information, and then subtract the fitted curve from the original data curve to obtain the noise curve, so as to solve the interference of the useful information on the jump value removal.

The specific steps of the principle of local weighted regression are to divide the entire data equally with a certain number of reference points on the horizontal axis, and calculate the linear local regression with these points as the center. When using the least square method to solve the regression parameters, The farther the data points are from the center point, the smaller the weights are, and finally get the regression values of these points, and then use interpolation to connect these regression value points, here you can use linear interpolation;

Further, for each training data point, it is necessary to make:

∑ _i w ⁽ⁱ⁾ (y ⁽ⁱ⁾ -θ ^T x ⁽ⁱ⁾ ) ² (1)

minimum;

Among them, i is the subscript of the number of training data; x refers to the time value of the time axis; y is the target value; θ is the coefficient vector of the regression equation, using quadratic regression, so θ is a three-dimensional vector; w is the Gaussian weight, indicating become:

The x without the superscript refers to the reference point on the selected horizontal axis, τ is the bandwidth (bandwidth), the larger the τ, the greater the strength of the local regression;

Locally weighted regression adds a Gaussian weight before each residual square term, and requires a quadratic regression curve for each reference point, and the curve parameters must be different. For any reference point x, there are:

θ＝(X ^T WX) ^-1 X ^T Wy (3)

Among them, X is an m-dimensional matrix composed of 1, x ⁽ⁱ⁾ , (x ⁽ⁱ⁾ ) ² , which is called the design matrix (design matrix) m is the number of training data, and X is written as:

W is an m-order diagonal matrix, written as diag(w ⁽¹⁾ …w ⁽ⁱ⁾ …w ⁽ⁿ⁾ ); y is an m-order column vector of target values, written as (y ⁽¹⁾ …y ^{(i )} …y ⁽ⁿ⁾ ) ^T ; the final θ is a 3×3 matrix, and the three elements from top to bottom in the first column of θ are taken as the constant front coefficients in the quadratic regression curve, once For the pre-term coefficient and the quadratic pre-term coefficient, for each reference point xck ^(j) , the regression regression curve has its corresponding regression value yck ^(j) , where j is the subscript of the number of reference point data, so that Form a regression point (xck ^(j) ,yck ^(j) );

Linear interpolation is performed on adjacent regression points to obtain a regression curve that regresses the entire data curve.

The process of using locally weighted regression to identify and remove jump values:

a. Perform local weighted regression on the original data curve to generate a fitting curve;

b. Subtract the fitted curve from the original data to obtain the residual curve;

c. Find the mean and standard deviation of the residual curve;

d. Traversing all the residual data, using the Raida criterion, select all the data that exceeds the limit:

e. Obtain the label of the data selected in d, and replace the original data in the corresponding label with the interpolation between the normal data at both ends of the jump data to achieve the purpose of smoothing.

Features and beneficial effects of the present invention are:

The invention has the characteristics of simple principle, fast calculation and remarkable effect. The invention has a good beneficial effect on the data of indoor air quality (IAQ) over time: the invention can effectively separate the noise from the original data; it can remove the jump abnormal value in the data by analyzing the characteristics of the noise, and It can ensure that large changes in data related to changes in human behavior will not be identified as outliers and eliminated.

Description of drawings:

Figure 1: The measured indoor air data of an office using sensors, the horizontal axis is time, in units of 1 second; the vertical axis is the corresponding numerical intensity. Both 1-2 and 1-3 in the figure have needle-shaped data jumps.

Figure 2: The data curve of measured formaldehyde over time with data gaps and data jumps.

Figure 3 Data smoothing flow chart.

Figure 4: Measured formaldehyde versus time data curve with data gaps interpolated.

Figure 5: The data fitting curve of the measured formaldehyde change over time after performing local weighted regression on the curve in Figure 1

Figure 6: The residual curve of the formaldehyde change data over time, it can be seen that the data jumps in it, and does not contain any useful information.

Figure 7: The data curve of the measured formaldehyde over time with data jumps smoothed, it can be seen that useful information has been preserved.

detailed description

In order to prevent the complexity of the algorithm and the amplification of errors, when preprocessing the indoor air data that may have short-term data gaps, first fill in the short-term and long-term data gaps, and ensure that there are no gaps of 0 in the entire data, and then perform The removal of the abnormal jump value of the data is to ensure that there is no needle-shaped data jump point, and then the zero offset is corrected, that is, the processed data is substituted into the calibration curve.

1) In the original data, all vacant values have been replaced by 0.

First select the time series data composed of n values, where n needs to be a number that is easily divisible, such as 2000, in order to effectively remove the data jump value. Then fill in the vacant value of the original data, and according to the nature of the data mentioned in (3), choose to interpolate the data here, and replace the result with the 0 value in the corresponding position.

2) When ensuring that there is no 0 value in the entire data, start to remove the jump value.

It is noted here that the large changes in data due to changes in human behavior cannot be removed. Taking the measured formaldehyde data with data gaps, data jumps, and large changes in data due to changes in human behavior as an example (as shown in Figure 2), it is found that the data jumps There is a big difference between data changes and data changes due to changes in human behavior:

As shown in the dotted line curve on the right side of Figure 2, at the corresponding moment, the office door is opened, causing the experimental formaldehyde gas in the laboratory opposite the door to flow into the office, resulting in an increase in the indoor formaldehyde concentration, which is a typical human behavior Variety. This variation in formaldehyde concentration cannot be identified as an outlier and removed. Secondly, the part of the solid red circle on the left is missing data, which is 0 value.

Here, it is not suitable to directly remove outliers from the data. On the one hand, the overall trend of the data shows a slow change, from 0.07 on the left to 0.047 on the right. On the other hand, the curve of the dotted red circle on the right is very Easy to be regarded as outliers, these are meaningful information in the data curve. Therefore, it is necessary to retain the meaningful information in the data curve, so the appropriate strategy is to use the fitting curve to fit the meaningful information, and at the same time do not fit the needle-shaped data jump and all high-frequency noise, which is selected here Local weighted regression (Local Weight Regression) fits the useful information, and then subtracts the fitted curve from the original data curve to obtain the noise curve, which can completely solve the interference of the useful information on the removal of the jump value.

3) Principle of local weighted regression

Locally weighted regression is an improved version of general linear regression, which can overcome the defects of underfitting or overfitting of the latter. Its steps are to divide the entire data equally with a certain number of reference points on the horizontal axis, and calculate the linear local regression with these points as the center. When using the least square method to solve the regression parameters, the farther away from the center The farther data points are given less weight. Finally, the regression values of these points can be obtained, and then these regression value points can be connected by interpolation, and linear interpolation can be used here.

For each training data point, make:

∑ _i w ⁽ⁱ⁾ (y ⁽ⁱ⁾ -θ ^T x ⁽ⁱ⁾ ) ² (1)

minimum.

Among them, i is the subscript of the number of training data; x is the feature value, which refers to the time value of the time axis in this article; y is the target value, which is the formaldehyde concentration value in this article; θ is the coefficient vector of the regression equation, and this method uses Quadratic regression, so θ is a three-dimensional vector; w is a Gaussian weight, expressed as:

The x without superscript refers to the selected reference point on the horizontal axis, τ is the bandwidth, and the larger τ is, the stronger the local regression is.

Local weighted regression adds a Gaussian weight before each residual square item, which will greatly weaken the influence of data far from the reference point on the fitting, and then achieve the purpose of local regression. For each reference point, a quadratic regression curve is required, and the curve parameters must be different. For any reference point x, there are:

θ＝(X ^T WX) ^-1 X ^T Wy (3)

Among them, X is an m-dimensional matrix composed of 1, x ⁽ⁱ⁾ , (x ⁽ⁱ⁾ ) ² , which is called the design matrix (design matrix). Curve, so the matrix has only three columns, if you want n times regression curve, the matrix has n+1 columns):

W is an m-order diagonal matrix, written as diag(w ⁽¹⁾ …w ⁽ⁱ⁾ …w ⁽ⁿ⁾ ); y is an m-order column vector of target values, written as (y ⁽¹⁾ …y ^{(i )} …y ⁽ⁿ⁾ ) ^T ; the final θ is a 3×3 matrix, and the three elements from top to bottom in the first column of θ are taken as the constant front coefficients in the quadratic regression curve, once Preterm coefficient and quadratic preterm coefficient. For each reference point xck ^(j) , the regression regression curve has its corresponding regression value yck ^(j) , where j is the subscript of the number of reference point data, thus forming a regression point (xck ^(j) , yck ^(j) ).

Linear interpolation of adjacent regression points can obtain a regression curve that regresses the entire data curve more accurately, and such a curve can retain all useful information.

4) The process of using local weighted regression to identify and eliminate jump values

a. Perform local weighted regression on the original data curve to generate a fitting curve.

b. Subtract the fitted curve from the original data to get the residual curve.

c. Find the mean and standard deviation of the residual curve.

d. Traversing all the residual data, using the Raida criterion, select all the data that exceeds the limit:

The Raida criterion stipulates that all data beyond the range of three standard deviations are eliminated. That is: remove all x, if x satisfies |x-μ|<3σ. Because the amount of data processed at one time is large, it far exceeds the lower limit of the data used in the Raida Guidelines: 100 data. Therefore, the simpler Raida criterion is adopted here.

e. Obtain the label of the data selected in d, and replace the original data (jump data) in the corresponding label with the interpolation between the normal data at both ends of the jump data to achieve the purpose of smoothing.

5) After using local weighted regression to identify and eliminate the jump value, the obtained processed data is substituted into the calibrated regression equation, and then the complete preprocessing result data is obtained.

1) The naming, input variable and output variable of the custom function.

There are two input variables: one is "raw data", which is imported from the excel table and exists in the form of a column vector; the other is "number of division intervals for the time dimension".

There is one output variable: "result data after abnormal data smoothing algorithm", and fill in the excel form.

2) The process of smoothing outliers of indoor air data based on local weighted regression on the original data (the original data cannot have

is the blank value of 0), as shown in Figure 3.

Here is an example of the measured formaldehyde change data curve with time after being filled (see Fig. 4): take the original data vector containing 1500 data, and take 50 for "the number of division intervals of the time dimension"; Fig. 5 shows the The fitting curve of the locally weighted regression of the curve in Figure 4, it can be seen that all useful information is preserved, and Figure 6 shows the residual curve obtained by subtracting the data curve in Figure 4 from the data curve in Figure 5, also known as noise It can be seen that the needle-shaped data jumps are separated from the original data, and the residual curve is judged according to the Raida criterion, and the jump data that do not meet the judgment are removed and replaced by interpolation, and then compared with the figure The fitting curves of 5 are added together to obtain the data curve smoothed by the jump value (as shown in Figure 7).

references:

[1] Salvador García, Julian Luengo, Tutorial on practical tips of the most influential data preprocessing algorithms in data mining. Knowledge-Based Systems, 2016; 98:1-29..

[2] J.R.Quinlan, C4.5: Programs for Machine Learning, Morgan KaufmannPub-lishers Inc., 1993. [3] Hao Zhou; Lifeng Qiao, Ph.D.; Yi Jiang, Ph.D.; HejiangSun, Ph.D. D.; Qingyan Chen, Ph.D. Recognition of air-conditioner operation from indoor air temperature and relative humidity by a data mining approach. Energy and Buildings, 2016; 111:233-241.

[4] C.E.Brodley, M.A.Friedl, Identifying mislabeled training data, J.Artif.Intell.Res.1999; 11:131–167.

[5] T.M. Khoshgoftaar, P. Rebours, Improving software quality prediction by noise filtering techniques, J. Comput. Sci. Technol. 2007; 22:387–396.

Claims (4)

1. a kind of room air data preprocessing method based on local weighted recurrence, it is characterized in that, long number in short-term is carried out first According to filling up for vacancy, it is to be ensured that then whole data carry out the removal of data exception hop value again in the absence of 0 value of vacancy, When no longer there is the data jump point of needle-like in guarantee, then carry out the correction of zero migration, and the data that will be handled well are updated to mark In determining curve.

2. the room air data preprocessing method of local weighted recurrence is based on as claimed in claim 1, it is characterized in that, carry out The removal of data exception hop value comprises the concrete steps that, fits significant information to come using matched curve, while not Fitting needle-like data jump and all of high-frequency noise, specifically from local weighted recurrence (Local Weight Regression the fitting of useful information) is carried out, then matched curve is subtracted with former data and curves to obtain noise curve, solve useful The interference that information is removed to hop value.

3. the room air data preprocessing method of local weighted recurrence is based on as claimed in claim 2, it is characterized in that, it is local Weighted regression principle comprises the concrete steps that, first separated whole data etc. with the reference point on the transverse axis of certain amount, and with this Ask calculation line locality to return centered on a little points respectively, when regression parameter is solved using least square method, from central point more away from Flexible strategy shared by data point are smaller, finally obtain the recurrence numerical value of these points, and these then are returned into numerical point with interpolation is connected, Used here as linear interpolation；

Further, to each training data point, will cause：

∑_iw⁽ⁱ⁾(y⁽ⁱ⁾-θ^Tx⁽ⁱ⁾)² (1)

It is minimum；

Wherein i is the number footmark of training data；X refers to the time value of time shaft；Y is desired value；θ be regression equation coefficient to Amount, using quadratic regression, therefore θ is a three-dimensional vector；W is Gauss flexible strategy, is expressed as：

$w^{\left(i\right)}=\exp (-\frac{{(x^{\left(i\right)}-x)}^2}{2{\mathrm{\&tau;}}^2})---\left(2\right)$

The reference point on selected transverse axis is referred to without superscript x, τ is bandwidth (bandwidth), and τ is bigger, it is local The intensity of recurrence is bigger；

Local weighted to return many Gauss power before each residuals squares, require secondary to each reference point returns Return curve, and parameter of curve must be different, to any one reference point x, have：

θ=(X^TWX)^-1X^TWy (3)

Wherein, X is by 1, x⁽ⁱ⁾, (x⁽ⁱ⁾)²The m dimension matrixes of composition, referred to as design matrix (design matrix) m is training number Data bulk, X writings：

$X=\left(\begin{array}{ccc}1& x^{\left(1\right)}& {\left(x^{\left(1\right)}\right)}^2\\ .& .& .\\ .& .& .\\ .& .& .\\ 1& x^{\left(i\right)}& {\left(x^{\left(i\right)}\right)}^2\\ .& .& .\\ .& .& .\\ .& .& .\\ 1& x^{\left(m\right)}& {\left(x^{\left(m\right)}\right)}^2\end{array}\right)---\left(4\right)$

W is m rank diagonal matrix, writing diag (w⁽¹⁾…w⁽ⁱ⁾…w⁽ⁿ⁾)；Y is the m rank column vectors that desired value is lined up, and is denoted as (y⁽¹⁾…y⁽ⁱ⁾…y⁽ⁿ⁾)^T；The θ for finally giving is the matrix of 3 × 3, takes three elements point from top to bottom in first row in θ Not as coefficient before the constant term in quadratic regression curve, coefficient before coefficient and quadratic term before first order, for each reference Point xck^(j), generation returns regression curve its corresponding regressand value yck^(j), wherein j is the number footmark with reference to point data, so Just a regression point (xck is formed^(j),yck^(j))；

By returning that adjacent regression point carries out that linear interpolation just can more accurately obtain returning whole data and curves Return curve.

4. the room air data preprocessing method of local weighted recurrence is based on as claimed in claim 2, it is characterized in that, use Local weighted recurrence is recognized and rejects the flow of hop value：

A. former data and curves are carried out into local weighted recurrence, generates matched curve；

B. former data are subtracted into matched curve and obtains residual error curve；

C. the average value and standard deviation of residual error curve are sought；

D. all of residual error data is traveled through, using Pauta criterion, all data beyond limitation is picked out：

E. the label of the data chosen in d is obtained, and the former data in correspondence label is substituted for the normal of saltus step data two ends Interpolation between data, reaches smooth purpose.