CN110619345B - Comprehensive verification method of tag credibility for cable-stayed bridge monitoring data validity - Google Patents

Abstract

The invention discloses a comprehensive tag reliability verification method for validity of cable-stayed bridge monitoring data. The invention relates to a method for comprehensively verifying confidence degrees of data validity labels detected by various abnormal points on the basis of a grey correlation degree theory, adopts a calculation method of Dun's correlation degree to pairwise correlate data of each monitoring point of a bridge every day, respectively adopts 3 algorithm, LOF algorithm and isolated forest algorithm to detect the abnormal points of the labeled data section obtained in the first step, verifies the confidence degree of the validity labels of the data section according to the probability of the abnormal point crossing, can diagnose the collected data before evaluation by the abnormal point detection with Isolation forest algorithm to eliminate the invisible invalid data or repair the invisible invalid data to form valid data, provides a scientific data base for the next safety evaluation, and has a certain use prospect.

Description

Comprehensive verification method of tag credibility for cable-stayed bridge monitoring data validity

technical field

The invention relates to the field of a comprehensive verification method for label credibility oriented to cable-stayed bridge monitoring data, in particular to a comprehensive verification method for label credibility oriented to the effectiveness of cable-stayed bridge monitoring data.

Background technique

Cable-stayed bridges are widely used because of their light structure, beautiful appearance, large span and good economy. Therefore, the monitoring and safety evaluation of cable-stayed bridges has always been a research hotspot. However, the basis of all this is the ability to collect accurate and effective data. However, due to the influence of various factors, the collected data is often not necessarily true and effective, or even full of errors and omissions, completely invalid.

Wherein, the failure of the data segment may be the failure of continuous data points, or the failure of several scattered single points causing the failure of the entire data segment. Several common data characteristics include: data segment drift, abnormal data frequency change, sudden jump of data segment, constant data value, etc. These types of failure data can be judged intuitively through image display, and there is a failure data segment On the surface, it is no different from many normal data segments, and there are no obvious characteristic fluctuations, so it is difficult to judge directly. And this kind of data occupies most of the failure data, which has a great impact on the safety evaluation of cable-stayed bridges. Therefore, the collected data must be diagnosed and processed before the evaluation, so as to eliminate the invisible and invalid data or repair it to form valid data, and provide a scientific data basis for the next safety evaluation.

At present, the artificial intelligence method with deep learning as the core is one of the main methods of recognized data validity diagnosis, and the credibility of the data validity label is an important basis for this method. Therefore, this application proposes A comprehensive verification method of tag credibility for cable-stayed bridge monitoring data validity.

Contents of the invention

The technical problem to be solved by the present invention is to overcome the existing defects and provide a comprehensive verification method for label credibility oriented to the effectiveness of cable-stayed bridge monitoring data, which can effectively solve the problems in the background technology.

In order to achieve the above object, the present invention provides the following technical solutions: a method for comprehensively verifying the credibility of tags facing the effectiveness of cable-stayed bridge monitoring data, comprising the following steps:

Step 1: Use the calculation method of Deng's correlation degree to correlate the daily data of each monitoring point of the bridge in pairs:

(1) Standardize the data collected at each measuring point;

(2) The data of each measuring point is used to complete the calculation of pairwise correlation with the Deng's correlation degree algorithm;

(3) Realize the automatic labeling of the validity of the data segment according to the position of the measuring point and the calculated value of the degree of correlation;

① Label the measurement point data: if r _oi ∈ [0.8,1], it belongs to strong association, and the label is a valid data segment; if r _oi ∈ [0.6,0.8), it belongs to general association, and the label is a general valid data segment; if r _oi ∈[0,0.6), the association is very weak, and the label is an invalid data segment;

②If the data sequence of the reference measurement point is strongly correlated with the data sequence of the surrounding measurement points and the same section, it means that the reference sequence is valid. labeling;

③If the data sequence of the reference measuring point is mostly strongly correlated with the data sequence of the surrounding measuring points and the same section, and a few are weakly correlated, remove the correlation degree of the weakly correlated data sequence, calculate the average correlation degree, and label according to the standard interval. In addition, the The weakly correlated data sequence is suspected to be relatively large. Use this as a reference data sequence to compare the correlation degree with the data sequence of adjacent monitoring points and the data sequence of the same section, and analyze again according to ②③④;

④ If the data sequence of the reference measuring point is weakly correlated with the surrounding measuring points and the data sequence of the same section, it can be basically judged as an invalid data segment, and the average value of the correlation degree is still calculated to determine the label of the data segment;

Step 2: Use the 3σ algorithm, LOF algorithm, and isolation forest algorithm to detect outliers on the labeled data segments obtained in step 1, and verify the credibility of the validity labels of the data segments according to the probability of crossing occurrences of abnormal points ;

(1) Data point detection based on 3σ algorithm;

① First, standardize the data to make the data meet the normal distribution;

② Assign the data parameters obtained by the algorithm to the single-point data, and make appropriate changes to the formula, >0 indicates that the point is abnormal: where is the mean value of the data sequence, is the standard deviation of the data sequence, and x is a certain data point;

(2) Data point detection based on the LOF algorithm: the local outlier factor LOF algorithm is a density-based algorithm, C1 is a cluster, C2 is another cluster, and O1 and O2 are isolated points far away from the two clusters. called outliers;

(3) Indicates the average number of the ratio of the local reachable density of the neighborhood points of point p to the local reachable density of the point. If the ratio is greater than 1, then the density of p is smaller than the density of its neighbor points, then it may be an abnormal point ;in:

is the local reachable density, which is the reciprocal of the average reachable distance from point to p in the k-th neighborhood of point p, and represents the density of points around point p. The farther away from the cluster, it is very likely to be an abnormal point, and the k-th reachable distance from point o to point p is the k-th distance neighborhood of point p, that is, all points within the k-th distance of point p;

(3) Outlier detection: The isolated forest uses a hyperplane to divide the space into two, and cuts it infinitely until it is cut into pieces of data; if it is a cluster with high density, the number of cuts is more; while the cluster with low density or itself It is a data that is far away from the cluster, and it will be cut soon. The isolated forest is composed of t isolated trees, and each isolated tree is a binary tree structure;

After obtaining t isolated trees, the isolated forest training is over, and then we can use the generated isolated trees to evaluate the test data. For each data point a, let the data point traverse each tree, and then calculate a and finally fall on Which layer is each tree, so as to obtain the average height of the data, set a certain threshold, the data below this threshold is abnormal, here set the average height of each data to, take=-0.05, if the average height If the value is <-0.05, it means that the data is abnormal;

Step 3: Comprehensive verification of label credibility, by using statistical algorithm, LOF algorithm and isolated forest algorithm (ISO) to detect abnormal points on the data segments of each measurement point, and verify the validity of the label according to the following steps;

(1) Calculate the outlier probability P1, P2, P3 of each algorithm and their average probability P;

(2) If the average probability P is greater than 10%, the data segment is considered to be abnormal data with a high probability, and the label is an invalid data segment; if the average probability P is less than or equal to 5%, it is not considered abnormal data, and the label is valid data segment; if the average probability P is greater than 5% and less than or equal to 10%, the label is a general valid data segment;

Compared with prior art, the beneficial effect of the present invention is:

In the present invention, on the basis of the theory of gray relational degree, combined with the comprehensive verification method of the data validity label confidence degree of various abnormal point detection, the calculation method of Deng's correlation degree is used to carry out two-by-two daily data of each monitoring point of the bridge. Correlation, using the 3σ algorithm, LOF algorithm, and isolation forest algorithm to detect abnormal points on the labeled data segments obtained in step 1, and verify the credibility of the validity labels of the data segments according to the probability of occurrence of abnormal point intersections. Through the outlier detection with the Isolation Froest (isolated forest) algorithm, the collected data can be diagnosed and processed before evaluation, so as to eliminate this invisible and invalid data or repair it to form effective data, which will provide a basis for the next Safety evaluation provides a scientific data basis, which is beneficial to people's use.

Description of drawings

Fig. 1 is isolated forest algorithm flow chart among the present invention;

Fig. 2 is a schematic diagram of distance-based abnormal points in the present invention;

detailed description

In order to make the technical means, creative features, goals and effects achieved by the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments.

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

As shown in Figure 1-2, the comprehensive verification method of label credibility for the effectiveness of cable-stayed bridge monitoring data, step 1: use the calculation method of Deng's correlation degree to correlate the daily data of each monitoring point of the bridge in pairs:

Standardize the data collected at each measuring point;

Use the data of each measuring point to complete the calculation of pairwise correlation by using Deng's correlation degree algorithm;

Then the formula for calculating the correlation coefficient is:

ρ is the resolution coefficient, which balances the determination of the correlation coefficient, and is generally in the interval (0, 1).

The formula for calculating the correlation degree is:

r _oi reflects the variation similarity between sequences.

②In practical application, X ₀ is the data sequence of a certain measuring point, X{X _i |i=1,2,...m} is the data sequence of other measuring points in the same section, m is the length of the data sequence, they Each has n components.

Automatic labeling of the validity of the data segment is realized according to the position of the measuring point and the calculated value of the correlation degree.

① Label the measurement point data: if r _oi ∈ [0.8,1], it belongs to strong association, and the label is a valid data segment; if r _oi ∈ [0.6,0.8), it belongs to general association, and the label is a general valid data segment; if r _oi ∈[0,0.6), the correlation is very weak, and the label is an invalid data segment.

②If the data sequence of the reference measurement point is strongly correlated with the data sequence of the surrounding measurement points and the same section, it means that the reference sequence is valid. Make a label.

③If the data sequence of the reference measuring point is mostly strongly correlated with the data sequence of the surrounding measuring points and the same section, and a few are weakly correlated, remove the correlation degree of the weakly correlated data sequence, calculate the average correlation degree, and label according to the standard interval. In addition, the The weakly correlated data sequence is more suspected, so use this as a reference data sequence to compare the correlation degree with the data sequence of adjacent monitoring points and the data sequence of the same section, and analyze again according to ②③④.

④ If the data sequence of the reference measuring point is weakly correlated with the surrounding measuring points and the data sequence of the same section, it can be basically judged as an invalid data segment, and the mean value of the correlation degree is still calculated to determine the label of the data segment.

Step 2: Use the 3σ algorithm, LOF algorithm, and isolation forest algorithm to detect outliers on the labeled data segments obtained in step 1, and verify the credibility of the validity labels of the data segments according to the probability of crossing occurrences of abnormal points .

Data point detection based on 3σ algorithm.

① First, standardize the data to make the data meet the normal distribution;

②Assign the data parameters obtained by the algorithm to the single-point data, and make appropriate changes to the formula, and ξ>0 indicates that the point is abnormal:

ξ=|x-μ|-3σ (3)

Among them, μ is the mean value of the data sequence, σ is the standard deviation of the data sequence, and x is a certain data point.

Data point detection based on LOF algorithm: local anomaly factor LOF algorithm is a density-based algorithm, C1 is a cluster, C2 is another cluster, and O1 and O2 are isolated points away from the two clusters, also called outliers .

Indicates the average number of the ratio of the local reachable density of the neighborhood points of point p to the local reachable density of the point. If the ratio is greater than 1, the density of p is smaller than the density of its neighbor points, so it may be an abnormal point, where:

is the local reachable density, which is the reciprocal of the average reachable distance from point to p in the k-th neighborhood of point p, and represents the density of points around point p. The farther away from the cluster, it is very likely to be an outlier point, and the k-th reachable distance from point o to point p is the k-th distance neighborhood of point p, that is, all points within the k-th distance of point p.

Outlier detection: The isolated forest uses a hyperplane to divide the space into two, and cuts it infinitely until it is cut into pieces of data; if it is a cluster with high density, the number of cuts is more; while the cluster with low density may itself be a far away The data of the cluster will be cut soon. The isolated forest consists of t isolated trees, and each isolated tree is a binary tree structure.

After obtaining t isolated trees, the isolated forest training is over, and then we can use the generated isolated trees to evaluate the test data. For each data point a, let the data point traverse each tree, and then calculate a and finally fall on Which layer is each tree, so as to obtain the average height of the data, set a certain threshold, the data below this threshold is abnormal, here set the average height of each data to, take=-0.05, if the average height If the value is <-0.05, it means that the data is abnormal.

Step 3: Comprehensive verification of label credibility, by using statistical algorithm, LOF algorithm and isolated forest algorithm (ISO) to detect outliers in the data segments of each measurement point, and verify the validity of the label according to the following steps.

(1) Calculate the outlier probabilities P1, P2, P3 and their average probability P of each algorithm respectively.

(2) If the average probability P is greater than 10%, the data segment is considered to be abnormal data with a high probability, and the label is an invalid data segment; if the average probability P is less than or equal to 5%, it is not considered abnormal data, and the label is valid data segment; if the average probability P is greater than 5% and less than or equal to 10%, the tag is a general valid data segment.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (1)

1. The comprehensive label credibility verification method for the effectiveness of the cable-stayed bridge monitoring data comprises the following steps: the method comprises the following steps:

the method comprises the following steps: performing pairwise association on data of each monitoring point of the bridge every day by adopting a Deng correlation calculation method:

standardizing data collected by each measuring point;

computing pairwise correlation of the data of each measuring point by using a Deng correlation algorithm;

the correlation coefficient is calculated by the following formula:

rho is a resolution coefficient, balances the judgment of the correlation coefficient and is generally between intervals (0,1);

the calculation formula of the correlation degree is as follows:

r _oi reflecting varying similarity between sequences;

(2) in practical application, X ₀ For a data sequence at a certain point, X { X } _i I =1,2,. M } is other measuring point data sequences of the same section, m is the length of the data sequence, and the measuring point data sequences have n components;

(1) labeling the measuring point data: if r _oi ∈[0.8,1]The label is valid, belonging to a strong associationA data segment; if r _oi E [0.6,0.8) belongs to general association, and the label is a general valid data segment; if r _oi E.g., [0,0.6), the association is weak, and the label is an invalid data segment;

(2) if the data sequence of the reference measuring point is strongly associated with the data sequences of the surrounding measuring points and the same section, the reference sequence is indicated to be effective, the average value of the association degree of the reference data sequence with the data sequences of the surrounding measuring points and the same section is calculated, and the labeling is carried out according to a standard interval;

(3) if the data sequence of the reference measuring point is strongly associated with most of the data sequences of the surrounding measuring points and the same section, and a small amount of weak association is performed, removing the association degree of the weak association data sequence, calculating the average association degree, labeling according to a standard interval, and in addition, the weak association data sequence is relatively suspected, taking the weak association data sequence as the reference data sequence, performing association degree comparison with the data sequence of the adjacent monitoring point and the data sequence of the same section, and analyzing again according to the steps (2), (3) and (4);

(4) if the correlation degree of the reference measuring point data sequence, the data sequences of the surrounding measuring points and the same section is weak, determining as an invalid data segment, and still performing correlation degree mean value calculation to determine a data segment label;

step two: respectively adopting a 3 sigma algorithm, an LOF algorithm and an isolated forest algorithm to detect abnormal points of the labeled data segments obtained in the step one, and verifying the reliability of the validity labels of the data segments according to the probability of the abnormal points;

(1) Data point detection based on a 3 sigma algorithm;

(1) firstly, carrying out standardization processing on data to enable the data to meet normal distribution;

(2) the data parameters obtained by the algorithm are assigned to single-point data, the formula is appropriately changed, and xi >0 shows that the point is abnormal as follows:

ξ＝|x-μ|-3σ

wherein mu is the mean value of the data sequence, sigma is the standard deviation of the data sequence, and x is a certain data point;

(2) Data point detection based on LOF algorithm: the local anomaly factor LOF algorithm is a density-based algorithm, wherein C1 is a cluster, C2 is another cluster, and O1 and O2 are isolated points far away from the two clusters, which are also called anomaly points;

(3) Representing the average of the ratio of the local reachable density of the neighborhood point of the point p to the local reachable density of the point, if the ratio is more than 1, the local reachable density is less than the neighborhood point density, and the point is considered as an abnormal point;

wherein:

the local reachable density is the reciprocal of the average reachable distance from the point p in the kth neighborhood of the point p to the point p, the density of the points around the point p is represented, the higher the density is, the naturally, the cluster is, the lower the density is, the farther the point p is from the cluster, the higher the possibility that the point p is an abnormal point is considered to be, and the kth reachable distance from the point o to the point p is the kth neighborhood of the point p, namely all the points within the kth distance of the point p;

(4) Abnormal point detection: the isolated forest cuts the space I into two by using a hyperplane, and continues to be cut infinitely until the space I is cut into data one by one; if the cluster is a high-density cluster, the cutting times are more; the clusters with low density or data far away from the clusters can be cut quickly, the isolated forest is composed of t isolated trees, and each isolated tree is of a binary tree structure;

after t isolated trees are obtained, training of the isolated forest is finished, then the generated isolated trees are used for evaluating test data, for each data point a, the data point is made to traverse each tree, then a is calculated to finally fall on the fourth layer of each tree, so that the average height of the data is obtained, a certain threshold value is set, the data below the threshold value is abnormal, the average height value of each data is set to be = -0.05, and if the average height value is less than-0.05, the data is abnormal;

step three: comprehensively verifying the reliability of the label, respectively detecting abnormal points of the data segments of each measuring point by using a statistical algorithm, an LOF algorithm and an isolated forest algorithm (ISO), and verifying the validity of the label according to the following steps;

(1) Respectively calculating the abnormal point probabilities P1, P2 and P3 of each algorithm and the average probability P of the abnormal point probabilities;

(2) If the average probability P is more than 10%, the data segment is considered as abnormal data, and the label is an invalid data segment; if the average probability P is less than or equal to 5%, the data is not considered as abnormal data, and the label is an effective data segment; if the average probability P is more than 5% and less than or equal to 10%, the label is a general valid data segment.

Images