CN112562771B - Disk anomaly detection method based on neighborhood partition and isolation reconstruction - Google Patents

Abstract

An embodiment of the present invention proposes a disk anomaly detection method based on neighborhood partitioning and isolation reconstruction, including: collecting disk SMART information, filtering out effective disk feature attributes to form a data set, and performing exponential smoothing processing on it to obtain a disk training set ; Sampling the training set randomly for multiple times to obtain multiple sub-training sets, in the subset, the distance between each point and its closest point is used as the radius to construct the disk feature isolation area, and the test points that do not belong to any area are regarded as global anomalies; for non-global anomalies For the test point, the ratio of the radius of the area where the two consecutive adjacent points are located is regarded as the former abnormal value of the test point in this area; the area is reconstructed after including the test point, and the radius ratio of the area where the test point is located before and after reconstruction is used as the test point. The back abnormal value in this area; the abnormal score is obtained by combining the front and back abnormal values of all the areas where the test point is located. The technical solution provided by the embodiment of the present invention can effectively improve the recall rate of abnormal disks.

Description

A Disk Anomaly Detection Method Based on Neighborhood Partitioning and Isolation Reconstruction

【Technical field】

The invention relates to an anomaly detection method in the field of machine learning, in particular to a disk anomaly detection method based on neighborhood partition and isolation reconstruction.

【Background technique】

At present, the most used computer to store data is the disk, and the operation of the disk is directly related to the security of the stored data. Data centers typically have hundreds or thousands of disks, which greatly increases the likelihood of system failure. Therefore, data centers need to adopt some mechanisms to detect disk anomalies, so as to avoid irreversible damage or loss of data.

The commonly used disk anomaly detection method is the threshold detection method based on SMART data. It can monitor, record and compare with the preset safety value set by the manufacturer by sending detection commands in the disk to monitor and record the hardware operation of the disk itself. If it is detected that certain attributes exceed or are about to exceed the safety range of the preset safety value, the monitoring hardware or software of the host will automatically alert the user and perform minor automatic repairs to ensure the safety of disk data. However, the success rate of only passing SMART early warning is not high, only 3%-10%, which cannot meet the actual requirements, so this method needs to be further improved.

Disks have dozens of SMART attributes. If you want to analyze and train these attributes, you need to process a large amount of data, and there is a certain relationship between these attributes. Machine learning can learn from these large amounts of data and automatically Explore the internal relationship of attributes, and process data by building corresponding learning algorithm models. At the same time, a large amount of data can continuously optimize the model, which can improve the accuracy of predicting disk failures and achieve the purpose of detecting disk failures.

When using the machine learning method to solve the problem of disk anomaly detection, there is a phenomenon that the distribution of positive and abnormal data on the disk is extremely unbalanced, that is, the number of abnormal samples is far less than the number of normal samples or there is no abnormal sample. In the case of very few abnormal data, methods based on supervised algorithms cannot effectively solve this problem. Therefore, for such cases, unsupervised algorithms are considered to solve such problems, among which the isolation forest algorithm performs better. The Isolation Forest algorithm uses a random hyperplane to cut the data space and each subspace generated after it cuts each time until there is only one data point in each subspace or a preset termination condition is reached. The algorithm can only use normal samples to effectively deal with the classification problem under the extreme distribution of sample data, but the isolated forest algorithm cannot effectively detect local anomalies and package anomalies. The nearest neighbor method based on isolation can solve the above problems well in low-dimensional data, but it is difficult to effectively detect anomalies in high-dimensional space, and it is difficult to effectively determine local anomalies and package anomalies under different density conditions. Therefore, it is necessary to consider combining the test points according to the number and position of the isolated areas, so as to locate the test points more accurately and realize the effective determination of abnormal disks.

[Content of the invention]

In view of this, an embodiment of the present invention proposes a disk anomaly detection method based on neighborhood partitioning and isolation reconstruction, so as to solve the problems of accurate sample location and special anomaly detection in a region under different density conditions.

An embodiment of the present invention proposes a disk anomaly detection method based on neighborhood partition and isolation reconstruction, including:

Collect disk SMART information and filter out effective disk attribute features to form a data set, and perform exponential smoothing on it to obtain a stable disk training set;

Randomly sample the disk data set for multiple times to obtain multiple sub-training sets. Combine the Euclidean distance to calculate the distance between each point in the subset and its closest point. Use this distance as the radius to construct a disk isolation area, and use the test points that do not belong to any area as the global abnormal;

For non-global abnormal test points, find all the training points in the region where they are located and the nearest training point of the training point, and take the ratio of the radius of the region where the corresponding two points are located as the abnormal measurement value of the test point before reconstruction in this region;

Reconstruct the area after including the test point, and take the area radius ratio of the area where the test point is located after reconstruction and before reconstruction as the abnormal measurement value of the test point after reconstruction in this area;

Combine the two measurement values to get the reconstruction score of the test point in one area, take the reciprocal of the sum of the reconstruction scores of all the areas where the test point is located as the isolation score, and take the average of the isolation scores in multiple subsets as the test point abnormality Fraction.

In the above method, collect disk SMART information and filter out valid disk attribute features to form a data set, and perform exponential smoothing processing on it to obtain a stable disk training set. Effectively changing the SMART attribute of the disk, the collected data is used as a data set, and the SMART attribute is generated into a sequence that can be used to generate a model through the exponential smoothing method. The exponential smoothing formula has the following definitions:

S _t =α·Y _t +(1-α)·S _t-1

where t is the time, Y _t is the actual value of the t-th data, and S _t is the smoothed value of the previous t data, which is calculated recursively based on the actual value of time t and the smoothed value of the previous t-1 data. The width of k is fixed as k, k∈[1,5], which can be set according to the actual situation. If the value of k is smaller, it will lead to a weaker smoothing effect, but it has higher sensitivity to new changes in the data. The parameter α controls the rate at which older observations decay, α∈[0,1], the closer α is to 1, the closer the smoothed value is to the data value at the current time.

In the above method, the disk data set is randomly sampled for multiple times to obtain multiple sub-training sets, the distance between each point in the subset and its closest point is calculated in combination with the Euclidean distance, and the disk isolation area is constructed with this distance as the radius. The test point is regarded as a global anomaly. The specific description is: perform multiple simple random sampling on the training set D, and obtain multiple sub-training sets S _i with the sample size ψ, where i is an integer and 1≤i≤t, and t is a subset of In each sub-training set S _i , the distance between the points is calculated based on the Euclidean distance, and each training point a is taken as the center of the region, and the distance from a to its nearest training point η The distance τ( _a ) of a is used as the area radius to construct a disk isolation area, so that point a is isolated from other training points in the subset, where a,η _a ∈S _i , the radius distance τ(a) of point a has the following definition :

τ(a)=||a- _ηa ||

For each sub-training set S _i , let c be the training point closest to the test point x, c∈S _i , for the test point x, x is the global anomaly if and only if τ(c)<τ(x), τ(x) and τ(c) are the radial distances of point x and point c, respectively.

作为测试点x在此区域的重构前异常度量值，

越大，表明B(η_c)和B(c)的相对半径差距越小，测试点异常程度越低；反之，测试点异常程度越高。In the above method, for the non-global abnormal test point, find all the training points in the area where it is located and the nearest training point of the training point, and take the ratio of the radius of the area where the corresponding two points are located as the test point before the reconstruction of this area. The abnormal measurement value is specifically described as follows: for each sub-training set S _i , let b be any training point in S _i , and denote the hypersphere constructed with b as the center and τ(b) as the radius as B(b ), then for any training point y in B(b), there is y:||yb||<τ(b). For a test point that is not a global anomaly, let c be the training point closest to the test point x point, η _c is the closest training point to c, c,η _c ∈S _i , B(η _c ) and B(c) are centered on η _c and c, respectively, and τ(η _c ) and τ( c) is the hypersphere region of radius, the

As the pre-reconstruction anomaly measure of test point x in this region,

The larger the value, the smaller the relative radius difference between B(η _c ) and B(c), and the lower the abnormal degree of the test point; on the contrary, the higher the abnormal degree of the test point.

为该测试点在此区域的重构后异常度量值，

越大，表明重构后与重构前的相对半径差距越小，测试点异常程度越低；反之，测试点异常程度越高。In the above method, the area is reconstructed after the test point is included, and the area radius ratio of the area where the test point is located after reconstruction and before the reconstruction is used as the abnormality metric value of the test point after reconstruction in this area. A test point x, set point c∈S _i is the training point contained in the area where x is located, then the original training point will re-establish the area according to the nearest neighbor principle after including the test point x, and the area radius of the reconstructed point c will be recorded. is τ(c)′, the radius ratio of the area where the test point x is located after reconstruction and before reconstruction

is the post-reconstruction anomaly measure of the test point in this region,

The larger the value, the smaller the relative radius difference between the post-reconstruction and the pre-reconstruction, and the lower the abnormality degree of the test point; otherwise, the higher the abnormality degree of the test point.

In the above method, the reconstruction score of the test point in one area is obtained by combining the two measurement values, the reciprocal of the sum of the reconstruction scores of all the areas where the test point is located is used as the isolation score, and the average value of the isolation scores in multiple subsets is used. As the abnormal score of the test point, the specific description is: for each test point x, set the point c∈S _i to be the training point contained in the area where x is located, then the reconstruction score R(x) is defined as follows:

where τ(c)′ is the radius of the isolation area constructed with c as the center of the sphere after reconstruction, and the reciprocal of the sum of the reconstruction scores of all areas where the test point is located is taken as the isolation score, then the isolation score A(x) is as follows definition:

有如下定义：Among them, k is the number of the area where the test point x is located, and multiple sub-training sets {S ₁ , S ₂ ,..., S _t } are obtained by sampling the normal sample data set multiple times, and t is the number of subsets , you can choose an appropriate value according to the actual situation, and calculate the isolation score of the test point x in each subset S _i (1≤i≤t), then for the abnormal score of the test point x

There are the following definitions:

可以衡量x的异常程度，

越大则表明测试点x异常程度越高；反之，

越小则表明测试点x异常程度越低。where A _i (x) is the isolation score of test point x in the ith subset; the anomaly score

can measure the degree of abnormality of x,

The larger the value, the higher the abnormal degree of the test point x; on the contrary,

The smaller the value, the lower the abnormal degree of the test point x.

【Description of drawings】

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic flowchart of a disk anomaly detection method based on neighborhood partitioning and isolation reconstruction proposed by an embodiment of the present invention;

FIG. 2 is a schematic diagram of calculating a test point isolation score by a disk anomaly detection method based on neighborhood partition and isolation reconstruction according to an embodiment of the present invention.

【Detailed ways】

In order to better understand the technical solutions of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

It should be understood that the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

An embodiment of the present invention provides a nearest neighbor anomaly detection method based on edge sample density metric, as shown in FIG. 1 , which is a schematic flowchart of a nearest neighbor anomaly detection method based on edge sample density metric proposed in an embodiment of the present invention. Include the following steps:

Step 101 , collect the SMART information of the disk and filter out the effective disk attribute features to form a data set, and perform exponential smoothing processing on the data set to obtain a stable disk training set.

Specifically, collect disk SMART attribute data and filter out the disk SMART attributes that are not missing and effectively change with time, including: underlying data read error rate (current value), underlying data read error rate (original value), disk read error rate (original value) Write flux performance (current value), disk read and write flux performance (original value), spindle spin-up time (current value), spindle spin-up time (original value), start-stop count (current value), start-stop count ( original value), remapped sector count (current value), remapped sector count (original value), seek error rate (current value), seek error rate (original value), seek performance (current value), Seek performance (original value), power-on time accumulation (current value), power-on time accumulation (original value), spindle spin-up retries (current value), spindle spin-up retries (original value), power-on cycle count ( current value), power-on cycle count (original value), serial port deceleration error count (current value), serial port deceleration error count (original value), I/O error detection and correction (current value), I/O error detection and correction Correction (Raw Value), Uncorrectable Error (Current Value), Uncorrectable Error (Raw Value), Command Timeout (Current Value), Command Timeout (Raw Value), Goofy Write (Current Value), Goofy Write Input (original value), airflow temperature (current value), airflow temperature (original value), shock error rate (current value), shock error rate (original value), power failure return count (current value), power failure return count ( raw value), head load/unload count (current value), head load/unload count (raw value), temperature (current value), temperature (raw value), programming error block count (current value), programming error block count ( Original value), current sector count to be mapped (current value), current sector count to be mapped (original value), offline uncorrectable sector count (current value), offline uncorrectable sector count (raw value ), Ultra Access Check Error Rate (Current Value), Ultra Access Check Error Rate (Original Value), Head Flight Time/Transmission Error Rate (Current Value), Head Flight Time/Transmission Error Rate (Original Value), LBA Write The total number of inputs (current value), the total number of LBA writes (original value), the total number of LBA reads (current value), the total number of LBA reads (original value), the collected data is used as a data set, and the SMART Attributes are generated as series that can be used to generate models, and the exponential smoothing formula has the following definitions:

S _t =α·Y _t +(1-α)·S _t-1

where t is the time, Y _t is the actual value of the t-th data, and S _t is the smoothed value of the previous t data, which is calculated recursively based on the actual value of time t and the smoothed value of the previous t-1 data. The width of k is fixed as k, k∈[1,5], which can be set according to the actual situation. If the value of k is smaller, it will lead to a weaker smoothing effect, but it has higher sensitivity to new changes in the data. The parameter α controls the rate at which older observations decay, α ∈ [0,1], a large α is used to assign lower weights to distant observations, the closer α is to 1, the closer the smoothed value is to the current time data value.

Step 102: Randomly sample the disk data set for multiple times to obtain multiple sub-training sets, calculate the distance between each point in the subset and its closest point in combination with the Euclidean distance, and use the distance as the radius to construct a disk isolation area, and test the test that does not belong to any area. dot as a global exception.

Specifically, multiple simple random samplings are performed on the training set D to obtain multiple sub-training sets S _i with a sample size of ψ, where i is an integer and 1≤i≤t, and t is the number of subsets, which can be determined according to the actual situation Select the appropriate value, in each sub-training set S _i , calculate the distance between the points based on the Euclidean distance, take each training point a as the center of the region, and take the distance τ(a) from a to its nearest training point η _a Construct a disk isolation region as the region radius to isolate point a from other training points in the subset, where a,η _a ∈S _i , the radius distance τ(a) for point a is defined as follows:

τ(a)=||a- _ηa ||

是确定x是否为全局异常的分界线。For each sub-training set S _i , let c be the training point closest to the test point x, c∈S _i , for the test point x, x is the global anomaly if and only if τ(c)<τ(x), τ(x) and τ(c) are the radial distance between point x and point c, respectively,

is the dividing line that determines whether x is a global exception.

Step 103, for the non-global abnormal test point, find all the training points in the area where it is located and the nearest training point of the training point, and take the ratio of the radius of the area where the corresponding two points are located as the abnormality of the test point before reconstruction in this area. metric.

Specifically, for each sub-training set _Si , let b be any training point in Si, and denote the _hypersphere constructed with b as the center of the sphere and τ(b) as the radius as B(b). Any training point y in B(b) is defined as follows:

y:||y-b||＜τ(b)

是训练点c相对于其邻域的异常度量，将

作为测试点x在此区域的重构前异常度量值，

越大，表明B(η_c)和B(c)的相对半径差距越小，测试点异常程度越低；反之，

越小，表明B(η_c)和B(c)的相对半径差距越大，测试点异常程度越高。For test points that are not global anomalies, let c be the training point closest to test point x, η _c be the training point closest to c, c, η _c ∈ S _i , B(η _c ) and B(c) are respectively The ratio of the radii of B(η _c ) to B(c) for the hypersphere region with η _c and c as the centers and τ(η _c ) and τ(c) as the radii

is the anomaly measure of the training point c relative to its neighborhood, set

As the pre-reconstruction anomaly measure of test point x in this region,

The larger the value is, the smaller the relative radius difference between B(η _c ) and B(c) is, and the lower the abnormal degree of the test point; otherwise,

The smaller the value, the larger the relative radius difference between B(η _c ) and B(c), and the higher the abnormal degree of the test point.

Step 104 includes reconstructing the area after the test point, and taking the area radius ratio of the area where the test point is located after reconstruction and before the reconstruction as the post-reconstruction abnormality metric value of the test point in this area.

为该测试点在此区域的重构后异常度量值，

越大，表明重构后与重构前的相对半径差距越小，测试点异常程度越低；反之，测试点异常程度越高。Specifically, for each test point x, set point c∈S _i to be the training point contained in the area where x is located, then the original training point will re-establish the area according to the nearest neighbor principle after including the test point x, and the reconstructed point The area radius of c is denoted as τ(c)′, and the area radius ratio of the area where the test point x is located after reconstruction and before reconstruction

is the post-reconstruction anomaly measure of the test point in this region,

The larger the value, the smaller the relative radius difference between the post-reconstruction and the pre-reconstruction, and the lower the abnormality degree of the test point; otherwise, the higher the abnormality degree of the test point.

In step 105, the reconstruction score of the test point in one area is obtained by combining the two metric values, and the reciprocal of the sum of the reconstruction scores of all the areas where the test point is located is used as the isolation score, and the average value of the isolation scores in the multiple subsets is used as the isolation score. Test point anomaly score.

Specifically, for each test point x, set point c∈S _i to be the training point contained in the area where x is located, then its reconstruction score R(x) is defined as follows:

where τ(c)′ is the radius of the isolation area constructed with c as the center of the sphere after reconstruction, and the reciprocal of the sum of the reconstruction scores of all areas where the test point is located is taken as the isolation score, then the isolation score A(x) is as follows definition:

有如下定义：Among them, k is the number of the area where the test point x is located, and multiple sub-training sets {S ₁ , S ₂ ,..., S _t } are obtained by sampling the normal sample data set multiple times, and t is the number of subsets , you can choose an appropriate value according to the actual situation, and calculate the isolation score of the test point x in each subset S _i (1≤i≤t), then for the abnormal score of the test point x

There are the following definitions:

可以衡量x的异常程度，

越大则表明测试点x异常程度越高；反之，

越小则表明测试点x异常程度越低。where A _i (x) is the isolation score of test point x in the ith subset; the anomaly score

can measure the degree of abnormality of x,

The larger the value, the higher the abnormal degree of the test point x; on the contrary,

The smaller the value, the lower the abnormal degree of the test point x.

Figure 2 is a schematic diagram of the method for calculating the test point isolation score, where O _m is the test point, m∈[1,5], R _m is the area radius before O _m reconstruction, r _m is the area radius after O _m reconstruction, x _n is the test point, n∈[1,3], according to the relative position of the area where x _n is located and the degree of change in the isolation area before and after the introduction of x _n , the disk anomaly is predicted.

Table 1 shows the names, sample numbers, dimensions, abnormal sample ratios, and data of each data set when the disk anomaly detection method (NPIR) based on neighborhood partitioning and isolation and reconstruction according to the embodiment of the present invention solves the classification tasks of five groups of public data sets. The comparison experiment results of the recall rates of the two methods, wherein the comparison method in the embodiment of the present invention is the isolation-based nearest neighbor algorithm (iNNE). It can be seen from Table 1 that the method proposed in the present invention has the highest recall rate improvement on the Disk dataset compared to the comparison method, reaching 8.1%. The method proposed in the embodiment of the present invention can fully consider the distribution characteristics of edge samples, and can effectively solve the problem of local anomaly detection in the vicinity of edge samples.

Table I

dataset name Number of samples dimension Proportion of abnormal samples (%) iNNE recall rate (%) NPIR recall (%) Skin 9521 3 3.7 78.2 83.4 Glass 214 9 4.2 66.6 72.6 Waveform 3505 twenty one 4.6 61.1 68.3 Disk 35578 54 1.1 63.4 71.5 Port (Har) 5195 561 10.1 53.2 60.8

To sum up, the embodiments of the present invention have the following beneficial effects:

In the technical scheme implemented by the present invention, the SMART information of the disk is collected and the effective disk characteristic attributes are selected to form a data set, and the disk training set is obtained by performing exponential smoothing processing on it; the training set is randomly sampled for multiple times to obtain a plurality of sub-training sets. The disk feature isolation area is constructed with the distance between each point and its closest point as the radius, and the test point that does not belong to any area is regarded as the global anomaly; for the non-global anomaly test point, the ratio of the radius of the area where two consecutive adjacent points are located is used as the The former abnormal value of the test point in this area; the area is reconstructed after including the test point, and the radius ratio of the area where the test point is located before and after reconstruction is taken as the back abnormal value of the test point in this area; Before and after outliers get an outlier score. The technical solutions provided by the embodiments of the present invention fully consider the determination of abnormal disk samples under different density conditions, and can effectively solve the problem of abnormal disk determination in the high-dimensional feature distribution space of disk samples.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (1)

1. A disk anomaly detection method based on neighborhood partition and isolation reconstruction is disclosed, and the method comprises the following steps:

(1) collecting SMART information of the magnetic discs, screening effective magnetic disc attribute characteristics to form a data set, and performing exponential smoothing processing on the data set to obtain a stable magnetic disc training set, which specifically comprises the following steps: collecting the SMART attribute data of the disks and screening out the SMART attributes of the disks which have no deficiency and effectively change along with time, wherein the SMART attributes of the disks comprise: a current value of a bottom layer data read error rate, an original value of a bottom layer data read error rate, a current value of a disk read-write flux performance, an original value of a disk read-write flux performance, a current value of a spindle spin-up time, an original value of a spindle spin-up time, a current value of a start-stop count, an original value of a start-stop count, a current value of a remapped sector count, an original value of a remapped sector count, a current value of a seek error rate, an original value of a seek error rate, a current value of a seek performance, an original value of a seek performance, a current value of a power-on time accumulation, an original value of a power-on time accumulation, a current value of a spindle spin-up retry number, an original value of a spindle spin-up retry number, a current value of a power-on cycle count, an original value of a power-on cycle count, a current value of a serial port slow-down error count, an original value of a serial port slow-down error count, a current value of an I/O error detection and correction, a current value of a disk read-write flux performance, a remapped sector count, a current value of a seek error rate correction, a seek error correction, a seek performance, a performance, An original value of I/O error detection and correction, a current value of uncorrectable errors, an original value of uncorrectable errors, a current value of command timeout, an original value of command timeout, a current value of high fly write, an original value of high fly write, a current value of air flow temperature, an original value of air flow temperature, a current value of shock error rate, an original value of shock error rate, a current value of power down return count, an original value of power down return count, a current value of head load/unload count, an original value of head load/unload count, a current value of temperature, an original value of temperature, a current value of program error block count, an original value of program error block count, a current value of current to-be-mapped sector count, an original value of current to-be-mapped sector count, a current value of offline uncorrectable sector count, an original value of offline uncorrectable sector count, a current value of power-off error block count, a power-off error rate, a power-on-off power-on power-off power-on power-off power-on power-off power-on power-off power-on power-off power-on-off power-on-power-off power-on-off power-on-power-off power-on-power-off power-on-power-on-power-off-power-off-power-off-power-off-power-on-power-off-power-on-power-off-on-off-power-on-off-, The method comprises the following steps of generating a SMART attribute into a sequence which can be used for generating a model by an exponential smoothing method by using collected data as a data set, wherein the current value of the Ultra access check error rate, the original value of the Ultra access check error rate, the current value of the head flight time/transmission error rate, the original value of the head flight time/transmission error rate, the current value of the total number of LBA writes, the original value of the total number of LBA writes, the current value of the total number of LBA reads and the original value of the total number of LBA reads, and the exponential smoothing formula is defined as follows:

S _t ＝α·Y _t +(1-α)·S _t-1

wherein t is time, Y _t Is the actual value of the t-th data, S _t Is a smooth value of the previous t data, is recursively calculated according to the actual value of the time t and the smooth value of the previous t-1 data, and the width of the window is fixed to k, k is equal to [1,5 ]]The parameter alpha is an exponential smoothing coefficient, and alpha belongs to [0,1 ]]；

(2) The method comprises the following steps of randomly sampling a disk data set for multiple times to obtain multiple sub-training sets, calculating the distance between each point in a subset and the nearest point of each point in the subset by combining Euclidean distance, constructing a disk isolation area by taking the distance as a radius, and taking test points which do not belong to any area as global anomalies, wherein the method specifically comprises the following steps: carrying out simple random sampling on the training set D for multiple times to obtain a plurality of sub-training sets S with the sample size phi _i I is an integer, i is more than or equal to 1 and less than or equal to t, t is the number of subsets, a proper numerical value can be selected according to actual conditions, and S is arranged in each sub-training set _i In, based on Euclidean distance meterCalculating the distance between each point, taking each training point a as the center of the region, and taking a to its nearest training point eta _a Is used as a zone radius to construct a disk isolation zone that isolates point a from other training points in the subset, where a, η _a ∈S _i The radial distance τ (a) for point a is defined as follows:

τ(a)＝||a-η _a ||

for each sub-training set S _i Let c be the training point closest to the test point x, c ∈ S _i For a test point x, x is a global anomaly if and only if τ (c) < τ (x), τ (x) and τ (c) are the radial distances of point x and point c, respectively,

is a boundary that determines whether x is a global exception;

(3) for the test point with non-global abnormality, finding out all the training points in the area where the test point is located and the nearest training point of the training points, and taking the ratio of the radiuses of the areas where the corresponding two points are located as the abnormality metric value of the test point before reconstruction in the area, specifically: for each sub-training set S _i B is S _i In the above description, when a hypersphere constructed with b as the center of sphere and τ (b) as the radius is denoted as b (b), the following definition is applied to any training point y in b (b):

y:||y-b||＜τ(b)

for test points which are not global anomalies, let c be the training point nearest to the test point x, η _c Is the training point nearest to c, η _c ∈S _i ，B(η _c ) And B (c) are each eta _c And c is the center of the sphere, in τ (. eta.) _c ) And τ (c) is the hypersphere region of radius, B (η) _c ) And B (c) the ratio of the radii

Is the anomaly measure of training point c relative to its neighborhood, will

As a measureThe test point x is the value of the anomaly metric before reconstruction in this region,

(4) reconstructing an area after the test point is included, and taking the radius ratio of the area where the test point is located after reconstruction and before reconstruction as the abnormal metric value of the test point in the area after reconstruction, specifically: for each test point x, setting a point c E to S _i If the test point is the training point contained in the area where the test point x is located, the original training point establishes an area again according to the nearest neighbor principle after the test point x is contained, the radius of the area of the point c after reconstruction is recorded as tau (c)', and the radius of the area where the test point x is located after reconstruction is larger than that of the area before reconstruction

The abnormal metric value of the test point after reconstruction in the area is obtained;

(5) combining the two measurement values to obtain the reconstruction score of the test point in one area, taking the reciprocal of the sum of the reconstruction scores of all the areas where the test point is located as an isolation score, and taking the average value of the isolation scores in a plurality of subsets as the abnormal score of the test point, wherein the method specifically comprises the following steps: for each test point x, setting a point c E to S _i If the training point included in the region where x is located is x, the reconstruction score r (x) is defined as follows:

wherein τ (c)' is the radius of an isolation region constructed by taking c as the sphere center after reconstruction, and the reciprocal of the sum of the reconstruction scores of all regions where the test points are located is taken as an isolation score, so that the isolation score A (x) is defined as follows:

k is the number of the areas where the test points x are located, and the normal sample data set is subjected toMultiple sampling to obtain multiple sub-training sets S ₁ ,S ₂ ,...,S _t T is the number of subsets, and a proper value can be selected according to actual conditions, and is respectively set in each subset S _i (1 ≦ i ≦ t) calculating the isolation score for test point x, and for the anomaly score A (x) for test point x, the following definition:

wherein A is _i (x) Is the isolation score of the test point x in the ith subset; abnormal score

The degree of abnormality of x can be measured,

the larger the test point x is, the higher the abnormal degree of the test point x is; on the contrary, the first step is to take the reverse,

smaller is indicative of a lower degree of abnormality for test point x.

Images