CN114328048A - Disk fault prediction method and device - Google Patents

Abstract

The invention discloses a disk failure prediction method and device, which acquires information of different types of disks, constructs a training data set based on the information of the disks; constructs a deep neural network model including a characteristic network structure and a classification network structure; wherein, the characteristic network The structure is used to extract the feature information of the disk based on the information of the disk; the classification network structure is used to determine whether the disk is faulty based on the feature information of the disk; the deep neural network model is trained based on the training data set, and the trained disk failure prediction model is obtained; Obtain the information of the target disk to be predicted in failure, and input the information of the target disk into the disk failure prediction model to obtain the failure prediction result of the target disk. It can be seen that the disk failure prediction model of the present application can simultaneously perform failure prediction for different types of disks, and has good generality.

Description

Disk failure prediction method and device

technical field

The present invention relates to the field of storage, in particular to a method and device for predicting a disk failure.

Background technique

With the development of emerging technologies such as cloud computing and blockchain, the demand for storage systems is increasing. At present, in the storage system, the disk is still the mainstream storage device. In the existing solution, S.M.A.R.T. (Self-Monitoring Analysis and Reporting Technology, Self-Monitoring Analysis and Reporting Technology) technology is usually used to predict whether the disks in the storage system are faulty. The normal threshold range is determined. When the disk has an indicator value outside the normal threshold range, it is considered that the disk is pre-faulty. However, there may be more than one type of disk in the same storage system, and the types of disks in different storage systems may also be different, and the distribution of various types of indicator data for different types of disks is not the same. The indicator data sets different normal threshold ranges, resulting in poor generality of this disk failure prediction method.

Therefore, how to provide a solution to the above technical problem is a problem that those skilled in the art need to solve at present.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a disk failure prediction method and device. The disk failure prediction model can simultaneously predict the failure of different types of disks, and has good versatility.

In order to solve the above technical problems, the present invention provides a disk failure prediction method, including:

Obtain information of different types of disks, and build a training data set based on the information of the disks;

Construct a deep neural network model including a feature network structure and a classification network structure; wherein, the feature network structure is used to extract the feature information of the disk based on the information of the disk; the classification network structure is used to extract the feature information of the disk based on the feature of the disk information to determine whether the disk is faulty;

The deep neural network model is trained based on the training data set to obtain a trained disk failure prediction model;

Obtain the information of the target disk to be predicted in failure, and input the information of the target disk into the disk failure prediction model to obtain the failure prediction result of the target disk.

Optionally, obtain information of different types of disks, and build a training data set based on the information of the disks, including:

Obtain various types of indicator data of different types of disks at different times, and obtain the failure information of the disks;

According to the failure information of the disk and the preset failure prediction advance time, label the various index data of the disk one by one to indicate whether the index data is fault data;

The various types of indicator data of the disk and their corresponding labels are combined to form the training data set.

Optionally, obtain information of different types of disks, and build a training data set based on the information of the disks, further comprising:

After acquiring various types of indicator data of different types of disks at different times, before combining various types of indicator data of the disks and their corresponding labels, erase invalid data in the various types of indicator data of the disks Remove processing, and normalize the various index data of the disk after the erasing process, so as to combine the normalized index data of the disk and their corresponding labels to form the the training dataset.

Optionally, construct a deep neural network model including a characteristic network structure and a classification network structure; train the deep neural network model based on the training data set, and obtain a trained disk failure prediction model, including:

Construct a feature network structure consisting of a first fully connected layer and a plurality of residual layers in turn; wherein, the feature network structure is used to extract high-dimensional implicit features corresponding to the index data one-to-one based on the various index data of the disk ;

constructing a classification network structure composed of a second fully connected layer; wherein, the output dimension of the classification network structure is 2, corresponding to fault and normal respectively;

Combining the constructed feature network structure and the classification network structure to obtain a deep neural network model;

The deep neural network model is trained based on the training data set to obtain a trained deep neural network model;

Combining the high-dimensional implicit features of the disk and their corresponding labels to form a new training data set, and based on the new training data set, the XGBoost classifier used to replace the second fully connected layer is trained to obtain training The completed XGBoost classifier;

The trained XGBoost classifier is replaced with the second fully connected layer in the trained deep neural network model, and the replaced deep neural network model is used as the disk failure prediction model.

Optionally, construct a feature network structure sequentially composed of a first fully connected layer and multiple residual layers, including:

The first fully-connected layer is constructed based on F _out =X*W+b; wherein, F _out is the output vector of the first fully-connected layer; X is the input vector of the first fully-connected layer; W is the first fully-connected layer. A network weight of the fully connected layer; b is the bias of the first fully connected layer;

Construct L residual layers based on x _a+1 =x _a +F(x _a ,W _a ), F(x _a ,W _a )=Relu(x _a *W _a ); where x _a+1 is the first The output of the residual layer of layer a; x _a is the input of the residual layer of layer a; F(x _a ,W _a ) is the residual learning function of layer a; Relu is the activation function, Relu(x)=max( 0,x); 1≤a≤L and a is an integer;

The constructed first fully connected layer and the L residual layers are combined to obtain the feature network structure.

Optionally, the deep neural network model is trained based on the training data set to obtain a trained deep neural network model, including:

Inputting various index data of the disk in the training data set into the deep neural network model to obtain the failure prediction result of the disk;

Substitute the failure prediction result of the disk and the labels corresponding to various index data of the disk into a preset loss calculation function to calculate the loss to obtain the first loss;

With the optimization goal of reducing the first loss to 0, the adjustable parameters of the neural network model are optimized and adjusted using a preset back-propagation algorithm until all the training data sets are trained on the neural network model. After completion, the trained deep neural network model is obtained.

Optionally, based on the new training data set, the XGBoost classifier for replacing the second fully connected layer is trained to obtain the trained XGBoost classifier, including:

Input each high-dimensional latent feature of the disk in the new training data set into the XGBoost classifier to obtain the failure classification result of the disk;

Substitute the fault classification result of the disk and the label corresponding to each high-dimensional implicit feature of the disk into a preset loss calculation function to calculate the loss to obtain the second loss;

Taking reducing the second loss to 0 as the optimization goal, using the preset back-propagation algorithm to optimize and adjust the adjustable parameters of the XGBoost classifier until all the new training data sets are on the XGBoost classifier After the training is completed, the trained XGBoost classifier is obtained.

Optionally, before inputting the information of the target disk into the disk failure prediction model, the disk failure prediction method further includes:

Learning the optimal structural parameters of the disk failure prediction model according to a preset heuristic genetic algorithm;

The structure parameters of the disk failure prediction model are adjusted according to the optimal structure parameters, so as to input the information of the target disk into the disk failure prediction model of the optimal structure.

Optionally, learn the optimal structural parameters of the disk failure prediction model according to a preset heuristic genetic algorithm, including:

Combining the structural parameters of the disk failure prediction model into a parameter vector, initializing the parameter vector randomly, and adding the initialized parameter vector to a preset vector priority queue; wherein, the vector priority queue is a large root Heap structure, the sorting key of the big root heap structure is the false alarm rate score value corresponding to each of the parameter vectors;

Adjust the structural parameters of the disk failure prediction model according to the parameter vector after initialization, so as to obtain the false alarm rate score value of the disk failure prediction model under the current structural parameters, and return the value of performing random initialization of the parameter vector. step until the initialization times reach the preset times threshold;

A preset first number of parameter vector pairs are randomly selected from the vector priority queue, and each parameter vector pair is crossed according to param _new =(param ₁ +param ₂ )/2 to obtain the disk The new false alarm rate score value of the fault prediction model under the parameter vector param _new ; wherein, param ₁ and param ₂ are two parameter vectors in each of the parameter vector pairs;

Sort the new false alarm rate score value and the current corresponding false alarm rate score value of the vector priority queue from small to large, and only keep the parameter vector of the second preset number first in the vector in the priority queue;

Traverse all parameter vectors in the vector priority queue, and mutate each parameter vector param according to param _var =param+rand*step based on mutation probability p=e ^iteration/3 to obtain the disk failure prediction model The score value of the mutation false alarm rate under the parameter vector param _var ; among them, iteration is the number of iterations; step is the basic unit of mutation of each parameter vector; rand is the random number generated in each mutation process;

Sort the variable false alarm rate score value and the current corresponding false alarm rate score value of the vector priority queue from small to large, and only keep the parameter vector of the second preset number first in the vector in the priority queue, and return to execute the step of randomly selecting a preset first number of parameter vector pairs from the vector priority queue to enter the next iteration, until the number of iterations reaches the preset number of iterations threshold;

After all iterations are completed, the parameter vector with the smallest false alarm rate score value is selected from the finally obtained vector priority queue as the optimal structure parameter.

In order to solve the above technical problems, the present invention also provides a disk failure prediction device, including:

memory for storing computer programs;

The processor is configured to implement the steps of any one of the above-mentioned methods for predicting disk failures when executing the computer program.

The invention provides a disk failure prediction method, which obtains information of different types of disks, and constructs a training data set based on the information of the disks; constructs a deep neural network model including a characteristic network structure and a classification network structure; The characteristic information of the disk is extracted from the information based on the disk; the classification network structure is used to determine whether the disk is faulty based on the characteristic information of the disk; the deep neural network model is trained based on the training data set, and the trained disk failure prediction model is obtained; information of the target disk for failure prediction, and input the information of the target disk into the disk failure prediction model to obtain the failure prediction result of the target disk. It can be seen that the disk failure prediction model of the present application can simultaneously perform failure prediction for different types of disks, and has good generality.

The present invention also provides a disk failure prediction device, which has the same beneficial effects as the above-mentioned failure prediction method.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the prior art and the accompanying drawings required in the embodiments. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a flowchart of a method for predicting a disk failure according to an embodiment of the present invention;

2 is a schematic diagram of a characteristic network structure provided by an embodiment of the present invention;

3 is a schematic diagram of a disk failure prediction model provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of a residual layer provided by an embodiment of the present invention;

FIG. 5 is a graph of variation of mutation probability with iteration times according to an embodiment of the present invention.

Detailed ways

The core of the present invention is to provide a disk failure prediction method and device. The disk failure prediction model can simultaneously perform failure prediction for different types of disks, and has good versatility.

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to FIG. 1 , which is a flowchart of a method for predicting a disk failure according to an embodiment of the present invention.

The disk failure prediction method includes:

Step S1: Obtain information of different types of disks, and build a training data set based on the information of the disks.

Specifically, the present application obtains information of different types of disks to construct a training data set DataSet ₁ based on the information of different types of disks (training is used for a deep neural network model for predicting disk failures)

Step S2: constructing a deep neural network model including a characteristic network structure and a classification network structure, and training the deep neural network model based on the training data set to obtain a trained disk failure prediction model.

Specifically, the present application constructs a deep neural network model including a feature network structure and a classification network structure, wherein the feature network structure is used to extract the feature information of the disk based on the information of the disk; the classification network structure is used to determine the disk based on the feature information of the disk. Whether it is faulty, that is, the information of a disk is input into the deep neural network model, and the deep neural network model can output the fault prediction result of the disk.

The present application trains the deep neural network model based on the training data set DataSet ₁ , in order to improve the accuracy of model prediction by training the deep neural network model, and finally obtain a disk failure prediction model with high accuracy.

Step S3: Obtain the information of the target disk to be predicted in failure, and input the information of the target disk into the disk failure prediction model to obtain the failure prediction result of the target disk.

Specifically, after obtaining a disk failure prediction model with high accuracy, the present application can obtain the information of the target disk to be predicted in failure, and then input the information of the target disk to be predicted in failure into the disk failure prediction model, and the target disk can be obtained failure prediction results.

It can be seen that the disk failure prediction model of the present application can simultaneously perform failure prediction for different types of disks, and has good generality.

On the basis of the above-mentioned embodiment:

As an optional embodiment, information of different types of disks is obtained, and a training data set is constructed based on the information of the disks, including:

Obtain various index data of different types of disks at different times, and obtain disk failure information;

According to the failure information of the disk and the preset failure prediction lead time, label the various index data of the disk one by one to indicate whether the index data is fault data;

The various index data of the disk and their corresponding labels are combined to form a training data set.

Specifically, the construction process of the training data set of the present application includes: 1) Obtaining various types of indicator data of different types of disks at different times (that is, SMART data information of different types of disks at different times, or from Backblaze (cloud storage) ) to obtain indicator data from the open source disk data set), and obtain the failure information of different types of disks (you can know the failure date of the disk); 2) According to the failure information of any type of disk and the preset failure prediction advance time, Labels indicating whether the indicator data is fault data are attached to the various indicator data of the disk of this model. For example, if the failure prediction advance time is set to 14 days, the various types of disks on the date of the failure and the total 14 days before the failure are set. The indicator data is marked with a label indicating that the indicator data is fault data (such as 1), and the indicator data of the disk at the rest of the time is marked with a label indicating that the indicator data is normal data (such as 0); 3) Different types of disks in different All kinds of indicator data of time and their corresponding labels are combined to form a training data set DataSet ₁ .

As an optional embodiment, acquiring information of different types of disks, and constructing a training data set based on the information of the disks, also includes:

After obtaining various types of indicator data of different types of disks at different times, before combining various types of indicator data of the disk and their corresponding labels, the invalid data in the various types of indicator data of the disk is erased, and The various index data of the disk after the erasure processing is normalized, so as to combine the various index data of the normalized disk and their corresponding labels to form a training data set.

Further, after obtaining various types of index data of different types of disks at different times, the present application can also perform erasing processing on invalid data in various types of index data of different types of disks, so as to avoid invalid data from affecting model training; Then, the various index data of different types of disks after erasing processing are normalized according to x=(x _i -x _min )/(x _max -x _min ) to facilitate subsequent data processing, where x _i is the index data to be normalized; x _max is the maximum index value of the index data to be normalized; x _min is the minimum index value of the index data to be normalized; x is the index data to be normalized The normalized processing result of the indicator data; finally, the normalized various types of indicator data and their corresponding labels of different types of disks at different times are combined to form a training data set DataSet ₁ .

As an optional embodiment, construct a deep neural network model including a feature network structure and a classification network structure; train the deep neural network model based on the training data set, and obtain a trained disk failure prediction model, including:

Constructing a feature network structure consisting of a first fully connected layer and multiple residual layers in turn; wherein, the feature network structure is used to extract high-dimensional hidden features corresponding to the index data one-to-one based on various index data of the disk;

Build a classification network structure composed of a second fully connected layer; wherein, the output dimension of the classification network structure is 2, corresponding to fault and normal respectively;

Combining the constructed feature network structure and classification network structure to obtain a deep neural network model;

The deep neural network model is trained based on the training data set, and the trained deep neural network model is obtained;

Combine the high-dimensional latent features of the disk and their corresponding labels to form a new training data set, and train the XGBoost classifier used to replace the second fully connected layer based on the new training data set to obtain the trained XGBoost classifier;

The trained XGBoost classifier is replaced with the second fully connected layer in the trained deep neural network model, and the replaced deep neural network model is used as the disk failure prediction model.

Specifically, the construction process of the disk failure prediction model of the present application includes: 1) As shown in FIG. 2 , the construction consists of a first FC (Fully Connected, fully connected) layer and a plurality of RESNET (Residual Network, residual network) layers in turn. The feature network structure FeatureNet composed of; wherein, the feature network structure is used to extract the high-dimensional hidden feature feature _h corresponding to the various index data x of the disk based on the various index data x of the disk, that is, feature _h = x·FeatureNet; 2) Construct the classification network structure ClassifierNet composed of the second FC layer, that is, the output of the classification network structure out=feature _h ClassifierNet; it should be noted that the output out part of the classification network structure needs to do softmax on the output of ClassifierNet (for Multi-classification function) processing, the dimension of output out is 2, corresponding to fault (1) and normal (0) respectively; 3) The constructed feature network structure (the first FC layer + multiple RESNET layers) and the classification network structure (The second FC layer) is combined to obtain a deep neural network model; 4) The deep neural network model is trained based on the training data set DataSet ₁ , and the trained deep neural network model is obtained; 5) Different types of disks at different times Various types of indicator data have a one-to-one corresponding label, and various types of indicator data of different types of disks at different times have a one-to-one corresponding high-dimensional implicit feature, so the high-dimensional implicit features of different types of disks at different times also correspond one-to-one. Label, each high-dimensional latent feature of different types of disks at different times and their corresponding labels are combined to form a new training data set DataSet ₂ ; 6) based on the new training data set DataSet ₂ to replace the XGBoost of the second FC layer (ExtremeGradient Boosting, Extreme Gradient Boosting) classifier is trained, and the trained XGBoost classifier is obtained; 7) The trained XGBoost classifier is replaced with the second FC layer in the trained deep neural network model, and the replaced The deep neural network model is used as the disk failure prediction model, that is, the final disk failure prediction model is the feature network structure FeatureNet+XGBoost classifier (as shown in Figure 3), which is used for disk failure prediction.

As an optional embodiment, construct a feature network structure sequentially composed of a first fully connected layer and multiple residual layers, including:

The first fully connected layer is constructed based on F _out =X*W+b; wherein, F _out is the output vector of the first fully connected layer; X is the input vector of the first fully connected layer; W is the network of the first fully connected layer weight; b is the bias of the first fully connected layer;

Construct L residual layers based on x _a+1 =x _a +F(x _a ,W _a ), F(x _a ,W _a )=Relu(x _a *W _a ); where x _a+1 is the first The output of the residual layer of layer a; x _a is the input of the residual layer of layer a; F(x _a ,W _a ) is the residual learning function of layer a; Relu is the activation function, Relu(x)=max( 0,x); 1≤a≤L and a is an integer;

The constructed first fully connected layer and L residual layers are combined to obtain the feature network structure.

Specifically, the construction process of the feature network structure of the present application includes: 1) constructing the first fully connected layer based on F _out =X*W+b, the function of the fully connected layer is to map the vector to another space, increasing the complexity of the model , to improve the degree of fit; where F _out is the output vector of the first fully connected layer, that is, the input vector of the first residual layer; X is the input vector of the first fully connected layer, the first fully connected layer during model training The input is the training data set DataSet _1. When the model is officially used, the first fully connected layer inputs the information of the target disk to be fault prediction; W is the network weight of the first fully connected layer; b is the bias of the first fully connected layer. 2) As shown in Figure 4, construct L residuals based on x _a+1 =x _a +F(x _a ,W _a ), F(x _a ,W _a )=Relu(x _a *W _a ) layer; where x _a+1 is the output of the residual layer of layer a, that is, the input of the residual layer of layer a+1; x _a is the input of the residual layer of layer a; F(x _a ,W _a ) is the residual learning function of the a-th layer; Relu is the activation function, Relu(x)=max(0,x); 3) Combine the constructed first fully connected layer and L residual layers to obtain a feature network Structure (the output of the last layer of residual layer Resnet _L is the high-dimensional implicit feature representation of various index data of the input disk, denoted as feature _h ).

As an optional embodiment, the deep neural network model is trained based on the training data set, and the trained deep neural network model is obtained, including:

Input the various index data of the disk in the training data set into the deep neural network model to obtain the failure prediction result of the disk;

Substitute the failure prediction result of the disk and the labels corresponding to various index data of the disk into the preset loss calculation function to calculate the loss to obtain the first loss;

With the optimization goal of reducing the first loss to 0, the preset back-propagation algorithm is used to optimize and adjust the adjustable parameters of the neural network model until all the training data sets are trained on the neural network model, and the trained deep neural network is obtained. network model.

Specifically, the training process of the deep neural network model of the present application includes: 1) inputting various index data of the disk in the training data set DataSet ₁ into the deep neural network model to obtain the failure prediction result of the disk; The labels corresponding to the prediction results and various index data of the disk are substituted into the preset loss calculation function (such as the CrossEntropy function) for loss calculation, and the first loss is obtained; 3) The optimization goal is to reduce the first loss to 0 ( That is to say, the prediction result of the disk failure of the deep neural network model is as consistent as possible with the actual failure situation of the disk as the optimization goal), and the preset back-propagation algorithm is used to optimize and adjust the adjustable parameters of the neural network model until the training data set DataSet ₁ All are trained on the neural network model, and the trained deep neural network model is obtained.

As an optional embodiment, the XGBoost classifier used to replace the second fully connected layer is trained based on the new training data set, and the trained XGBoost classifier is obtained, including:

Input the high-dimensional latent features of the disks in the new training data set into the XGBoost classifier to obtain the failure classification results of the disks;

Substitute the fault classification result of the disk and the labels corresponding to each high-dimensional implicit feature of the disk into the preset loss calculation function to calculate the loss to obtain the second loss;

With the optimization goal of reducing the second loss to 0, the preset back-propagation algorithm is used to optimize and adjust the adjustable parameters of the XGBoost classifier until all the new training data sets are trained on the XGBoost classifier, and the trained XGBoost classifier is obtained. Classifier.

Specifically, the training process of the XGBoost classifier of the present application includes: 1) inputting each high-dimensional implicit feature of the disk in the new training data set DataSet ₂ into the XGBoost classifier to obtain the failure classification result of the disk; 2) classifying the failure of the disk The result and the labels corresponding to the high-dimensional hidden features of the disk are substituted into the preset loss calculation function (such as the CrossEntropy function) for loss calculation, and the second loss is obtained; 3) The optimization goal is to reduce the second loss to 0 (that is, use the XGBoost classifier The optimal disk failure classification result is as consistent as possible with the actual disk failure situation as the optimization goal), and the tunable parameters of the XGBoost classifier are optimized and adjusted by using the preset backpropagation algorithm until the new training dataset DataSet ₂ is all in the XGBoost classifier. After the above training is completed, the trained XGBoost classifier is obtained.

It can be seen that this application uses a deep neural network to extract the high-dimensional implicit features of various index data of the disk, and uses the XGBoost classifier to replace the fully connected classification layer of the deep neural network. A better function approximation tool can construct a better classifier, which can greatly reduce the false alarm rate while ensuring a relatively high prediction accuracy.

As an optional embodiment, before the information of the target disk is input into the disk failure prediction model, the disk failure prediction method further includes:

The optimal structural parameters of the disk failure prediction model are learned according to the preset heuristic genetic algorithm;

Adjust the structure parameters of the disk failure prediction model according to the optimal structure parameters, so as to input the information of the target disk into the disk failure prediction model with the optimal structure.

Further, before inputting the information of the target disk into the disk failure prediction model, the present application can also learn the optimal structural parameters of the disk failure prediction model according to the preset heuristic genetic algorithm (such as learning the neurons of the first fully connected layer. The number m, the number of layers of the residual network L, the batch size of the model training batch_size, the number of XGboost decision trees n and the optimal parameter value of the maximum depth k of each decision tree), and adjust the disk failure prediction according to the optimal structural parameters The structural parameters of the model are used to input the information of the target disk into the disk failure prediction model of the optimal structure for failure prediction.

As an optional embodiment, the optimal structural parameters of the disk failure prediction model are learned according to a preset heuristic genetic algorithm, including:

Combine the structural parameters of the disk failure prediction model into a parameter vector, initialize the parameter vector randomly, and add the initialized parameter vector to the preset vector priority queue; wherein, the vector priority queue is a large root heap structure, and the order of the large root heap structure key is the false alarm rate score value corresponding to each parameter vector;

Adjust the structural parameters of the disk failure prediction model according to the initialized parameter vector to obtain the false alarm rate score value of the disk failure prediction model under the current structural parameters, and return to the step of randomly initializing the parameter vector until the initialization times reach the preset value number of thresholds;

A preset first number of parameter vector pairs are randomly selected from the vector priority queue, and each parameter vector pair is crossed according to param _new =(param ₁ +param ₂ )/2, so as to obtain the disk failure prediction model in the parameter vector The new false alarm rate score value under param _new ; where param ₁ and param ₂ are the two parameter vectors in each parameter vector pair;

Sort the new false alarm rate score value and the current corresponding false alarm rate score value of the vector priority queue from small to large, and only keep the parameter vector of the second preset number in the vector priority queue;

Traverse all parameter vectors in the vector priority queue, and mutate each parameter vector param according to param _var =param+rand*step based on the mutation probability p=e ^iteration/3 to obtain the disk failure prediction model under the parameter vector param _var where, iteration is the number of iterations; step is the basic unit of mutation of each parameter vector; rand is the random number generated in each mutation process;

Sort the variable false alarm rate score value and the current corresponding false alarm rate score value of the vector priority queue from small to large, and only keep the preset second number of parameter vectors in the first order in the vector priority queue, and return Perform the step of randomly selecting a preset first number of parameter vector pairs from the vector priority queue to enter the next iteration, until the number of iterations reaches the preset number of iterations threshold;

After all iterations are over, the parameter vector with the smallest false alarm rate score value is selected from the final vector priority queue as the optimal structure parameter.

Specifically, the learning process of the optimal structural parameters of the disk failure prediction model of the present application includes: 1) Combining the structural parameters of the disk failure prediction model into a parameter vector, such as the number of neurons in the first fully connected layer m, the residual error The number of layers L of the network, the batch size batch_size of model training, the number n of XGboost decision trees, and the maximum depth k of each decision tree are combined into a 5-dimensional parameter vector [m, L, batch_size, n, k], denoted as param , and randomly initialize the parameter vector param, and add the initialized parameter vector param to the preset vector priority queue queue; among them, the vector priority queue queue is a big root heap structure (big root heap is a complete binary tree), the sorting key of the big root heap structure (Node key value) is the false alarm rate score value corresponding to each parameter vector; 2) Adjust the structural parameters (m, L, batch_size, n, k) of the disk failure prediction model according to the initialized parameter vector param to obtain The score value of the false alarm rate of the disk failure prediction model under the current structural parameters, and returns to the step of executing the random initialization parameter vector param until the initialization times reach the preset number of times threshold (such as 10 times); 3) From the vector priority queue queue Randomly select (2*preset first number) parameter vectors, and form a team of (2*preset first number) parameter vectors in pairs to obtain a preset first number (such as 20) of parameter vector pairs, Perform cross operation on each parameter vector pair according to param _new =(param ₁ +param ₂ )/2 to obtain the new false alarm rate score value of the disk failure prediction model under each parameter vector param _new ; where, param ₁ and param ₂ are two parameter vectors in each parameter vector pair; 4) Sort the new false alarm rate score value of the first number and the current corresponding false alarm rate score value of the vector priority queue from small to large, Only the parameter vector param of the second preset number (such as 20) in the first order is kept in the vector priority queue queue, that is, only the parameter vector param with the smallest false alarm rate score value in the first 20 is kept, and the other parameter vector param is done. Natural selection is discarded; 5) Traverse all parameter vectors param in the vector priority queue queue, and mutate each parameter vector param according to param _var =param+rand*step based on mutation probability p=e ^iteration/3 to obtain disk failure The score value of the mutation false alarm rate of the prediction model under each parameter vector param _var ; among them, iteration is the number of iterations, and p is the mutation probability. The relationship between the two is shown in Figure 5. The more iterations, the higher the mutation probability. small; step is the basic unit of variation of each parameter vector, such as step=[10,1,16,5,1], the basic unit of variation of the number m of neurons in the first fully connected layer is 10, and the same is true for other variables; ran d is a random number generated in each mutation process, which can be generated between [-10, 10]; 6) The score value of the false alarm rate of each mutation and the current corresponding value of the false alarm rate of the vector priority queue are from small To the large sorting, only the preset second number of parameter vectors in the first order are retained in the vector priority queue, that is, only the first 20 parameter vectors param with the smallest false alarm rate score are retained, and other parameter vectors param are selected naturally. Discard, and return to step 3) to enter the next iteration until the number of iterations reaches the preset number of iterations threshold; 7) After all iterations are over, select the parameter vector with the smallest false alarm rate score from the final vector priority queue. as the optimal structural parameter.

It can be seen that this strategy solves the optimal model parameter search problem in disk failure prediction. Compared with the classical genetic algorithm, the heuristic genetic algorithm in this application optimizes the mutation and selection strategy, and the mutation probability decreases with the increase of the number of iterations, and Combined with the heuristic search method of the priority queue, the solution can be quickly searched in each dimension, and the optimal solution can be approached as fast as possible.

To sum up, this application learns a large number of samples of various index data features of disks through deep neural networks, excavates the characteristics of each feature and the implicit correlation between different feature combinations, and extracts higher-dimensional features. These features use an ensemble tree model for fault prediction, and use an automatic learning strategy based on genetic algorithm to learn the optimal network structure, which greatly reduces the false positive rate while maintaining a high recognition accuracy. The deep learning-based disk failure prediction method of the present application is suitable for operation and maintenance scenarios with large disk usage and large data such as cloud platforms, storage service providers, government data centers, etc., and can perform fault analysis and analysis of complex disk data information. Prediction, discovering problematic disks in advance and making corresponding preparations and maintenance can improve the efficiency of operation and maintenance personnel, reduce operation and maintenance costs, and improve product competitiveness.

The present application also provides a disk failure prediction device, including:

memory for storing computer programs;

The processor is configured to implement the steps of any one of the above-mentioned disk failure prediction methods when executing the computer program.

For the introduction of the disk failure prediction device provided by the present application, please refer to the above-mentioned embodiments of the disk failure prediction method, which will not be repeated in this application.

It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities or operations. There is no such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A disk failure prediction method is characterized by comprising the following steps:

acquiring information of disks of different models, and constructing a training data set based on the information of the disks;

constructing a deep neural network model comprising a characteristic network structure and a classification network structure; the characteristic network structure is used for extracting characteristic information of the disk based on information of the disk; the classification network structure is used for judging whether the disk fails or not based on the characteristic information of the disk;

training the deep neural network model based on the training data set to obtain a trained disk failure prediction model;

and acquiring information of a target disk to be subjected to failure prediction, and inputting the information of the target disk into the disk failure prediction model to obtain a failure prediction result of the target disk.

2. The disk failure prediction method of claim 1, wherein obtaining information of disks of different models and constructing a training data set based on the information of the disks comprises:

acquiring various index data of different types of disks at different time, and acquiring fault information of the disks;

according to the fault information of the disk and preset fault prediction advance time, labels indicating whether the index data are fault data or not are marked on various index data of the disk one by one;

and combining the various index data of the disk and the corresponding labels thereof to form the training data set.

3. The disk failure prediction method of claim 2, wherein obtaining information of disks of different models and constructing a training data set based on the information of the disks further comprises:

after various types of index data of different types of disks at different times are obtained, before the various types of index data of the disks and corresponding labels are combined, the invalid data in the various types of index data of the disks are erased, and the various types of index data of the disks after the erasure processing are normalized, so that the various types of index data of the disks after the normalization processing and the corresponding labels are combined to form the training data set.

4. The disk failure prediction method of claim 2, wherein a deep neural network model is constructed that includes a feature network structure and a classification network structure; training the deep neural network model based on the training data set to obtain a trained disk failure prediction model, and the method comprises the following steps:

constructing a characteristic network structure sequentially consisting of a first full-connection layer and a plurality of residual error layers; the characteristic network structure is used for extracting high-dimensional implicit characteristics which correspond to the index data one by one on the basis of various index data of the disk;

constructing a classification network structure consisting of second full connection layers; the output dimensionality of the classification network structure is 2, and the classification network structure corresponds to a fault and a normal respectively;

combining the constructed characteristic network structure and the classification network structure to obtain a deep neural network model;

training the deep neural network model based on the training data set to obtain a trained deep neural network model;

combining the high-dimensional hidden features of the disk and the labels corresponding to the high-dimensional hidden features to form a new training data set, and training the XGboost classifier for replacing the second full-connection layer based on the new training data set to obtain a trained XGboost classifier;

and replacing a second full connection layer in the trained deep neural network model by the trained XGboost classifier, and taking the replaced deep neural network model as the disk fault prediction model.

5. The disk failure prediction method of claim 4, wherein constructing a feature network structure consisting of a first fully-connected layer and a plurality of residual layers in sequence comprises:

based on F_outConstructing a first fully connected layer; wherein, F_outIs the output vector of the first fully-connected layer; x is the input vector of the first fully-connected layer; w is the network weight of the first fully-connected layer; b is the bias of the first fully-connected layer;

based on x_a+1＝x_a+F(x_a,W_a)、F(x_a,W_a)＝Relu(x_a*W_a) Constructing L residual error layers; wherein x is_a+1Is the output of the a-th layer residual error layer; x is the number of_aIs the input of the a-th layer residual error layer; f (x)_a,W_a) A residual learning function for layer a; relu is an activation function, Relu (x) max (0, x); a is more than or equal to 1 and less than or equal to L, and a is an integer;

and combining the constructed first full-connection layer with the L residual error layers to obtain the characteristic network structure.

6. The disk failure prediction method of claim 4, wherein training the deep neural network model based on the training data set to obtain a trained deep neural network model comprises:

inputting various index data of the disk in the training data set to the deep neural network model to obtain a fault prediction result of the disk;

substituting the failure prediction result of the disk and labels corresponding to various index data of the disk into a preset loss calculation function to perform loss calculation to obtain a first loss;

and taking the first loss reduced to 0 as an optimization target, and optimizing and adjusting the adjustable parameters of the neural network model by using a preset back propagation algorithm until the training of the training data set on the neural network model is completed, so as to obtain a trained deep neural network model.

7. The disk failure prediction method of claim 4, wherein training the XGboost classifier used to replace the second fully-connected layer based on the new training data set to obtain a trained XGboost classifier comprises:

inputting each high-dimensional implicit characteristic of the disk in the new training data set into the XGboost classifier to obtain a fault classification result of the disk;

substituting the fault classification result of the disk and the label corresponding to each high-dimensional hidden feature of the disk into a preset loss calculation function to perform loss calculation to obtain a second loss;

and taking the second loss reduced to 0 as an optimization target, and optimizing and adjusting the adjustable parameters of the XGboost classifier by using a preset back propagation algorithm until the new training data set is completely trained on the XGboost classifier, so as to obtain the trained XGboost classifier.

8. The disk failure prediction method according to any one of claims 1 to 7, wherein before inputting the information of the target disk to the disk failure prediction model, the disk failure prediction method further comprises:

learning the optimal structure parameters of the disk fault prediction model according to a preset heuristic genetic algorithm;

and adjusting the structural parameters of the disk failure prediction model according to the optimal structural parameters so as to input the information of the target disk into the disk failure prediction model with an optimal structure.

9. The disk failure prediction method of claim 8, wherein learning the optimal structural parameters of the disk failure prediction model according to a predetermined heuristic genetic algorithm comprises:

synthesizing the structural parameters of the disk failure prediction model into a parameter vector, randomly initializing the parameter vector, and adding the initialized parameter vector to a preset vector priority queue; the vector priority queue is a big root heap structure, and the sorting key of the big root heap structure is the false alarm rate score value corresponding to each parameter vector;

adjusting the structural parameters of the disk failure prediction model according to the initialized parameter vector to obtain a false alarm rate score value of the disk failure prediction model under the current structural parameters, and returning to the step of executing the random initialization of the parameter vector until the initialization times reach a preset time threshold;

randomly selecting a preset first number of parameter vector pairs from the vector priority queue, and enabling each parameter vector pair to be in accordance with param_new＝(param₁+param₂) Performing a crossover operation to obtain the disk failurePrediction model in parameter vector param_newObtaining a new false alarm rate score value; wherein param₁And param₂For both of said pairs of parameter vectors;

sorting the new false alarm rate score values and the current corresponding false alarm rate score values of the vector priority queue from small to large, and only keeping the parameter vectors sorted in the first preset second quantity in the vector priority queue;

traversing all parameter vectors in the vector priority queue and based on the variation probability p ═ e^iteration/3Following each of said parameter vectors param to param_varPerforming mutation on param + rand step to obtain the parameter vector param of the disk failure prediction model_varA lower variation false alarm rate score value; wherein iteration is iteration times; step is the basic unit of variation of each parameter vector; rand is a random number generated in each variation process;

sorting the variation false alarm rate score values and the false alarm rate score values currently corresponding to the vector priority queue from small to large, only keeping the parameter vectors of the preset second number sorted in the vector priority queue, and returning to the step of randomly selecting the parameter vector pairs of the preset first number from the vector priority queue for entering the next iteration until the iteration number reaches a preset iteration number threshold;

and after all iterations are finished, selecting the parameter vector with the minimum false alarm rate score value from the finally obtained vector priority queue as the optimal structure parameter.

10. A disk failure prediction apparatus, comprising:

a memory for storing a computer program;

a processor for implementing the steps of the disk failure prediction method according to any of claims 1-9 when executing said computer program.

Images