CN110222840B - A method and device for cluster resource prediction based on attention mechanism - Google Patents

Abstract

The invention discloses a cluster resource prediction method and device based on an attention mechanism, which adopts an improved attention mechanism and integrates the improved attention mechanism into an LSTM, so that the correlation among a plurality of time sequences can be mined, a scheme of predicting the resource demand in a cluster by using the plurality of time sequences is provided, the prediction accuracy is effectively improved, the resource planning can be effectively assisted, the resource utilization rate of the cluster is improved, and the operation and maintenance cost of a data center is effectively reduced.

Description

A cluster resource prediction method and device based on attention mechanism

Technical Field

The present invention relates to the technical field of cluster resource management, and more specifically to a cluster resource prediction method and device based on an attention mechanism.

Background Art

The current data center is getting bigger and bigger. Effective resource management of the cluster in the data center can improve the utilization of hardware resources, reduce operation and maintenance costs, and increase operation and maintenance profits. One of the effective ways to improve resource utilization is to predict the future resource demand of the cluster, so as to plan resources in advance and reduce resource waste.

Currently, cluster resource demand forecasting mainly uses cluster resource time series data. Common time series forecasting models include ARIMA (Autoregressive Integrated Moving Average), VAR (Vector Autoregressive), GBRT (Gradient Boosted Regression Tree), LSTM (Long Short-Term Memory Network), etc. They can all be directly used for cluster resource demand forecasting.

However, there are two main problems with current cluster resource prediction methods: 1. These methods mainly use a single time series as a feature for prediction (such as ARIMA), and almost no methods use multiple time series for resource demand prediction. The accuracy of the prediction depends on whether the historical values of this time series contain obvious rules; 2. Although there are many general multi-time series prediction models (such as VAR), these model methods do not consider the characteristics of clusters in data centers, especially the correlation and mutual interference between application loads in clusters. The above two problems will lead to inaccurate cluster resource prediction results.

Therefore, how to provide a method for accurately predicting cluster resource results is an urgent problem that those skilled in the art need to solve.

Summary of the invention

In view of this, the present invention provides a cluster resource prediction method and device based on an attention mechanism, which uses multiple resource time series to predict future resource demand. In addition, based on the characteristics of resource usage by application loads within the cluster, an improved deep learning attention mechanism is used to mine the correlation between multiple resource demand time series, which can effectively improve the accuracy of resource prediction.

In order to achieve the above object, the present invention adopts the following technical solution:

A cluster resource prediction method based on an attention mechanism, comprising:

S1: Take the state of the first hidden layer at the previous moment, all time series data belonging to the same deployment unit as the target instance, all time series data belonging to the same host unit as the target instance, and the target time series at the historical moment as the input of the input attention layer to obtain the first input vector;

S2: Input the first input vector to the LSTM encoder to obtain the current first hidden layer state;

S3: Input the current first hidden layer state and the second hidden layer state at the previous moment into the time-dependent attention layer to obtain a context vector;

S4: Input the context vector, the second hidden layer state at the previous moment, and the target time sequence at the historical moment into the LSTM decoder to obtain the current second hidden layer state;

S5: Perform a linear transformation on the current second hidden layer state and the context vector to obtain a predicted value.

Preferably, step S1 specifically includes:

S11: taking the state of the first hidden layer at the previous moment and all time series data belonging to the same deployment unit as the target instance as the input of the deployment unit attention layer, and obtaining the deployment unit attention layer output vector;

S12: taking the state of the first hidden layer at the previous moment and all time series data belonging to the same host unit as the target instance as the input of the host unit attention layer, and obtaining the host unit attention output vector;

S13: Taking the state of the first hidden layer at the previous moment and the target time sequence at the historical moment as the input of the autocorrelation attention layer, and obtaining the output vector of the autocorrelation attention layer;

S14: Combine the deployment unit attention layer output vector, the host unit attention output vector and the autocorrelation attention layer output vector as the first input vector.

Preferably, step S11 specifically includes:

Calculate the first attention weight based on the first hidden layer state at the previous moment and all time series data belonging to the same deployment unit as the target instance;

Based on the first attention weight, the normalized deployment unit attention weight is calculated using the softmax function;

Calculate the deployment unit attention layer output vector based on the state of the first hidden layer at the previous moment, all time series data belonging to the same deployment unit as the target instance, and the normalized deployment unit attention weight;

Step S12 specifically includes:

Calculate the first-order time series correlation coefficient of each time series data belonging to the same host unit as the target instance relative to the historical target time series, and obtain the static time correlation weights of all time series in the corresponding host unit and the historical target time series;

Calculate the second attention weight based on all time series data of the first hidden layer state and the target instance belonging to the same host unit at the previous moment;

Obtaining the host unit attention weight based on the static time correlation weight and the second attention weight, and normalizing them to obtain the normalized host unit attention weight;

Calculate the host unit attention output vector based on the state of the first hidden layer at the previous moment, all time series data belonging to the same host unit as the target instance, the target time series at the historical moment, and the normalized host unit attention weight;

Step S13 specifically includes:

Calculate the correlation coefficient between the historical moment target time series in different time windows and obtain the corresponding autocorrelation weights;

Calculate the third attention weight based on the state of the first hidden layer at the previous moment and the target time sequence at different historical moments;

The autocorrelation unit attention weight is obtained based on the autocorrelation weight and the third attention weight, and normalized to obtain the normalized autocorrelation unit attention weight;

Based on the state of the first hidden layer at the previous moment, the target time series at the historical moment, and the normalized autocorrelation unit attention weight, the autocorrelation attention layer output vector is calculated.

Preferably, step S3 specifically includes:

Calculate the temporal attention layer weight based on the state of the second hidden layer at the previous moment, and normalize it to obtain the normalized temporal attention layer weight;

The context vector is calculated based on the current first hidden layer state and the normalized temporal attention layer weights.

A cluster resource prediction device based on an attention mechanism, comprising:

A first input vector calculation module is used to use the first hidden layer state at the previous moment, all time series data belonging to a deployment unit with the target instance, all time series data belonging to a host unit with the target instance, and the target time series at the historical moment as inputs to the input attention layer to obtain a first input vector;

A first hidden layer state calculation module, used for inputting the first input vector into the LSTM encoder to obtain a current first hidden layer state;

A context vector calculation module, used to input the current first hidden layer state and the second hidden layer state at the previous moment into the time correlation attention layer to obtain a context vector;

A second hidden layer state calculation module is used to input the context vector, the second hidden layer state at the previous moment and the target time sequence at the historical moment into the LSTM decoder to obtain the current second hidden layer state;

The linear transformation module is used to linearly transform the current second hidden layer state and the context vector to obtain a predicted value.

Preferably, the first input vector calculation module specifically includes:

A first computing unit is used to use the state of the first hidden layer at the previous moment and all time series data belonging to the same deployment unit as the target instance as inputs of the deployment unit attention layer to obtain an output vector of the deployment unit attention layer;

The second computing unit is used to use the state of the first hidden layer at the previous moment and all time series data belonging to the same host unit as the target instance as the input of the host unit attention layer to obtain the host unit attention output vector;

A third calculation unit is used to use the state of the first hidden layer at the previous moment and the target time sequence at the historical moment as the input of the autocorrelation attention layer to obtain an output vector of the autocorrelation attention layer;

A merging unit is used to merge the deployment unit attention layer output vector, the host unit attention output vector and the autocorrelation attention layer output vector as the first input vector.

Preferably, the first calculation unit specifically includes:

A first attention weight calculation subunit, used to calculate a first attention weight based on the first hidden layer state at the previous moment and all time series data belonging to the same deployment unit as the target instance;

A first normalized weight calculation subunit, used to calculate a normalized deployment unit attention weight using a softmax function based on the first attention weight;

A first attention layer output vector calculation subunit, used to calculate a deployment unit attention layer output vector based on the first hidden layer state at the previous moment, all time series data belonging to the same deployment unit as the target instance, and a normalized deployment unit attention weight;

The second calculation unit specifically includes:

The static time correlation weight calculation subunit is used to calculate the first-order time correlation coefficient of each time series data belonging to the same host unit as the target instance relative to the historical target time series, and obtain the static time correlation weights of all time series in the corresponding host unit and the historical target time series;

A second attention weight calculation subunit, used to calculate a second attention weight based on the first hidden layer state at the previous moment and all time series data of the target instance belonging to the same host unit;

A second normalization weight subunit, used to obtain the host unit attention weight based on the static time correlation weight and the second attention weight, and perform normalization to obtain a normalized host unit attention weight;

The second attention layer output vector calculation subunit is used to calculate the host unit attention layer output vector based on the first hidden layer state at the previous moment, all time series data belonging to the same host unit as the target instance, the target time series at the historical moment, and the normalized host unit attention weight;

The third calculation unit specifically includes:

The autocorrelation weight calculation subunit is used to calculate the correlation coefficient between the historical moment target time series in different time windows and obtain the corresponding autocorrelation weight;

A third attention weight calculation subunit, used to calculate a third attention weight based on the state of the first hidden layer at the previous moment and the target time series at different historical moments;

A third normalization weight subunit, used to obtain an autocorrelation unit attention weight based on the autocorrelation weight and the third attention weight, and perform normalization to obtain a normalized autocorrelation unit attention weight;

The third attention layer output vector is a quantum unit, which calculates the autocorrelation attention layer output vector based on the state of the first hidden layer at the previous moment, the target time series at the historical moment, and the normalized autocorrelation unit attention weight.

Preferably, the context vector calculation module specifically includes:

A fourth normalization weight subunit is used to calculate the temporal attention layer weight based on the state of the second hidden layer at the previous moment, and perform normalization to obtain a normalized temporal attention layer weight;

The context vector calculation subunit is used to calculate the context vector based on the current first hidden layer state and the normalized temporal attention layer weight.

It can be seen from the above technical solution that compared with the prior art, the present invention discloses a cluster resource prediction method and device based on the attention mechanism, which adopts an improved attention mechanism and integrates it into LSTM, can mine the correlation between multiple time series, and proposes a solution for using multiple time series to predict resource demand within the cluster, which effectively improves the prediction accuracy and can more effectively assist resource planning, thereby improving the resource utilization of the cluster and more effectively reducing the operation and maintenance costs of the data center.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

FIG1 is a flow chart of a cluster resource prediction method based on an attention mechanism provided by the present invention;

FIG2 is a specific flow chart of calculating the first input vector provided by the present invention;

FIG3 is a schematic diagram of a cluster resource prediction device based on an attention mechanism provided by the present invention;

FIG4 is a schematic diagram showing the composition of a first input vector calculation module provided by the present invention;

FIG5 is a schematic diagram of a time series acquisition architecture provided by the present invention;

FIG6 is a schematic diagram of a prediction model based on an attention mechanism provided by the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Referring to FIG. 1 , an embodiment of the present invention discloses a cluster resource prediction method based on an attention mechanism, comprising:

S1: Take the state of the first hidden layer at the previous moment, all time series data belonging to the same deployment unit as the target instance, all time series data belonging to the same host unit as the target instance, and the target time series at the historical moment as the input of the input attention layer to obtain the first input vector;

S2: Input the first input vector to the LSTM encoder to obtain the current first hidden layer state;

S3: Input the current first hidden layer state and the second hidden layer state at the previous moment into the time-dependent attention layer to obtain a context vector;

S4: Input the context vector, the second hidden layer state at the previous moment, and the target time sequence at the historical moment into the LSTM decoder to obtain the current second hidden layer state;

S5: Linearly transform the current second hidden layer state and context vector to obtain a predicted value.

When performing resource prediction, the present invention uses multiple resources and time series instead of one time series for prediction. In addition, based on the characteristics of resource usage by application loads within the cluster, an improved deep learning attention mechanism is used to mine the correlation between multiple resource time series, ultimately effectively improving the accuracy of resource prediction.

Referring to FIG. 2 , in order to further optimize the above technical solution, the embodiment of the present invention further discloses that step S1 specifically includes:

S11: taking the state of the first hidden layer at the previous moment and all time series data belonging to the same deployment unit as the target instance as the input of the deployment unit attention layer, and obtaining the deployment unit attention layer output vector;

S12: taking the state of the first hidden layer at the previous moment and all time series data belonging to the same host unit as the target instance as the input of the host unit attention layer, and obtaining the host unit attention output vector;

S13: Taking the state of the first hidden layer at the previous moment and the target time sequence at the historical moment as the input of the autocorrelation attention layer, and obtaining the output vector of the autocorrelation attention layer;

S14: Combine the deployment unit attention layer output vector, the host unit attention output vector and the autocorrelation attention layer output vector as the first input vector.

In order to further optimize the above technical solution, the embodiment of the present invention further discloses that step S11 specifically includes:

Calculate the first attention weight based on the first hidden layer state at the previous moment and all time series data belonging to the same deployment unit as the target instance;

Based on the first attention weight, the normalized deployment unit attention weight is calculated using the softmax function;

Calculate the deployment unit attention layer output vector based on the state of the first hidden layer at the previous moment, all time series data belonging to the same deployment unit as the target instance, and the normalized deployment unit attention weight;

Step S12 specifically includes:

Calculate the first-order time series correlation coefficient of each time series data belonging to the same host unit as the target instance relative to the historical target time series, and obtain the static time correlation weights of all time series in the corresponding host unit and the historical target time series;

Calculate the second attention weight based on all time series data of the first hidden layer state and the target instance belonging to the same host unit at the previous moment;

Obtaining the host unit attention weight based on the static time correlation weight and the second attention weight, and normalizing them to obtain the normalized host unit attention weight;

Calculate the host unit attention output vector based on the state of the first hidden layer at the previous moment, all time series data belonging to the same host unit as the target instance, the target time series at the historical moment, and the normalized host unit attention weight;

Step S13 specifically includes:

Calculate the correlation coefficient between the historical moment target time series in different time windows and obtain the corresponding autocorrelation weights;

Calculate the third attention weight based on the state of the first hidden layer at the previous moment and the target time sequence at different historical moments;

The autocorrelation unit attention weight is obtained based on the autocorrelation weight and the third attention weight, and normalized to obtain the normalized autocorrelation unit attention weight;

Based on the state of the first hidden layer at the previous moment, the target time series at the historical moment, and the normalized autocorrelation unit attention weight, the autocorrelation attention layer output vector is calculated.

In order to further optimize the above technical solution, the embodiment of the present invention further discloses that step S3 specifically includes:

Calculate the temporal attention layer weight based on the state of the second hidden layer at the previous moment, and normalize it to obtain the normalized temporal attention layer weight;

The context vector is calculated based on the current first hidden layer state and the normalized temporal attention layer weights.

Referring to FIG. 3 , an embodiment of the present invention further discloses a cluster resource prediction device based on an attention mechanism, comprising:

A first input vector calculation module is used to use the first hidden layer state at the previous moment, all time series data belonging to a deployment unit with the target instance, all time series data belonging to a host unit with the target instance, and the target time series at the historical moment as inputs to the input attention layer to obtain a first input vector;

A first hidden layer state calculation module, used for inputting the first input vector into the LSTM encoder to obtain the current first hidden layer state;

A context vector calculation module is used to input the current first hidden layer state and the second hidden layer state at the previous moment into the time correlation attention layer to obtain a context vector;

The second hidden layer state calculation module is used to input the context vector, the second hidden layer state at the previous moment and the target time sequence at the historical moment into the LSTM decoder to obtain the current second hidden layer state;

The linear transformation module is used to linearly transform the current second hidden layer state and the context vector to obtain a predicted value.

Referring to FIG. 4 , in order to further optimize the above technical solution, an embodiment of the present invention further discloses that the first input vector calculation module specifically includes:

A first computing unit is used to use the state of the first hidden layer at the previous moment and all time series data belonging to the same deployment unit as the target instance as inputs of the deployment unit attention layer to obtain an output vector of the deployment unit attention layer;

The second computing unit is used to use the state of the first hidden layer at the previous moment and all time series data belonging to the same host unit as the target instance as the input of the host unit attention layer to obtain the host unit attention output vector;

A third calculation unit is used to use the state of the first hidden layer at the previous moment and the target time sequence at the historical moment as the input of the autocorrelation attention layer to obtain an output vector of the autocorrelation attention layer;

A merging unit is used to merge the deployment unit attention layer output vector, the host unit attention output vector and the autocorrelation attention layer output vector as the first input vector.

In order to further optimize the above technical solution, an embodiment of the present invention further discloses that the first computing unit specifically includes:

A first attention weight calculation subunit, used to calculate a first attention weight based on the first hidden layer state at the previous moment and all time series data belonging to the same deployment unit as the target instance;

A first normalized weight calculation subunit, used to calculate a normalized deployment unit attention weight using a softmax function based on the first attention weight;

A first attention layer output vector calculation subunit, used to calculate a deployment unit attention layer output vector based on the first hidden layer state at the previous moment, all time series data belonging to the same deployment unit as the target instance, and a normalized deployment unit attention weight;

The second computing unit specifically includes:

The static time correlation weight calculation subunit is used to calculate the first-order time correlation coefficient of each time series data belonging to the same host unit as the target instance relative to the historical target time series, and obtain the static time correlation weights of all time series in the corresponding host unit and the historical target time series;

A second attention weight calculation subunit, used to calculate a second attention weight based on the first hidden layer state at the previous moment and all time series data of the target instance belonging to the same host unit;

A second normalization weight subunit, used to obtain the host unit attention weight based on the static time correlation weight and the second attention weight, and perform normalization to obtain a normalized host unit attention weight;

The second attention layer output vector calculation subunit is used to calculate the host unit attention layer output vector based on the first hidden layer state at the previous moment, all time series data belonging to the same host unit as the target instance, the target time series at the historical moment, and the normalized host unit attention weight;

The third calculation unit specifically includes:

The autocorrelation weight calculation subunit is used to calculate the correlation coefficient between the historical moment target time series in different time windows and obtain the corresponding autocorrelation weight;

A third attention weight calculation subunit, used to calculate a third attention weight based on the state of the first hidden layer at the previous moment and the target time series at different historical moments;

A third normalization weight subunit, used to obtain an autocorrelation unit attention weight based on the autocorrelation weight and the third attention weight, and perform normalization to obtain a normalized autocorrelation unit attention weight;

The third attention layer output vector is a quantum unit, which calculates the autocorrelation attention layer output vector based on the state of the first hidden layer at the previous moment, the target time series at the historical moment, and the normalized autocorrelation unit attention weight.

In order to further optimize the above technical solution, the embodiment of the present invention further discloses that the context vector calculation module specifically includes:

A fourth normalization weight subunit is used to calculate the temporal attention layer weight based on the state of the second hidden layer at the previous moment, and perform normalization to obtain a normalized temporal attention layer weight;

The context vector calculation subunit is used to calculate the context vector based on the current first hidden layer state and the normalized temporal attention layer weight.

The prediction method adopted by the present invention uses an attention mechanism to mine the correlation of multiple time series with correlations, and uses weights to represent the correlation of these time series to the target time series, and then uses these correlations to predict the target time series, effectively improving the prediction accuracy. Applying the model provided by the present invention to cluster resource prediction can more accurately predict the future resource requirements of the cluster, can more effectively assist resource planning, thereby improving the resource utilization of the cluster, and more effectively reducing the operation and maintenance costs of the data center.

The technical solution provided by the present invention is further described in detail below in conjunction with a specific implementation method.

In a modern cluster, an application is generally composed of multiple application instances. These multiple application instances belonging to one application are classified as a deployment unit. These instances are generally distributed on different physical hosts, so there will be multiple different application instances on each physical host. If the application instances existing on a physical host are classified as a host unit, then there is a high probability that there is a correlation between a deployment unit and the application instances in a host unit. Therefore, for a target instance to be predicted, when collecting the time series data of this target instance, the time series data of other application instances in the deployment unit where the target instance is located and the time series data of other application instances in the host unit where the target instance is located can also be collected at the same time, and finally all these time series data are used for the prediction of the target instance.

Before introducing the method provided by the present invention in detail, it is stated that the mathematical symbols used to describe the input and output of the model in the present invention are as shown in the following table:

Table 1

First, as shown in Figure 5, a time series data collection architecture is designed: a local time series data database is deployed on each host, and the local time series databases of all hosts can upload data to the global time series database. Then for a target instance to be used for resource prediction, it only needs to obtain its own time data (target sequence) and all time series data ^Xi that belong to the same host unit as the target instance locally, and then query and obtain all time series data ^Xo that belong to the same deployment unit as the target instance from the global time series database.

Based on these data, the present invention designs a prediction model based on the attention mechanism and names it MLA-LSTM (Multi-level Attention LSTM, a long short-term memory network with multi-layer attention).

其中，F为需要训练的模型。The model sets a time window of size T. Each time series uses T values in this window and then predicts the value of the target instance at the next time point, that is, the value at time point T+1. This process can be abstracted as:

Among them, F is the model that needs to be trained.

The model includes two LSTMs: the first LSTM is used as an encoder to process multiple input time series and output the hidden state h _t ; the second LSTM is used as a decoder to process the hidden state h _t output by the first LSTM and finally output the predicted value. The schematic diagram of this model is shown in Figure 6.

1. LSTM Encoder

The calculation process of the LSTM encoder is defined as:

为LSTM的输入，这个输入由三个注意力层的计算获得，下面将会详细介绍三个注意力层的计算过程。将LSTM编码器按时间维度展开，如图6所示。Among them, h _t is the hidden state vector of LSTM at time point t, and its length is set to m.

is the input of LSTM, which is obtained by the calculation of three attention layers. The calculation process of the three attention layers will be described in detail below. The LSTM encoder is expanded along the time dimension, as shown in Figure 6.

For the LSTM encoder, three attention layers are merged into one input attention layer to mine the correlation between time series. The three attention layers are:

(1) A common attention mechanism is used to mine the correlation of multiple time series ^Xi within a deployment unit, and it is called the deployment unit attention layer.

(2) We use an improved attention mechanism to mine the correlations of multiple time series ^Xo within a host unit and call it the host unit attention layer.

(3) An improved attention mechanism is used to mine the autocorrelation of the time series of the target instance, and it is called the autocorrelation attention layer.

1. The calculation formula for the deployment unit attention layer is as follows:

The parameters are described as follows:

In summary, the input and output of the deployment unit attention layer are:

2. The calculation formula of the host unit attention layer is as follows:

(1) First, we need to calculate the first order temporal correlation coefficient (CORT) of each time series relative to the target time series.

Taking the lth time series x ^o,l in the host unit as an example, to calculate the first-order time series correlation coefficient between it and the target time series Y _T , some trimming processing needs to be done on the two series.

First, remove the last value of x ^o,l and get

Remove the first value of the target sequence's lag value Y _T at time T, and we get

与

的CORT绝对值C^o，l：Then calculate

and

The absolute value of CORT ^Co,l :

This absolute value is taken as the static time correlation weight of the time series x ^o,l and the target sequence in time. The calculation method of CORT is:

Among them, S ₁ and S ₂ are two time series with length q, S _1,t and S _2,t are the values of S ₁ and S ₂ at time t respectively.

Finally, the static time correlation weights of all timings in the host unit and the target timing can be obtained and combined into a vector C _out :

(2) Use the common attention mechanism to calculate the attention weights.

Still taking the lth time series xo ^,l as an example, calculate the attention weight of this time series at time t:

Then the attention weight of the entire host unit's time series at time t can form an attention weight vector g _t at time t:

(3) Combine the time-correlation weight vector C _out and the attention weight vector g _t obtained in the above two steps. The combination is completed through a linear transformation, and the new weight vector θ _t obtained after the transformation is:

In order to normalize all elements of this vector, a softmax function is used to map and obtain the normalized weight value of the lth time series in the host unit at time t

(4) Obtain the value vector of the weighted host unit time series at time t:

这个权重与对应的第l个主机单元时间序列在t时刻的值相乘获得加权后的值。所有主机单元的时间序列在t时刻加权值可以组成一个向量

The normalized weights obtained in the previous step

This weight is multiplied by the value of the corresponding lth host unit time series at time t to obtain the weighted value. The weighted values of the time series of all host units at time t can form a vector

In summary, the input and output of the host unit attention layer are:

3. The calculation method of the autocorrelation attention layer is as follows:

(1) Similar to the host unit attention layer, the correlation coefficient between the target time series in different time windows is calculated first. First, the CORT coefficient C ^a,r between the target time series Y _r ending at time r and the target time series Y _T ending at time T needs to be calculated:

C ^{a, T} =||CORT(Y _T , Y _r )||

Then within the time window T, the CORT coefficient of the corresponding target time series at each moment can form an autocorrelation vector C _auto of length T:

(2) Use the common attention mechanism to calculate the attention weights.

(3) Combine the time-correlation weight vector C _auto and the attention weight vector μ _t obtained in the above two steps. Use linear transformation to convert the two weight vectors into a weight vector φ _t :

(4) Obtain the weighted target time series vector:

描述了一个时间窗口T内，目标时间序列的时刻r的值对时刻T的值的影响程度，也就是目标时间序列自己与自己在不同时刻的相关性。用权重

对Y_t内r时刻的值进行加权，获得一个t时刻的输出向量

The normalized weights obtained in the previous step

Describes the degree of influence of the value of the target time series at time r on the value at time T within a time window T, that is, the correlation between the target time series and itself at different times.

Weight the value of Y _t at time r to obtain an output vector at time t

Finally, the output vectors of the three attention layers are combined as the input vector of the encoder LSTM at time t, where

2. Decoder LSTM

Define the decoder LSTM as:

Let _h′t be the hidden state vector of the decoder LSTM at time t, and let the number of vector elements be n. It should be noted that it is different from the hidden state vector _ht of the encoder LSTM. We expand this decoder LSTM along the time dimension, as shown in Figure 6.

Integrate a time-dependent attention layer into this LSTM. The weight calculation method of this time-dependent attention layer is as follows:

可以与h_p加权求和，获得一个上下文向量

The normalized weight at time t obtained by the above method

It can be weighted and summed with h _p to obtain a context vector

The context vector c _t at time t is merged with the target time series value y _t and a linear transformation is performed to obtain the decoder input at time t

会被输入到解码器LSTM当中进行运算。也就是它和当前时刻t的隐藏状态向量h′_t同时被用于更新下一时刻t+1的解码器LSTM隐藏状态h′_t+1：As mentioned earlier,

It will be input into the decoder LSTM for calculation. That is, it and the hidden state vector h′ _t at the current time t are used to update the decoder LSTM hidden state h′ _t+1 at the next time t+1:

The above updating process is repeated until the end of time T, and the hidden state vector h′ _T and cell state vector c _T at time T are obtained.

The calculation method of the predicted value at time T+1 output by the final decoder LSTM is as follows:

In summary, the input and output of the temporal correlation attention layer are summarized as follows:

Finally, MSE (mean square error) is used as the training criterion of the model:

The gradient descent algorithm is used to train this model and determine the specific values of the weight coefficient matrix/vector/bias of the above neural network.

Next, the technical solution of the present invention is further described in conjunction with specific examples.

This example uses cluster-trace, a cluster data released by Alibaba in 2018. It randomly selects one of the containers (id is c_66550) as a target instance and uses its CPU utilization time series data as the resource time series data of the target instance. The instances of the target instance belonging to the same deployment unit and the same host unit are found, and the time series data of these instances are extracted. Finally, they are processed into time series data with an interval of 300 seconds.

Finally, we obtained 33 time series from the same deployment unit, 23 from the same host, and 1 from the target instance. We aligned these time series by time and divided them into three data sets: training set, validation set, and test set. The training set has 10,141 time points, the validation set has 563 time points, and the test set has 564 time points. Each data set has the same number of time series.

The model has multiple hyperparameters. The window size is set to T = {25, 35, 45, 60}, the hidden state vector and cell state vector size of the hidden layer of the encoder and decoder LSTM are m = n = {32, 64, 128}, MSE and MAE (mean absolute error) are used as error criteria respectively, and the batch stochastic gradient descent algorithm is used to optimize the training model, with a learning rate of 5e-4.

Finally, the model is trained using the grid search method, and the hyperparameters that achieve the best results on the validation set are taken as the optimal parameters of the model. Then predictions are made on the test set, and finally the MSE is used to accumulate the errors in the test set.

In the experiment, in order to distinguish the LSTMs that use single sequence and multiple sequences for prediction, the one that uses a single sequence is denoted as LSTM-Un, and the one that uses multiple sequences is denoted as LSTM-Mul.

The experimental results are shown in the following table:

The experimental results show that the error of the model proposed in the present invention is much smaller than that of the three single-sequence models and the two multi-sequence models: under MSE, it is 98.26% better than the best VAR; under MAE, it is 74.40% better than the best VAR, with high prediction accuracy, thus proving the effectiveness of the multi-time series prediction model based on the attention mechanism proposed in the present invention.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims (6)

1. A cluster resource prediction method based on an attention mechanism, characterized by comprising: S1: Take the state of the first hidden layer at the previous moment, all time series data belonging to the same deployment unit as the target instance, all time series data belonging to the same host unit as the target instance, and the target time series at the historical moment as the input of the input attention layer to obtain the first input vector; Step S1 specifically includes: S11: taking the state of the first hidden layer at the previous moment and all time series data belonging to the same deployment unit as the target instance as the input of the deployment unit attention layer, and obtaining the deployment unit attention layer output vector; S12: taking the state of the first hidden layer at the previous moment and all time series data belonging to the same host unit as the target instance as the input of the host unit attention layer, and obtaining the host unit attention output vector; S13: Taking the state of the first hidden layer at the previous moment and the target time sequence at the historical moment as the input of the autocorrelation attention layer, and obtaining the output vector of the autocorrelation attention layer; S14: merging the deployment unit attention layer output vector, the host unit attention layer output vector and the autocorrelation attention layer output vector as the first input vector; S2: Input the first input vector to the LSTM encoder to obtain the current first hidden layer state; S3: Input the current first hidden layer state and the second hidden layer state at the previous moment into the time-dependent attention layer to obtain a context vector; S4: Input the context vector, the second hidden layer state at the previous moment, and the target time sequence at the historical moment into the LSTM decoder to obtain the current second hidden layer state; S5: Perform a linear transformation on the current second hidden layer state and the context vector to obtain a predicted value. 2. According to the cluster resource prediction method based on the attention mechanism of claim 1, it is characterized in that step S11 specifically comprises: Calculate the first attention weight based on the first hidden layer state at the previous moment and all time series data belonging to the same deployment unit as the target instance; Based on the first attention weight, the normalized deployment unit attention weight is calculated using the softmax function; Calculate the deployment unit attention layer output vector based on the state of the first hidden layer at the previous moment, all time series data belonging to the same deployment unit as the target instance, and the normalized deployment unit attention weight; Step S12 specifically includes: Calculate the first-order time series correlation coefficient of each time series data belonging to the same host unit as the target instance relative to the historical target time series, and obtain the static time correlation weights of all time series in the corresponding host unit and the historical target time series; Calculate the second attention weight based on all time series data of the first hidden layer state and the target instance belonging to the same host unit at the previous moment; Obtaining the host unit attention weight based on the static time correlation weight and the second attention weight, and normalizing them to obtain the normalized host unit attention weight; Calculate the host unit attention output vector based on the state of the first hidden layer at the previous moment, all time series data belonging to the same host unit as the target instance, the target time series at the historical moment, and the normalized host unit attention weight; Step S13 specifically includes: Calculate the correlation coefficient between the historical moment target time series in different time windows and obtain the corresponding autocorrelation weights; Calculate the third attention weight based on the state of the first hidden layer at the previous moment and the target time sequence at different historical moments; The autocorrelation unit attention weight is obtained based on the autocorrelation weight and the third attention weight, and normalized to obtain the normalized autocorrelation unit attention weight; Based on the state of the first hidden layer at the previous moment, the target time series at the historical moment, and the normalized autocorrelation unit attention weight, the autocorrelation attention layer output vector is calculated. 3. The cluster resource prediction method based on the attention mechanism according to any one of claims 1 to 2, characterized in that step S3 specifically comprises: Calculate the temporal attention layer weight based on the state of the second hidden layer at the previous moment, and normalize it to obtain the normalized temporal attention layer weight; The context vector is calculated based on the current first hidden layer state and the normalized temporal attention layer weights. 4. A cluster resource prediction device based on an attention mechanism, characterized by comprising: A first input vector calculation module is used to use the first hidden layer state at the previous moment, all time series data belonging to the same deployment unit as the target instance, all time series data belonging to the same host unit as the target instance, and the target time series at the historical moment as inputs to the input attention layer to obtain a first input vector; The first input vector calculation module specifically includes: A first computing unit is used to use the state of the first hidden layer at the previous moment and all time series data belonging to the same deployment unit as the target instance as inputs of the deployment unit attention layer to obtain an output vector of the deployment unit attention layer; The second computing unit is used to use the state of the first hidden layer at the previous moment and all time series data belonging to the same host unit as the target instance as the input of the host unit attention layer to obtain the host unit attention output vector; A third calculation unit is used to use the state of the first hidden layer at the previous moment and the target time sequence at the historical moment as the input of the autocorrelation attention layer to obtain an output vector of the autocorrelation attention layer; A merging unit, used to merge the deployment unit attention layer output vector, the host unit attention output vector and the autocorrelation attention layer output vector as a first input vector; A first hidden layer state calculation module, used for inputting the first input vector into the LSTM encoder to obtain a current first hidden layer state; A context vector calculation module, used to input the current first hidden layer state and the second hidden layer state at the previous moment into the time correlation attention layer to obtain a context vector; A second hidden layer state calculation module is used to input the context vector, the second hidden layer state at the previous moment and the target time sequence at the historical moment into the LSTM decoder to obtain the current second hidden layer state; The linear transformation module is used to linearly transform the current second hidden layer state and the context vector to obtain a predicted value. 5. The cluster resource prediction device based on the attention mechanism according to claim 4, characterized in that the first computing unit specifically comprises: A first attention weight calculation subunit, used to calculate a first attention weight based on the first hidden layer state at the previous moment and all time series data belonging to the same deployment unit as the target instance; A first normalized weight calculation subunit, used to calculate a normalized deployment unit attention weight using a softmax function based on the first attention weight; A first attention layer output vector calculation subunit, used to calculate a deployment unit attention layer output vector based on the first hidden layer state at the previous moment, all time series data belonging to the same deployment unit as the target instance, and a normalized deployment unit attention weight; The second calculation unit specifically includes: The static time correlation weight calculation subunit is used to calculate the first-order time correlation coefficient of each time series data belonging to the same host unit as the target instance relative to the historical target time series, and obtain the static time correlation weights of all time series in the corresponding host unit and the historical target time series; A second attention weight calculation subunit, used to calculate a second attention weight based on the first hidden layer state at the previous moment and all time series data of the target instance belonging to the same host unit; A second normalization weight subunit, used to obtain the host unit attention weight based on the static time correlation weight and the second attention weight, and perform normalization to obtain a normalized host unit attention weight; The second attention layer output vector calculation subunit is used to calculate the host unit attention layer output vector based on the first hidden layer state at the previous moment, all time series data belonging to the same host unit as the target instance, the target time series at the historical moment, and the normalized host unit attention weight; The third calculation unit specifically includes: The autocorrelation weight calculation subunit is used to calculate the correlation coefficient between the historical moment target time series in different time windows and obtain the corresponding autocorrelation weight; A third attention weight calculation subunit, used to calculate a third attention weight based on the state of the first hidden layer at the previous moment and the target time series at different historical moments; A third normalization weight subunit, used to obtain an autocorrelation unit attention weight based on the autocorrelation weight and the third attention weight, and perform normalization to obtain a normalized autocorrelation unit attention weight; The third attention layer output vector is a quantum unit, which calculates the autocorrelation attention layer output vector based on the state of the first hidden layer at the previous moment, the target time series at the historical moment, and the normalized autocorrelation unit attention weight. 6. The cluster resource prediction device based on the attention mechanism according to any one of claims 4 to 5, characterized in that the context vector calculation module specifically comprises: A fourth normalization weight subunit is used to calculate the temporal attention layer weight based on the state of the second hidden layer at the previous moment, and perform normalization to obtain a normalized temporal attention layer weight; The context vector calculation subunit is used to calculate the context vector based on the current first hidden layer state and the normalized temporal attention layer weight.

Images