CN115983438A - Method and device for determining operation strategy of data center terminal air conditioning system - Google Patents

Abstract

The invention discloses a method and a device for determining an operation strategy of a data center tail end air conditioning system, wherein the method comprises the following steps: building a temperature field distribution model of a data center machine room; constructing a Markov decision process model of an operation strategy of an air conditioning system at the tail end of a data center; in the temperature field distribution model, training is carried out by using a reinforcement learning algorithm based on Markov decision process models with different strategy functions and different parameters respectively, so as to generate operation strategies of the air conditioning system at the tail end of various data centers and construct a strategy library; according to a sequence optimization method, evaluating the performance of each operation strategy in a strategy library in a temperature field distribution model, and determining a selection set from the strategy library; and respectively applying each operation strategy in the selected set to the real operation environment of the data center machine room, and determining the optimal operation strategy in the selected set. The method and the device can accurately determine the optimal operation strategy of the air conditioning system at the tail end of the data center.

Description

Method and device for determining operation strategy of terminal air conditioning system in data center

technical field

The invention relates to the technical field of energy-saving optimization of Internet data centers, in particular to a method and device for determining an operation strategy of an air-conditioning system at a data center terminal.

Background technique

This section is intended to provide a background or context to embodiments of the invention that are recited in the claims. The descriptions herein are not admitted to be prior art by inclusion in this section.

In addition to server IT load power consumption, the largest energy consumption in the data center is the cooling infrastructure, about 1/3 to 1/2 of the total power consumption of the data center is used for the cooling system, the growing energy consumption of the data center requires better thermal management to improve energy efficiency. The energy consumption of the refrigeration system of the data center includes the energy consumption of the chiller side and the energy consumption of the terminal air conditioner. The energy consumption optimization of the chiller side has relatively mature technical means (such as the method of optimizing the energy consumption of the chiller based on load forecasting), but the energy consumption of the terminal air conditioner Optimization involves the internal temperature field distribution of the data center computer room, and the simulation of the temperature field distribution inside the computer room involves complex fluid mechanics and thermodynamic analysis, and the temperature field distribution generally changes with time. Therefore, under the premise of ensuring the thermal safety of server IT equipment, Minimizing the operating power consumption of the terminal air conditioning system in the data center is a key challenge and technical difficulty.

However, there is currently a lack of a selection scheme for the terminal air conditioner operation strategy of the data center.

Contents of the invention

An embodiment of the present invention provides a method for determining an operation strategy of a data center terminal air-conditioning system to accurately determine an optimal operation strategy of a data center terminal air-conditioning system. The method includes:

Build the temperature field distribution model of the data center computer room;

Construct a Markov decision process model for the operation strategy of the terminal air conditioning system in the data center;

In the temperature field distribution model, use the reinforcement learning algorithm to train based on different policy functions and Markov decision process models with different parameters, generate a variety of operating strategies for the terminal air-conditioning system of the data center, and build a strategy library;

According to the sequential optimization method, the performance of each operation strategy in the strategy library is evaluated in the temperature field distribution model, and the selection set is determined from the strategy library;

Each operation strategy in the selection set is applied to the real operation environment of the data center computer room, and the optimal operation strategy in the selection set is determined.

The embodiment of the present invention also provides a data center terminal air conditioning system operation strategy determination device, which is used to accurately determine the optimal operation strategy of the data center terminal air conditioning system. The device includes:

The temperature field distribution model building module is used to build the temperature field distribution model of the data center computer room;

The Markov decision process model building block is used to construct the Markov decision process model of the terminal air conditioning system operation strategy of the data center;

The strategy library construction module is used to use the reinforcement learning algorithm in the temperature field distribution model to conduct training based on different strategy functions and Markov decision process models with different parameters, and generate a variety of operating strategies for the terminal air conditioning system of the data center. Build a strategy library;

The selection set determination module is used to evaluate the performance of each operation strategy in the strategy library in the temperature field distribution model according to the order optimization method, and determine the selection set from the strategy library;

The optimal operation strategy determination module is used to apply each operation strategy in the selection set to the real operation environment of the data center computer room, and determine the optimal operation strategy in the selection set.

An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, when the processor executes the computer program, the above data center terminal air conditioning system can be operated Policy determination method.

An embodiment of the present invention also provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned method for determining the operation strategy of the data center terminal air-conditioning system is implemented.

An embodiment of the present invention also provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned method for determining the operation strategy of the data center terminal air-conditioning system is implemented.

In the embodiment of the present invention, the temperature field distribution model of the data center computer room is built; the Markov decision process model of the operation strategy of the data center terminal air-conditioning system is built; in the temperature field distribution model, reinforcement learning algorithms are used, respectively based on different strategy functions , different parameters of the Markov decision process model to train, generate a variety of operating strategies for the terminal air conditioning system of the data center, and build a strategy library; according to the order optimization method, in the temperature field distribution model, the performance of each operating strategy in the strategy library Carry out evaluation, and determine the selection set from the strategy library; apply each operation strategy in the selection set to the real operating environment of the data center computer room, and determine the optimal operation strategy in the selection set. In the above process, the Markov decision process model based on different parameters and the policy function form a strategy library containing multiple operating strategies. Compared with the traditional two-stage method and reinforcement learning method, which only generate a single operating strategy, this scheme comprehensively Multiple different forms of policy functions are considered, and multiple running strategies are reasonably selected, so the final running strategy is more guaranteed than the single running strategy obtained by the traditional method, that is, it can better guarantee the running strategy in the actual data center environment. It not only ensures the thermal safety of server IT equipment, but also minimizes the energy consumption of terminal air conditioners. In the process of selecting the operation strategy, different from the traditional method of calculating the real performance of all operation strategies and selecting the best strategy after sorting, this scheme adopts the order optimization method to obtain the selection set, which greatly reduces the number of strategies in the strategy library. The number of evaluations of the operation strategy in the real operation environment further ensures the thermal safety of the server IT equipment in the data center and saves manpower, material and financial resources.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work. In the attached picture:

FIG. 1 is a flow chart of a method for determining an operation strategy of a data center terminal air-conditioning system in an embodiment of the present invention;

Fig. 2 is the flowchart of building the temperature field distribution model of data center machine room in the embodiment of the present invention;

Fig. 3 is the flow chart of determining selection set in the embodiment of the present invention;

4 is a schematic diagram of a device for determining an operation strategy of a data center terminal air-conditioning system in an embodiment of the present invention;

Fig. 5 is a schematic diagram of a computer device in an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments and descriptions of the present invention are used to explain the present invention, but not to limit the present invention.

The inventors found that most of the traditional methods for solving the energy-saving problem of strategic optimization of the data center refrigeration system, especially the terminal air-conditioning system, are based on a two-stage framework. In the first stage, an approximate system model is established through a mechanism analysis method or a data-driven method. This model usually includes fluid dynamics, heat transfer and mechanical principles, and needs to take into account the distribution of the temperature field in the data center computer room. In the second stage, using the approximate system model, the optimal decision sequence of the controllable variables of the air conditioner is obtained through the strategy optimization algorithm. Common strategy optimization algorithms mainly include dynamic programming, model predictive control algorithms, and so on. However, these traditional optimization methods based on the two-stage framework need to establish an approximate model of the temperature field of the data center computer room first, and the distribution of the temperature field involves professional knowledge such as fluid dynamics and heat transfer. If the method of mechanism analysis is used to model the It is necessary to establish complex partial differential equations. For large-scale data centers that have been expanding in size in recent years, the process of establishing the mechanism model of the temperature field in the computer room is complex, difficult, and error-prone. Therefore, these traditional model-based optimization algorithms are difficult to solve. Strategy optimization problem for terminal air conditioning system in data center.

The reinforcement learning method continuously learns to obtain the optimal operation strategy in the process of interacting with the environment, and does not require the dynamic characteristics of the system to be known (especially the model-free reinforcement learning method). Because the mechanism model of the temperature field distribution in the data center computer room is very complex, it may be an effective way to solve the strategy optimization problem of the terminal air conditioning system in the data center by using the reinforcement learning method. At present, some literatures have used this type of method to solve the data center cooling problem. System strategy optimization problem. Generally, in order to prevent losses caused by overheating of server IT equipment, reinforcement learning algorithms usually cannot be directly trained in a real data center environment, so it is still necessary to use Computational Fluid Dynamics (CFD) simulation software to establish data center terminal air conditioners and computer rooms Simulation model of temperature field. Although the reinforcement learning method can effectively avoid the mechanism modeling and analysis of the temperature field in the computer room, the existing mainstream reinforcement learning methods have the disadvantages of low sample utilization, unstable strategy training process, and the performance of the training strategy is greatly affected by parameters. As a result, the performance of the operation strategy of the data center terminal air conditioning system obtained through the final training cannot be guaranteed, and because the training is carried out in a simulation environment, and the simulation environment will inevitably be different from the real environment, the strategy obtained by training is in the real environment. performance is also not guaranteed.

Based on this, an embodiment of the present invention proposes a method for selecting an operation strategy of a data center air-conditioning system based on strategy library and sequence optimization.

Fig. 1 is a flowchart of a method for determining an operation strategy of a data center terminal air-conditioning system in an embodiment of the present invention, including:

Step 101, building the temperature field distribution model of the computer room of the data center;

Step 102, constructing a Markov decision process model of the operation strategy of the terminal air conditioning system of the data center;

Step 103, in the temperature field distribution model, use a reinforcement learning algorithm to train based on different policy functions and Markov decision process models with different parameters, generate various operating strategies for the terminal air-conditioning system of the data center, and build a strategy library;

Step 104, according to the sequential optimization method, evaluate the performance of each operating strategy in the strategy library in the temperature field distribution model, and determine the selection set from the strategy library;

Step 105, apply each operation strategy in the selection set to the actual operation environment of the data center computer room, and determine the optimal operation strategy in the selection set.

Wherein, in step 101, a temperature field distribution model of a data center computer room is built; FIG. 2 is a flow chart of building a temperature field distribution model of a data center computer room in an embodiment of the present invention, including:

Step 201, using CFD software, based on the computer room layout CAD drawings, using CFD simulation software to model and simulate the data center computer room space structure and air conditioner and IT equipment models, and establish a temperature field distribution model of the data center computer room;

The temperature field of the data center computer room is affected by boundary conditions such as the server IT load and the speed of the terminal air-conditioning fan, and changes with time and space distribution. The simple temperature distribution model established by traditional fluid dynamics and heat transfer mechanism analysis methods is difficult to accurately describe the temperature change of each measuring point in the computer room with time and space distribution, and it is difficult to capture the local hot spot next to a server IT equipment in time, making the computer room IT equipment has the potential to overheat. Therefore, the embodiment of the present invention adopts the CFD simulation software for the data center to simulate the temperature field distribution of the data center computer room, according to the computer room layout CAD drawings, and utilizes the rich original library of the CFD simulation software (air conditioner originals, IT equipment originals, etc.) Space structure (including server IT equipment space layout, hot and cold aisle space layout, air conditioning space layout, etc., temperature sensor space layout, air conditioning system structure, etc.) The distribution model can more accurately describe the change of the temperature of each measuring point in the computer room with time and space distribution.

Although the above computer room temperature field distribution model is modeled based on the computer room layout CAD drawings, there is still a certain difference between the actual temperature field and the actual temperature field in the actual computer room. Therefore, the temperature field distribution model needs to be adjusted more carefully.

Step 202, collecting real operating environment data in the computer room;

The real operating environment data includes historical data such as temperature measurement points in the actual computer room, return air temperature set point, air-conditioning fan speed, and environmental conditions.

Step 203, comparing the collected real operating environment data with the operating environment data simulated by using the temperature field distribution model, and continuously adjusting the temperature field distribution model, so that the operating environment data of the adjusted temperature field distribution model is consistent with the actual operating environment data. The matching degree of environmental data reaches the preset threshold.

Continuously adjust the temperature field distribution model, that is, fine-tune the air conditioner and server IT equipment space position, operation setting parameters, etc. The preset threshold value is determined by the user according to the actual needs to achieve a high degree of matching.

In step 102, the Markov decision process model of the data center terminal air conditioning system operation strategy is constructed; in one embodiment, the Markov decision process model is composed of state space S, action space A, state transition function P, reward function Composed of R and discount factor γ; can be expressed as a five-tuple

The state of the state space S is selected from the observed variables;

The actions in the action space A are selected from the control variables;

The reward function R is obtained according to the energy consumption penalty of the air conditioner and the overtemperature penalty of the server IT equipment;

The state transition function P is obtained according to the temperature field distribution model;

At each time t, learn and select an action A _t according to the state S _t observed by the environment at time t, and the environment responds to the action A _t accordingly, presenting a new state S _t+1 and generating a reward R _t+1 , take the reward as the long-term maximization target in the action selection process.

In the above example, the states S _t+1 and R _t+1 only depend on P and A _t , and have nothing to do with the earlier states and actions, which are the basic characteristics of states and rewards in the Markov decision process model (Markovian).

参考上述控制变量选取系统动作

结合空调的能耗惩罚和服务器IT设备的超温惩罚设计奖励函数

温度场分布模型提供状态转移函数

选取合适的折扣因子γ∈(0，1)，构建末端空调系统马尔可夫决策过程模型

在该马尔可夫决策过程模型的基础上，应用强化学习算法，在仿真环境中训练得到最优末端空调系统运行策略。In the actual operating environment of a typical data center terminal air-conditioning system, the observed variables generally include: temperature measurement points located in the cold/hot aisle and air-conditioning air supply/return air outlet, IT load rate of servers in each cabinet, outdoor temperature and light intensity ; Control variables generally include: air conditioner supply/return air temperature set point, air conditioner fan speed, etc.; at each time t, control variables generally affect the cold/hot aisle and air conditioner supply/return at the next time (t+1 time) Observable variables such as the measured values of the temperature measuring points of the tuyere, and the observable variables such as the server IT load rate, outdoor temperature, and light intensity in each cabinet are not affected by the control variables. predict. Generally, the system state can be selected with reference to the above observed variables

Refer to the above control variables to select system actions

Combining the energy consumption penalty of the air conditioner and the overtemperature penalty of the server IT equipment to design a reward function

The temperature field distribution model provides the state transition function

Select the appropriate discount factor γ∈(0, 1) to construct the Markov decision process model of the terminal air-conditioning system

On the basis of the Markov decision process model, the reinforcement learning algorithm is applied to train the optimal terminal air-conditioning system operation strategy in the simulation environment.

The operation strategy of the terminal air-conditioning system in the data center is optimized to minimize the energy consumption of the terminal air-conditioning while ensuring the thermal safety of the IT equipment in the computer room. Since the temperature field (controlled variable) in the computer room changes with time, it is affected by the return air temperature set point of the terminal air conditioner at each moment, the fan speed and other parameters (controlled variable) and the IT load rate of the server, so it is necessary to combine each moment The measured value of each temperature measuring point in the temperature field and the IT load rate should reasonably adjust the value of the control variable at this moment, which essentially forms a sequential decision-making problem. For this kind of sequential decision-making problem, it is generally necessary to clarify the state-action space, establish a Markov decision process model, and then use reinforcement learning and other methods to train the policy function.

Since the temperature measurement points of the hot and cold aisles in the air-conditioning system environment of the data center terminal are continuous, the values of the return air temperature set point and fan speed are also continuous, and the temperature field changes complexly with time and space, so the sequence The decision problem is a rather difficult problem. The existing function approximation reinforcement learning method can train and obtain a certain running strategy, but it is difficult to guarantee the optimality of the strategy. Therefore, the reinforcement learning algorithm can be used to generate a variety of data center terminal air-conditioning system operating strategies in the simulation environment, build a strategy library Π, and reasonably select the optimal strategy in the strategy library.

对于该集合中的各马尔可夫决策过程模型分别应用强化学习算法训练得到最优策略。In the optimization problem of the operation strategy of the terminal air-conditioning system in the data center, it is often possible to construct a Markov decision process model with different parameters, such as the selection of different states S, different actions A, different designed reward functions R, and discount factors γ Selecting different Markov decision process models will correspond to different parameters. For Markov decision process models with different parameters, the optimal strategies obtained by applying reinforcement learning training are different, and in the problems of the embodiments of the present invention, often only the actual application of the operating strategies obtained in the temperature field distribution model can be implemented. Objectively evaluate the performance of this operation strategy, and then judge what kind of Markov decision process model to use to train the optimal operation strategy with the best performance. Therefore, the Markov decision process model set of terminal air-conditioning system can be constructed by selecting different states S, different actions A, different designed reward functions R, and different selections of discount factor γ

For each Markov decision process model in the set, the reinforcement learning algorithm is used to train to obtain the optimal strategy.

In step 103, in the temperature field distribution model, use a reinforcement learning algorithm to train based on different policy functions and Markov decision process models with different parameters, generate a variety of operating strategies for the terminal air conditioning system of the data center, and build a strategy library ;

In one embodiment, step 103 includes:

Determine the various strategy functions used;

Determine the action value function of the operation strategy, the action value function represents the cumulative discount reward weighted to the reward R for taking action a under the state s under the situation of using the operation strategy, which can be expressed as follows:

For each strategy function, in the temperature field distribution model, under the framework of the reinforcement learning algorithm, in the process of constantly updating the action value function and the strategy function, it converges to the optimal operation strategy, and the optimal operation strategy is used as a The running policy is added to the policy library.

In an embodiment, the policy function includes a neural network type policy function and a basis function linear weighted type policy function.

Since the neural network has good representation ability and generalization properties, the neural network is often used to fit the running strategy:

π(s)=f ^NN (s, θ)

Among them, π(s) is the operation strategy, f ^NN (s, θ) represents the neural network strategy function, the input of the neural network is the state s, the output is the action a, and θ represents the neural network and training parameters. Multiple Markov decision process models with different parameters can use DDPG, TD3, SAC and other algorithms to train neural network-type optimal data center terminal air-conditioning system operation strategy functions, and put the strategies into the strategy library Π.

In addition to the neural network-type strategy function, the basis function linear weighted strategy function can also be used:

π(s)=f ^basis (s,w)=w ^T φ(s)

为权重向量,

为基函数，基函数的选取可以依据数据中心安全运行的先验知识，结合末端空调系统运行的特点进行设计。同样，基于不同参数的马尔可夫决策过程模型，可以在AC强化学习算法的框架下进行训练，不断更新权重向量，最终训练出基函数线性加权型的最优数据中心末端空调系统运行策略函数，并将策略放置进策略库Π。Among them, π(s) is the operation strategy, f ^basis (s, w) is the linear weighted strategy function of the basis function,

is the weight vector,

The selection of the basis function can be designed based on the prior knowledge of the safe operation of the data center and the characteristics of the operation of the terminal air conditioning system. Similarly, the Markov decision process model based on different parameters can be trained under the framework of the AC reinforcement learning algorithm, the weight vector is continuously updated, and finally the optimal data center terminal air-conditioning system operation strategy function of the linear weighted basis function is trained. And put the strategy into the strategy library Π.

The representation ability of the linear weighting of the basis function may be weaker than that of the neural network, but it only needs to update the weight vector of the linear weighting, which is more interpretable, more stable in the training process, and better in convergence.

In step 104, according to the sequence optimization method, the performance of each operation strategy in the strategy library is evaluated in the temperature field distribution model, and the selection set is determined from the strategy library; the obtained data center terminal air-conditioning operation strategy library Π contains N Operation strategies, these N operation strategies are all generated in the temperature field distribution model, but since there will be unavoidable deviations between the temperature field distribution model and the real operation environment, it is necessary to perform a real-time analysis of the operation strategies in the strategy library Π in the real operation environment The performance of the strategy is evaluated to pick out the optimal strategy π ^* in the strategy library Π:

In one embodiment, the formula for evaluating the performance of each operating strategy in the strategy library in the temperature field distribution model is as follows:

为t时刻数据中心机房第i个服务器IT设备出风口的温度，T_max为机柜出风口允许温度上限，λ为权重参数。Among them, π ^* is the optimal operation strategy, and J(π) is the performance evaluation function of the operation strategy in the operating environment of the data center computer room, indicating the total energy consumption and total overtemperature of the operation strategy within the time T; Δt is t The time interval to t+1, P _t is the operating power of the terminal air conditioner at time t,

is the temperature at the air outlet of the i-th server IT equipment in the data center computer room at time t, T _max is the upper limit of the allowable temperature at the air outlet of the cabinet, and λ is a weight parameter.

In addition to the above formula (1), J(π) can also be expressed as:

Due to the security constraints of the real operating environment, it is not practical to evaluate all the operating policies in the policy library Π in the real operating environment. Only a small part of the policy library Π is more likely to be selected reasonably. The strategy of the optimal strategy is evaluated in the real operating environment, but in the simulation environment of the temperature field distribution model, the performance of the operating strategy can be evaluated without restriction; in actual engineering implementation, it is often not necessary to pursue the above evaluation formula ( 1), but to obtain the "good enough" solution of the above formula (1) can meet the engineering needs, that is, it is not necessary to pursue the operation strategy with the smallest performance J(π) in the strategy library Π, but to finally obtain The real performance J(π) of the running strategy is within the smallest set of g strategies to meet the requirements.

Considering the characteristics of the above problems, the sequential optimization method can be used to select the running strategies in the strategy library Π. Specifically, the strategy performance evaluation function J(π) of the real operating environment is used as a detailed model (Detailed Model). In the temperature field distribution model, the above formula is also used as the strategy performance evaluation function, but because the strategy in the temperature field distribution model There are differences between the performance evaluation and the strategy performance evaluation of the real operating environment, so the performance evaluation obtained in the simulation environment is recorded as J′(π), which is regarded as a rough model (Crude Model); the estimated sequence performance curve OPC category and the rough model The noise level of the given "good enough" set (strategy selection set) G size g and alignment level k according to user preference, the alignment level k is related to the alignment probability. Determine the size s of the selection set S according to the selection set formula of order optimization, evaluate the performance of all operating strategies in the strategy library Π according to the rough model, and select s strategies with the smallest J′(π) to form the selection set S. The optimization theory can ensure that the strategy set S contains at least k strategies with the lowest performance of the real strategy before g with a probability of 95%. It can also be understood that in the selected selection set S, there are k elements (design) in S that are truly "good enough" with a probability greater than or equal to 95%.

In summary, the following steps are formed.

Fig. 3 is the flow chart of determining the selection set in the embodiment of the present invention, in one embodiment, according to the sequential optimization method, in the temperature field distribution model, evaluate the performance of each operation strategy in the strategy library, determine from the strategy library Pick a collection, including:

Step 301, using the strategy performance evaluation function J'(π) obtained in the temperature field distribution model as a rough model;

Step 302, estimating the sequence performance curve category and the noise level of the rough model;

Step 303, obtaining the size g and alignment level k of the policy selection set G determined by the user; the alignment level k is related to the alignment probability.

Step 304: Determine the size s of the selection set S according to the selection set formula of the order optimization. The parameters of the selection set formula of the order optimization include the type of the order performance curve, the noise level of the rough model, the size g of the strategy selection set G, and the alignment Level k; The selection set formula for order optimization is as follows:

Among them, Z ₁ , Z ₂ , Z ₃ , and Z ₄ are determined according to the category of the sequence performance curve and the noise level of the rough model, and are determined based on a large amount of historical data.

Step 305, calculate the rough model values of all the running strategies in the strategy library, and select the running strategies of the size s of the selection set S with the smallest value of the rough model to form the selection set S.

In step 105, each operation strategy in the selection set is applied to the real operation environment of the data center computer room, and the optimal operation strategy in the selection set is determined.

In one embodiment, each operation strategy in the selection set is applied to the real operation environment of the data center computer room, and the optimal operation strategy in the selection set is determined, including:

The strategic performance evaluation function J(π) of the real operating environment of the data center computer room is used as a detailed model;

Apply each operation strategy in the selection set to the real operation environment of the data center computer room, and obtain the value of the detailed model of each operation strategy;

The operation strategy with the minimum value of the detailed model is taken as the final operation strategy of the terminal air conditioning system of the data center.

In summary, the method proposed in the embodiment of the present invention has the following beneficial effects:

First, the Markov decision process model based on different parameters and the policy function form a strategy library containing multiple operating strategies. Compared with the traditional two-stage method and reinforcement learning method that only generates a single operating strategy, this scheme comprehensively considers Multiple policy functions in different forms, and reasonable selection of multiple operating strategies, so the final operating strategy is more guaranteed than the single operating strategy obtained by the traditional method, that is, it can better guarantee that the operating strategy can be used in the actual data center environment. Ensure the thermal safety of server IT equipment and minimize the energy consumption of terminal air conditioners.

Second, in the process of selecting the operation strategy, unlike the traditional method of calculating the real performance of all operation strategies and selecting the best strategy after sorting, this scheme adopts the order optimization method to obtain the selection set, which greatly reduces the strategy cost. The evaluation times of the operating strategies in the library in the real operating environment further ensure the thermal safety of the data center server IT equipment and save manpower, material and financial resources.

The embodiment of the present invention also proposes a device for determining the operation strategy of the terminal air-conditioning system of the data center, the principle of which is similar to the method for determining the operation strategy of the terminal air-conditioning system of the data center, and will not be repeated here.

Fig. 4 is a schematic diagram of a device for determining an operation strategy of a data center terminal air-conditioning system in an embodiment of the present invention, including:

The temperature field distribution model building module 401 is used to build the temperature field distribution model of the data center computer room;

Markov decision process model construction module 402, used to construct the Markov decision process model of the terminal air conditioning system operation strategy of the data center;

The strategy library construction module 403 is used to use the reinforcement learning algorithm in the temperature field distribution model to perform training based on different strategy functions and Markov decision process models with different parameters, so as to generate a variety of operating strategies for the terminal air conditioning system of the data center , build a strategy library;

The selection set determination module 404 is used to evaluate the performance of each operation strategy in the strategy library in the temperature field distribution model according to the order optimization method, and determine the selection set from the strategy library;

The optimal operation strategy determination module 405 is configured to apply each operation strategy in the selection set to the actual operation environment of the data center computer room, and determine the optimal operation strategy in the selection set.

In one embodiment, the temperature field distribution model building module is specifically used for:

Through CFD software, according to the computer room layout CAD drawings, use CFD simulation software to model and simulate the data center computer room space structure and air conditioner and IT equipment models, and establish the temperature field distribution model of the data center computer room;

Collect real operating environment data in the computer room;

Comparing the collected real operating environment data with the operating environment data simulated by the temperature field distribution model, and continuously adjusting the temperature field distribution model, so that the operating environment data of the adjusted temperature field distribution model matches the real operating environment data reached the preset threshold.

In one embodiment, the Markov decision process model is composed of state space S, action space A, state transition function P, reward function R and discount factor γ;

The state of the state space S is selected from the observed variables;

The actions in the action space A are selected from the control variables;

The reward function R is obtained according to the energy consumption penalty of the air conditioner and the overtemperature penalty of the server IT equipment;

The state transition function P is obtained according to the temperature field distribution model;

At each time t, learn and select an action A _t according to the state S _t observed by the environment at time t, and the environment responds to the action A _t accordingly, presenting a new state S _t+1 and generating a reward R _t+1 , take the reward as the long-term maximization target in the action selection process.

In one embodiment, in the temperature field distribution model, a reinforcement learning algorithm is used to perform training based on different policy functions and Markov decision process models with different parameters, so as to generate a variety of operating strategies for the terminal air conditioning system of the data center, and construct Policy library, including:

Determine the various strategy functions used;

Determine the action value function of the operation strategy, where the action value function represents the cumulative discount reward for taking action a under the state s and weighting the reward R under the situation of using the operation strategy;

For each strategy function, in the temperature field distribution model, under the framework of the reinforcement learning algorithm, in the process of constantly updating the action value function and the strategy function, it converges to the optimal operation strategy, and the optimal operation strategy is used as a The running policy is added to the policy library.

In an embodiment, the policy function includes a neural network type policy function and a basis function linear weighted type policy function.

In one embodiment, the formula for evaluating the performance of each operating strategy in the strategy library in the temperature field distribution model is as follows:

为t时刻数据中心机房第i个服务器IT设备出风口的温度，T_max为机柜出风口允许温度上限，λ为权重参数。Among them, π ^* is the optimal operation strategy, and J(π) is the performance evaluation function of the operation strategy in the operating environment of the data center computer room, indicating the total energy consumption and total overtemperature of the operation strategy within the time T; Δt is t The time interval to t+1, P _t is the operating power of the terminal air conditioner at time t,

is the temperature at the air outlet of the i-th server IT equipment in the data center computer room at time t, T _max is the upper limit of the allowable temperature at the air outlet of the cabinet, and λ is a weight parameter.

In one embodiment, the selection set determination module is specifically used for:

The strategic performance evaluation function J′(π) obtained in the temperature field distribution model is used as a rough model;

Estimation of the ordinal performance curve class and the noise level of the coarse model;

Obtain the size g and alignment level k of the policy selection set G determined by the user;

Determine the size s of the selection set S according to the selection set formula of the order optimization, the parameters of the selection set formula of the order optimization include the order performance curve category, the noise level of the rough model, the size g of the strategy selection set G and the alignment level k;

Calculate the rough model values of all running strategies in the strategy library, and select the s running strategies with the smallest value of the rough model to form the selection set S.

In one embodiment, the optimal operation strategy determination module is specifically used for:

The strategic performance evaluation function J(π) of the real operating environment of the data center computer room is used as a detailed model;

Apply each operation strategy in the selection set to the real operation environment of the data center computer room, and obtain the value of the detailed model of each operation strategy;

The operation strategy with the minimum value of the detailed model is taken as the final operation strategy of the terminal air conditioning system of the data center.

In summary, the beneficial effects of the device proposed in the embodiment of the present invention are as follows:

First, the Markov decision process model based on different parameters and the policy function form a strategy library containing multiple operating strategies. Compared with the traditional two-stage method and reinforcement learning method that only generates a single operating strategy, this scheme comprehensively considers Multiple policy functions in different forms, and reasonable selection of multiple operating strategies, so the final operating strategy is more guaranteed than the single operating strategy obtained by the traditional method, that is, it can better guarantee that the operating strategy can be used in the actual data center environment. Ensure the thermal safety of server IT equipment and minimize the energy consumption of terminal air conditioners.

Second, in the process of selecting the operation strategy, unlike the traditional method of calculating the real performance of all operation strategies and selecting the best strategy after sorting, this scheme adopts the order optimization method to obtain the selection set, which greatly reduces the strategy cost. The evaluation times of the operating strategies in the library in the real operating environment further ensure the thermal safety of the data center server IT equipment and save manpower, material and financial resources.

The embodiment of the present invention also provides a computer device. FIG. 5 is a schematic diagram of the computer device in the embodiment of the present invention. The computer device 500 includes a memory 510, a processor 520, and is stored on the memory 510 and can run on the processor 520. A computer program 530, when the processor 520 executes the computer program 530, implements the above-mentioned method for determining the operation strategy of the terminal air-conditioning system of the data center.

An embodiment of the present invention also provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned method for determining the operation strategy of the data center terminal air-conditioning system is implemented.

An embodiment of the present invention also provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned method for determining the operation strategy of the data center terminal air-conditioning system is implemented.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Protection scope, within the spirit and principles of the present invention, any modification, equivalent replacement, improvement, etc., shall be included in the protection scope of the present invention.

Claims (12)

1. A method for determining an operation strategy of a data center terminal air conditioning system is characterized by comprising the following steps:

building a temperature field distribution model of a data center machine room;

constructing a Markov decision process model of an operation strategy of an air conditioning system at the tail end of a data center;

in the temperature field distribution model, training is carried out by using a reinforcement learning algorithm based on Markov decision process models with different strategy functions and different parameters respectively, operation strategies of the air conditioning system at the tail end of various data centers are generated, and a strategy library is constructed;

according to a sequence optimization method, evaluating the performance of each operation strategy in a strategy library in a temperature field distribution model, and determining a selection set from the strategy library;

and respectively applying each operation strategy in the selected set to the real operation environment of the data center machine room, and determining the optimal operation strategy in the selected set.

2. The method of claim 1, wherein building a temperature field distribution model for a data center room comprises:

modeling and simulating the space structure of the data center machine room and the models of the air conditioner and the IT equipment by CFD software according to the CAD drawing of the machine room layout and by utilizing CFD simulation software, and establishing a temperature field distribution model of the data center machine room;

collecting real operating environment data in a machine room;

and comparing the acquired real operating environment data with the operating environment data simulated by the temperature field distribution model, and continuously setting the temperature field distribution model, so that the matching degree of the operating environment data of the set temperature field distribution model and the real operating environment data reaches a preset threshold value.

3. The method of claim 1, wherein the markov decision process model consists of a state space S, an action space a, a state transition function P, a reward function R, and a discount factor γ;

selecting the state of the state space S from the observation variables;

selecting the action in the action space A from the control variables;

the reward function R is obtained according to the energy consumption punishment of the air conditioner and the overtemperature punishment of the server IT equipment;

the state transfer function P is obtained according to the temperature field distribution model;

at each time t, a state S observed in dependence on the environment at time t _t Performing learning and selecting action A _t Environment to action A _t Respond correspondingly and present a new state S _t+1 With the generation of a prize R _t+1 The reward is targeted for long term maximization in the action selection process.

4. The method of claim 3, wherein in the temperature field distribution model, training is performed by using a reinforcement learning algorithm based on Markov decision process models with different strategy functions and different parameters respectively to generate a plurality of operation strategies of the data center end air conditioning system, and a strategy library is constructed, comprising:

determining various adopted strategy functions;

determining an action cost function of the operation strategy, wherein the action cost function represents the accumulated discount reward for weighting the reward R of taking the action a under the condition of using the operation strategy;

for each strategy function, in a temperature field distribution model, under the framework of a reinforcement learning algorithm, the action value function and the strategy function are converged to an optimized operation strategy in the process of continuously and alternately updating, and the optimized operation strategy is added into a strategy library as an operation strategy.

5. The method of claim 3, wherein the policy function comprises a neural network type policy function, a basis function linear weighted type policy function.

6. The method of claim 1, wherein the formula for evaluating the performance of each operating strategy in the strategy library in the temperature field distribution model is as follows:

wherein, pi ^* For the optimal operation strategy, J (pi) is a performance evaluation function of the operation strategy in the operation environment of the data center machine room, and represents the total energy consumption and total overtemperature condition of the operation strategy within time T; Δ t is the time interval t to t +1, P _t For the operation power of the air conditioner at the end of time t,

the temperature of the air outlet of the ith server IT equipment of the data center machine room at the moment T, T _max And the upper limit of the allowable temperature of the air outlet of the cabinet is defined, and lambda is a weight parameter.

7. The method of claim 6, wherein the performance of each operating policy in the policy repository is evaluated in the temperature field distribution model according to an order optimization method, and the determining a selection set from the policy repository comprises:

taking a strategy performance evaluation function J' (pi) obtained in the temperature field distribution model as a rough model;

estimating the category of the sequence performance curve and the noise level of the rough model;

obtaining the size G and the alignment level k of a strategy selection set G determined by a user;

determining the size S of a selection set S according to a selection set formula of sequence optimization, wherein parameters of the selection set formula of sequence optimization comprise sequence performance curve types, noise levels of rough models, the size G of a strategy selection set G and an alignment level k;

and calculating the values of the rough models of all the operation strategies in the strategy library, and selecting S operation strategies with the minimum values of the rough models to form a selection set S.

8. The method of claim 7, wherein the step of applying each operation strategy in the selected set to a real operation environment of the data center machine room respectively to determine an optimal operation strategy in the selected set comprises:

taking a strategic performance evaluation function J (pi) of a real operation environment of a data center machine room as a detailed model;

respectively applying each operation strategy in the selected set to a real operation environment of the data center machine room to obtain a value of a detailed model of each operation strategy;

and taking the operation strategy with the minimum value of the detailed model as the final operation strategy of the data center terminal air conditioning system.

9. An operation strategy determination device for an air conditioning system at the tail end of a data center is characterized by comprising the following steps:

the temperature field distribution model building module is used for building a temperature field distribution model of a data center machine room;

the Markov decision process model building module is used for building a Markov decision process model of the operation strategy of the air conditioning system at the tail end of the data center;

the strategy base building module is used for training a Markov decision process model based on different strategy functions and different parameters respectively by using a reinforcement learning algorithm in the temperature field distribution model, generating operation strategies of the air-conditioning system at the tail end of various data centers and building a strategy base;

the selected set determining module is used for evaluating the performance of each operation strategy in the strategy library in the temperature field distribution model according to the sequence optimization method and determining a selected set from the strategy library;

and the optimal operation strategy determining module is used for respectively applying each operation strategy in the selection set to the real operation environment of the data center machine room and determining the optimal operation strategy in the selection set.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the method of any one of claims 1 to 8.

12. A computer program product, characterized in that the computer program product comprises a computer program which, when being executed by a processor, carries out the method of any one of claims 1 to 8.

Images