CN118519340B - Intelligent heating control method based on deep reinforcement learning and adaptive control - Google Patents
Intelligent heating control method based on deep reinforcement learning and adaptive control Download PDFInfo
- Publication number
- CN118519340B CN118519340B CN202410578917.3A CN202410578917A CN118519340B CN 118519340 B CN118519340 B CN 118519340B CN 202410578917 A CN202410578917 A CN 202410578917A CN 118519340 B CN118519340 B CN 118519340B
- Authority
- CN
- China
- Prior art keywords
- thermal power
- time period
- heat load
- heating
- heat
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000010438 heat treatment Methods 0.000 title claims abstract description 89
- 238000000034 method Methods 0.000 title claims abstract description 59
- 230000002787 reinforcement Effects 0.000 title claims abstract description 48
- 230000003044 adaptive effect Effects 0.000 title claims abstract description 14
- 238000012549 training Methods 0.000 claims abstract description 13
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 12
- 238000007781 pre-processing Methods 0.000 claims abstract description 8
- 230000009471 action Effects 0.000 claims description 59
- 230000033228 biological regulation Effects 0.000 claims description 43
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 29
- 239000000446 fuel Substances 0.000 claims description 23
- 239000013598 vector Substances 0.000 claims description 18
- 238000004364 calculation method Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 claims description 14
- 238000004519 manufacturing process Methods 0.000 claims description 11
- 230000001186 cumulative effect Effects 0.000 claims description 6
- 230000007613 environmental effect Effects 0.000 claims description 6
- 230000010354 integration Effects 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 5
- 230000005855 radiation Effects 0.000 claims description 4
- 238000013450 outlier detection Methods 0.000 claims 1
- 239000008400 supply water Substances 0.000 claims 1
- 238000009826 distribution Methods 0.000 abstract description 6
- 230000020169 heat generation Effects 0.000 description 12
- 230000001276 controlling effect Effects 0.000 description 8
- 230000008859 change Effects 0.000 description 5
- 238000005265 energy consumption Methods 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 5
- 238000005457 optimization Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000002485 combustion reaction Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000017525 heat dissipation Effects 0.000 description 2
- 230000001105 regulatory effect Effects 0.000 description 2
- 101100295776 Drosophila melanogaster onecut gene Proteins 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000004134 energy conservation Methods 0.000 description 1
- 238000003912 environmental pollution Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000035515 penetration Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
- 238000002759 z-score normalization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B13/00—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion
- G05B13/02—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric
- G05B13/04—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators
- G05B13/042—Adaptive control systems, i.e. systems automatically adjusting themselves to have a performance which is optimum according to some preassigned criterion electric involving the use of models or simulators in which a parameter or coefficient is automatically adjusted to optimise the performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/092—Reinforcement learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Economics (AREA)
- Theoretical Computer Science (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Tourism & Hospitality (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Primary Health Care (AREA)
- Development Economics (AREA)
- Biophysics (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- Biomedical Technology (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Educational Administration (AREA)
- Life Sciences & Earth Sciences (AREA)
- Game Theory and Decision Science (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Medical Informatics (AREA)
- Automation & Control Theory (AREA)
- Air Conditioning Control Device (AREA)
- Feedback Control In General (AREA)
- Supply And Distribution Of Alternating Current (AREA)
Abstract
本发明涉及供热管网调控的技术领域，公开了一种基于深度强化学习和自适应控制的智能化供热调控方法，包括以下步骤：S1：收集室外环境数据与每个热力站的热负荷数据，并进行数据预处理；S2：基于所述室外环境数据与热负荷数据，训练深度强化学习模型；S3：基于所述深度强化学习模型制定供热策略；S4：通过自适应控制算法执行所述供热策略。本发明提高了能源利用效率，降低了不必要的能源损耗；增强了整个供热系统的稳定性和可靠性，保证供热质量的同时，优化了能源分配。
The present invention relates to the technical field of heating network control, and discloses an intelligent heating control method based on deep reinforcement learning and adaptive control, comprising the following steps: S1: collecting outdoor environment data and heat load data of each heating station, and performing data preprocessing; S2: training a deep reinforcement learning model based on the outdoor environment data and heat load data; S3: formulating a heating strategy based on the deep reinforcement learning model; S4: executing the heating strategy through an adaptive control algorithm. The present invention improves energy utilization efficiency, reduces unnecessary energy loss, enhances the stability and reliability of the entire heating system, ensures heating quality, and optimizes energy distribution.
Description
Technical Field
The invention relates to the technical field of heat supply pipe network regulation and control, in particular to an intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control.
Background
Along with the acceleration of the urban process and the deep penetration of the environment-friendly and energy-saving concepts, the intelligent and refined management of the heating system becomes the necessary trend of industry development. Especially in northern areas, central heating systems are in charge of guaranteeing the heating demands of residents in winter, and the running efficiency and the energy utilization rate of the central heating systems directly influence the energy consumption structure and the environmental protection level of cities. Modern heating systems have evolved from initially simple temperature control to employing automatic control systems, implementing the basic function of on-demand heating. The use of automation devices such as sensors and controllers enables the heating system to monitor the indoor and outdoor environmental temperature changes in real time and adjust the operating conditions of the heating device accordingly. Meanwhile, an advanced data analysis technology is also applied to the heat supply field, and future heat load demands can be predicted through mining and analyzing historical data, so that certain predictive regulation and control are realized.
However, the conventional heat supply regulation and control modes are mostly based on fixed rules or simplified mathematical models, and it is difficult to sufficiently cope with the complicated and changeable external environment conditions and the change of the heat demands of users. For example, a sudden weather change may cause an increase in the error of the existing prediction model, and thus the heating strategy cannot be timely and accurately adjusted, thereby causing a problem of excessive or insufficient heating. In addition, due to the lack of a dynamic optimization mechanism, the system is often difficult to realize globally optimal energy configuration, and the heat supply efficiency is greatly limited. The current heating strategies mostly depend on statically set thresholds and empirical rules, and do not have the capability of online learning and continuous optimization. In the face of rapidly changing environmental conditions and user behavior habits, such "one-cut" regulation and control methods are prone to waste of heat supply resources or unbalance of supply and demand. Because the regulation and control means are relatively single and not accurate enough, large deviation often exists between the energy consumption and the actual heat demand, and therefore, the operation cost is increased, and the environmental pollution is also aggravated. In summary, although the automation and informatization of the heating industry has advanced, many challenges remain in practical operation.
For example, china patent with the publication number CN110736129B discloses an intelligent balance control system and method for an urban heating network. The system comprises a first adjusting component, a second adjusting component, a plate heat exchanger unit and a control cabinet. Through set up first governing valve on the water supply pipe way once, set up first temperature sensor on the water return pipe way once, set up second governing valve and second temperature sensor on the water supply pipe way of secondary, set up the third governing valve on the bypass pipeline, adjust each valve according to the temperature value, separate the regulation of primary pipe network and the regulation of secondary pipe network, the regulation of primary pipe network relies on the return water temperature of primary pipe network to realize, the regulation of secondary pipe network relies on secondary water supply temperature to realize, when secondary network water supply temperature demand changes, the hydraulic operating mode of primary network is not influenced, the hydraulic operating mode of primary network is stable is guaranteed, the decoupling operation of primary pipe network and secondary pipe network has been realized, the difficult problem of primary pipe network and secondary pipe network hydraulic imbalance has been solved.
The patent with the application publication number of CN108592173A discloses a heat supply pipe network regulation and control method which comprises the following steps of intelligent monitoring information acquisition, corresponding module information, temperature information corresponding to a heat supply pipe network, pipeline information and heat source information in the heat supply pipe network, obtaining the heat supply load of the heat supply pipe network according to the current outdoor temperature information and the building information, and obtaining an actual value corresponding to the heat supply pipe network at the minimum total time consumption according to the heat supply load, the heat dissipation condition, the pipeline information and the heat source information. The invention can solve the disadvantage of the traditional heat supply network and reduce the total energy consumption of the heat supply network.
The problems of the prior art are that the complex and changeable external environment conditions and the change of the heat demand of the user are difficult to fully cope with, the regulation and control means are relatively single and inaccurate, and large deviation exists between the energy consumption and the actual heat demand.
The information disclosed in this background section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person of ordinary skill in the art.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides an intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control, improves energy utilization efficiency, reduces unnecessary energy loss, enhances the stability and reliability of the whole heat supply system, ensures heat supply quality and optimizes energy distribution.
In order to solve the technical problems, the invention provides the following technical scheme:
An intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control comprises the following steps: S1, collecting outdoor environment data and heat load data of each heating power station, and preprocessing the data;
s2, training a deep reinforcement learning model based on the outdoor environment data and the heat load data;
S3, formulating a heating strategy based on the deep reinforcement learning model;
and S4, executing the heating strategy through an adaptive control algorithm.
As a preferable scheme of the intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control, the outdoor environment data comprises temperature, humidity, wind speed and solar radiation intensity;
The thermal load data represents the amount of heat that the thermal station needs to provide to maintain the indoor temperature of all users to the heating standard temperature.
The preprocessing comprises missing value filling, abnormal value detection and replacement, data normalization and data synchronous integration, wherein the method for data synchronous integration is to correspond heat load data and outdoor environment data according to time, so as to form a time sequence corresponding to the heat load data and the outdoor environment data.
As an optimal scheme of the intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control, the method for training the deep reinforcement learning model comprises the following steps:
S100, setting a time period for heat supply regulation, constructing a state space, an action set and a reward function, constructing a strategy network and initializing parameters;
S200, selecting an action from the action collection set and executing the action;
S300, calculating a cumulative rewards value and updating parameters of a strategy network;
S400, entering the next time period, generating a new action set and updating the feature vector of the state space;
S500, repeating the steps S200-S400 until the accumulated rewards value converges to complete the training of the strategy network, storing the strategy network and deploying and applying the deep reinforcement learning model.
As a preferable scheme of the intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control, the time period is the minimum time unit for heat supply regulation and control;
the state space is composed of the outdoor environment data and heat load data;
the action set is dynamically generated based on the state space and the time period, and the method comprises the following steps:
For any time period A, extracting heat load data of each heating power station in a time period corresponding to the time period A in each of the previous n years, wherein n is a positive integer, averaging n heat load data of the ith heating power station, namely E Ai, wherein the value range of i is 1,2, m and m represent the number of the heating power stations, setting an arithmetic sequence containing k elements, namely Q, Q= { Q 1,q2,......,qk }, wherein Q 1 is the smallest element in the sequence Q, the value range is (0, 1), Q k is the largest element in the sequence Q, and the value range is (1, 2), and for the time period A, the form of an action set A a is as follows:
Aa＝{a1,a2,......,ak};
the specific form of each action is as follows:
aj＝{qj·EA1,qj·EA2,......,qj·EAm};
Where a j represents a j-th action in the action set, and j has a value range of 1, 2.
As an optimal scheme of the intelligent heat supply regulation method based on deep reinforcement learning and self-adaptive control, the calculation formula of the reward function is as follows:
Wherein s j represents a feature vector of a state space at a jth time period, a j represents an action performed at the jth time period, R (s j,aj) represents a bonus function value of the action a j performed at the time when the feature vector of the state space is s j, α is a weight parameter, β is a proportionality coefficient, and E ij represents a heat load supply amount allocated to the ith heat station at the jth time period; e j represents the total heat generation amount of the thermal power plant in the jth time period, and the calculation formula is as follows:
Where η i denotes the heat loss coefficient when the thermal power plant delivers a heat load to the i-th thermal power station.
The intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control is characterized in that a characteristic vector of a state space consists of historical outdoor environment data, real-time outdoor environment data and historical heat load data of each heating power station, the updating method is that the current entering time period is set as B, the heat load data and the outdoor environment data of each heating power station in a time period corresponding to the time period B in each year of the previous n years are extracted, the real-time outdoor environment data are obtained, and the characteristic vector of the state space corresponding to the time period B is formed.
The intelligent heat supply regulation and control method based on the deep reinforcement learning and the self-adaptive control is characterized in that when the time period of each heat supply regulation and control starts, real-time outdoor environment parameters are collected and input into a deep reinforcement learning model, an action set is automatically generated by the model and the action with the highest selection probability is output, the heat load supply quantity of a heat power plant to each heat station in the current time period of the heat supply regulation and control is determined based on the action with the highest selection probability, and the total heat production quantity of the heat power plant in the current time period of the heat supply regulation and control is calculated.
The intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control is characterized in that the heat supply strategy is implemented by specifically comprising the steps of self-adaptively controlling the total heat generation amount of a thermal power plant and the heat load supply amount of the self-adaptively controlling the thermal power plant to each thermal power plant, wherein the method for self-adaptively controlling the total heat generation amount of the thermal power plant comprises the following steps:
Collecting fuel consumption in real time;
A PID controller is designed for the thermal power plant, and an error term e 0 (t) is calculated, and the formula is as follows:
e0(t)＝Ej-E(t);
wherein E (t) represents the actual heat generation amount at the current moment, and the calculation formula is as follows:
E(t)=η0·m(t)·Hv;
Wherein η 0 denotes a boiler heat efficiency, m (t) denotes a fuel mass which has been currently consumed, H v denotes a heating value of the fuel;
Calculating a control output u 0 (t) at the current moment through a PID algorithm, wherein u 0 (t) represents the regulated supply quantity of the fuel;
Based on u 0 (t), adjusting the fuel supply by the hardware device;
the actual heat production is continuously monitored, and the fuel supply is adjusted until the actual heat production reaches the total heat production.
As an optimal scheme of the intelligent heat supply regulation method based on deep reinforcement learning and self-adaptive control, the method for self-adaptively controlling the heat load supply quantity of the thermal power plant to each heat power station comprises the following steps:
Collecting the primary water supply temperature, the primary backwater temperature and the primary water supply instantaneous flow of each heating power station in real time;
The PID controller is designed for each heating station, and the error term e i (t) of the PID controller of the ith heating station is calculated as follows:
ei(t)＝Eij-Ei(t);
Wherein E i (t) represents the heat load value actually obtained by the ith heating station, and the calculation formula is as follows:
Ei(t)＝c·(Tri-Tsi)·Qi(t);
Wherein c is the specific heat capacity of water, T ri is the primary backwater temperature of the ith heating station, T si is the primary water supply temperature of the ith heating station, Q i (T) is the total primary water supply amount of the ith heating station from the current time period to the current time, and the total primary water supply amount is obtained by integrating the instantaneous primary water supply flow with time;
Calculating a control output u i (t) of each heating power station at the current moment through a PID algorithm, wherein u i (t) represents the opening degree of a branch valve of the ith heating power station and the frequency of a circulating pump;
And continuously monitoring the heat load value actually acquired by each heating power station, and adjusting the opening degree of the branch valve and the frequency of the circulating pump until the heat load value actually acquired by each heating power station reaches the distributed heat load supply quantity.
Compared with the prior art, the invention has the following beneficial effects:
The deep reinforcement learning is applied to a heat supply regulation scene, so that the transition from passive on-demand heat supply to active prediction and optimization of a heat supply strategy is realized, and the intelligent level of a heat supply system is remarkably improved. The method comprises the steps of finishing macroscopic heat supply strategy planning through deep reinforcement learning, considering complex factors such as seasonality, weather change, predicted user demands and the like, rapidly responding according to the current actual conditions through self-adaptive control, overcoming the limitation of a single technology in heat supply regulation and control, reducing the complexity of a deep reinforcement learning model, relieving the burden of the deep reinforcement learning model and accelerating the convergence speed of a network, and organically combining the two, thereby not only meeting the long-term stable and efficient operation of a heat supply system, but also taking into account the short-term flexible regulation and control demands.
The heat supply strategy based on deep learning can be more accurately matched with the actual heat demand, and the situation of excessive heat supply or insufficient heat supply is avoided, so that the energy utilization efficiency is improved, the unnecessary energy loss is reduced, and the energy conservation and emission reduction are facilitated. The self-adaptive control algorithm is used for executing the heat supply strategy, the total heat generation amount of the thermal power plant and the heat load supply amount received by each heat station can be dynamically adjusted according to actual conditions, the stability and reliability of the whole heat supply system are enhanced, the heat supply quality is ensured, and meanwhile, the energy distribution is optimized.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
FIG. 1 is a flow chart of an intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control provided by the invention;
FIG. 2 is a flow chart of a method for training a deep reinforcement learning model for formulating a heating strategy in accordance with the present invention.
Detailed Description
The following detailed description of the present invention is made with reference to the accompanying drawings and specific embodiments, and it is to be understood that the specific features of the embodiments and the embodiments of the present invention are detailed description of the technical solutions of the present invention, and not limited to the technical solutions of the present invention, and that the embodiments and the technical features of the embodiments of the present invention may be combined with each other without conflict.
Example 1
This embodiment describes an intelligent heat supply regulation and control method based on deep reinforcement learning and adaptive control, and referring to fig. 1, the method includes the following steps:
An intelligent heat supply regulation and control method based on deep reinforcement learning and self-adaptive control comprises the following steps:
s1, collecting outdoor environment data and heat load data of each heating power station, and preprocessing the data;
The outdoor environment data comprises temperature, humidity, wind speed and solar radiation intensity, and the outdoor environment data can be used for predicting the heat load requirement of each heating station. Temperature is one of the most significant factors in the heat load demand. Lower outdoor temperatures may result in increased heating heat load. In heating season, the high humidity can make the human body feel cooler, and the demand of people for heating is increased. The wind power influences the convection heat exchange of the outer surface of the building, and the heat dissipation process of the building shell can be accelerated due to the fact that the wind speed is increased, so that the heat loss is increased. Solar radiation has direct influence on the heat gain of a building, especially for a building with a large-area glass curtain wall or a south window, the indoor temperature can be obviously improved by sunlight irradiation, and the heating requirement is reduced.
The thermal load data represents the amount of heat that the thermal station needs to provide to maintain the indoor temperature of all users to the heating standard temperature.
The preprocessing comprises missing value filling, abnormal value detection and replacement, data normalization and data synchronous integration;
the method for synchronously integrating the data is to correspond the heat load data and the outdoor environment data according to time to form a time sequence corresponding to the heat load data and the outdoor environment data.
The method for filling the missing values is one of linear interpolation, polynomial interpolation and prediction based on a time sequence model;
The method of data normalization is one of minimum-maximum normalization and Z-score normalization, and because outdoor environment parameters and heat load data tend to have different dimensions and numerical ranges, the parameters and the heat load data need to be converted to the same dimension so that a deep reinforcement learning model can process all input features. After the preprocessing step is completed, the collected data can be used as effective input of a deep reinforcement learning model, so that an intelligent model capable of adjusting a heating strategy according to real-time environmental conditions is constructed.
S2, training a deep reinforcement learning model based on the outdoor environment data and the heat load data, wherein referring to FIG. 2, the method comprises the following steps:
S100, setting a time period for heat supply regulation, constructing a state space, an action set and a reward function, constructing a strategy network and initializing parameters;
The time period is the minimum time unit for heat supply regulation and control, for example, one day is set as one time period;
the state space is composed of the outdoor environment data and heat load data;
the action set is dynamically generated based on the state space and the time period, and the method comprises the following steps:
For any time period A, extracting heat load data of each heating power station in a time period corresponding to the time period A in each of the previous n years, wherein n is a positive integer, averaging n heat load data of the ith heating power station, namely E Ai, wherein the value range of i is 1,2, m and m represent the number of the heating power stations, setting an arithmetic sequence containing k elements, namely Q, Q= { Q 1,q2,......,qk }, wherein Q 1 is the smallest element in the sequence Q, the value range is (0, 1), Q k is the largest element in the sequence Q, and the value range is (1, 2), and for the time period A, the form of an action set A a is as follows:
Aa＝{a1,a2,......,ak};
the specific form of each action is as follows:
aj＝{qj·EA1,qj·EA2,......,qj·EAm};
Wherein a j represents the j-th action in the action set, and the value range of j is 1, 2. Q j represents a j-th element in the arithmetic array Q;
The scheme is used for setting the action based on the historical heat load requirement of each heating station as a reference, for example, the optional action of the ith heating station is set to be 0.8 times, 0.9 times, 1.0 times, 1.1 times and 1.2 times of the historical average heat load requirement, and the strategy network optimizes the action based on the historical data setting through the comparison of the real-time outdoor environment parameters and the historical outdoor environment parameters, so that the training speed of the deep reinforcement learning model is improved.
The calculation formula of the reward function is as follows:
Wherein s j represents a feature vector of a state space in a jth time period, a j represents an action performed in the jth time period, R (s j,aj) represents a bonus function value of the action a j performed when the feature vector of the state space is s j, alpha is a weight parameter, beta is a proportionality coefficient, which is set by a person skilled in the art according to actual requirements, E ij represents a heat load supply amount allocated to an ith heat station in the jth time period, and the heat load supply amount is determined by the action selected to be performed in the jth time period; e j represents the total heat generation amount of the thermal power plant in the jth time period, and the calculation formula is as follows based on the heat load supply amount of each thermal power station:
Wherein η i represents the heat loss coefficient of the thermal power plant when delivering heat load to the ith thermal power station, which is experimentally determined by one skilled in the art;
the rewarding function gives consideration to the balance between the total heat production of the thermal power plant and the supply and demand of each thermal power station, saves the total energy consumption, and maximally balances the uniformity degree of the heat load supply quantity of each thermal power station.
The strategy network comprises an input layer, a hiding layer and an output layer, wherein the input layer is used for inputting characteristic vectors of a state space, the hiding layer is used for further extracting characteristics of the state space, and the output layer is used for generating selection probability of each action in the action aggregation set under the current state space. Converting the output into a probability distribution using a softmax function to ensure that the sum of the selection probabilities of all actions is 1;
S200, selecting an action from the action collection set and executing the action;
The method for selecting an action is as follows:
inputting the feature vector of the current state space into the strategy network to obtain the selection probability of each action in the action aggregation set under the current state space;
setting a threshold parameter epsilon, wherein the value range is (0,0.15);
Generating a random number r, wherein the value range is [0,1], if r is more than or equal to epsilon, executing the action with highest selection probability, and if r is less than epsilon, randomly selecting one action from the action collection set and executing the action;
S300, calculating a cumulative rewards value and updating parameters of a strategy network;
the cumulative prize value is calculated as follows:
Wherein R N represents the current cumulative prize value, N represents the number of actions that have been performed, β represents the discount factor, β j represents the power of the discount factor β to the power of j, R (s j,aj) represents the prize value for performing action a j in state space s j, and j has a range of values of 1, 2.
The calculation formula for updating the parameters of the policy network is as follows:
wherein δ represents any one parameter in the policy network; The gradient of the function pair delta in brackets is represented, eta is the learning rate, L N is the objective function, and the calculation formula is as follows:
LN＝ln(p(sN,aN)·RN);
Where p (s N,aN) represents the probability of selection of action a N in the environmental state s N.
S400, entering the next time period, generating a new action set and updating the feature vector of the state space;
The characteristic vector of the state space consists of historical outdoor environment data, real-time outdoor environment data and historical heat load data of each heating power station, and the characteristic vector updating method of the state space comprises the steps of enabling the current entering time period to be B, extracting the heat load data and the outdoor environment data of each heating power station in the time period corresponding to the time period B in each year of the previous n years, acquiring the real-time outdoor environment data and forming the characteristic vector of the state space corresponding to the time period B.
In the training stage of the deep reinforcement learning model, the real-time outdoor environment data is obtained by actually reading outdoor environment data in a corresponding time period in a state space, and after model training is completed, the real-time outdoor environment data is collected in real time through a sensor and other detection equipment in the actual deployment and application process.
S500, repeating the steps S200-S400 until the accumulated rewards value converges to complete the training of the strategy network, storing the strategy network and deploying and applying the deep reinforcement learning model.
After multiple iterations, the jackpot value tends to stabilize and no significant fluctuations occur, i.e., the jackpot value is considered to converge, and the policy network has been able to make decisions to maximize the jackpot value.
S3, formulating a heating strategy based on the deep reinforcement learning model, wherein the method comprises the following steps:
And determining the heat load supply quantity of the heat power plant to each heat station in the current heat supply regulation time period based on the action with the highest selection probability, and calculating the total heat production quantity of the heat power plant in the current heat supply regulation time period.
And S4, executing the heating strategy through an adaptive control algorithm. The method comprises the steps of adaptively controlling the total heat generation amount of the thermal power plant and adaptively controlling the heat load supply amount of the thermal power plant to each thermal power plant.
The method for adaptively controlling the total heat generation amount of the thermal power plant comprises the following steps:
Collecting fuel consumption in real time;
A PID controller is designed for the thermal power plant, and an error term e 0 (t) is calculated, and the formula is as follows:
e0(t)＝Ej-E(t);
wherein E (t) represents the actual heat generation amount at the current moment, and the calculation formula is as follows:
E(t)=η0·m(t)·Hv;
Wherein η 0 denotes the boiler thermal efficiency, the efficiency of converting the heat energy released by the combustion of the fuel into the heat of the feed water for the boiler, m (t) denotes the mass of fuel currently consumed, H v denotes the heating value of the fuel, the heat released when the fuel is completely combusted for a unit mass is generally given in the product manual of the fuel;
Calculating a control output u 0 (t) at the current moment through a PID algorithm, wherein u 0 (t) represents the regulated supply quantity of the fuel;
Mapping the control output signal u 0 (t) into the adjustment range of the actual fuel supply quantity, and driving a fuel valve by adjusting a combustion controller or a servo motor to realize accurate control of the fuel supply quantity;
Continuously monitoring the actual heat generation amount, and adjusting the fuel supply amount until the actual heat generation amount reaches the total heat generation amount;
The parameters of the PID controller are preliminarily set through on-site debugging and an empirical formula according to the requirements of the response characteristics of the boiler and the stability of the system, and are adjusted regularly.
The method for adaptively controlling the heat load supply amount of the thermal power plant to be transmitted to each thermal power station comprises the following steps:
the method comprises the steps of collecting the primary water supply temperature, the primary backwater temperature and the primary water supply instantaneous flow of each heating station in real time, and acquiring heat supply network data in real time by installing a flowmeter and a thermometer on a main pipe network.
The PID controller is designed for each heating station, and the error term e i (t) of the PID controller of the ith heating station is calculated as follows:
ei(t)＝Eij-Ei(t);
Wherein E i (t) represents the heat load value actually obtained by the ith heating station, and the calculation formula is as follows:
Ei(t)＝c·(Tri-Tsi)·Qi(t);
Wherein c is the specific heat capacity of water, T ri is the primary backwater temperature of the ith heating station, T si is the primary water supply temperature of the ith heating station, Q i (T) is the total primary water supply amount of the ith heating station from the current time period to the current time, and the total primary water supply amount is obtained by integrating the instantaneous primary water supply flow with time;
Calculating a control output u i (t) of each heating power station at the current moment through a PID algorithm, wherein u i (t) represents the opening degree of a branch valve of the ith heating power station and the frequency of a circulating pump;
The distribution proportion of the primary water supply quantity transmitted by the thermal power station to each thermal power station can be adjusted by adjusting the opening of the branch valve and the frequency of the circulating pump, and then the distribution proportion of the heat load supply quantity of each thermal power station is adjusted.
And continuously monitoring the heat load value actually acquired by each heating power station, and adjusting the opening degree of the branch valve and the frequency of the circulating pump until the heat load value actually acquired by each heating power station reaches the distributed heat load supply quantity.
In deep reinforcement learning, the advantages of adaptive control can help improve the performance and convergence speed of the algorithm. The self-adaptive control can solve part of bottom layer control problems by monitoring and adjusting system parameters in real time, and especially can perform local optimization faster when facing the change of the dynamic characteristics of the system, so as to keep the stability of the system.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are all within the protection of the present invention.
Claims (6)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410578917.3A CN118519340B (en) | 2024-05-10 | 2024-05-10 | Intelligent heating control method based on deep reinforcement learning and adaptive control |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410578917.3A CN118519340B (en) | 2024-05-10 | 2024-05-10 | Intelligent heating control method based on deep reinforcement learning and adaptive control |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118519340A CN118519340A (en) | 2024-08-20 |
| CN118519340B true CN118519340B (en) | 2025-02-11 |
Family
ID=92276615
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410578917.3A Active CN118519340B (en) | 2024-05-10 | 2024-05-10 | Intelligent heating control method based on deep reinforcement learning and adaptive control |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118519340B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118780941A (en) * | 2024-09-12 | 2024-10-15 | 广东联航智能科技有限公司 | Energy-saving management early warning method and system based on big data |
| CN119554904B (en) * | 2025-01-24 | 2025-05-06 | 深圳大学 | Temperature storage device based on chemical reversible thermal effect and control method of temperature storage device |
| CN120043149A (en) * | 2025-04-24 | 2025-05-27 | 北京智合瑞行能源科技有限公司 | Heat network balancing system and method based on heat station load characteristic difference weighted algorithm |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109631150A (en) * | 2018-12-19 | 2019-04-16 | 天津宏达瑞信科技有限公司 | A kind of accurate heat supply method of temperature based on secondary network, pressure and user's room temperature |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102715372B1 (en) * | 2022-02-25 | 2024-10-14 | 경희대학교 산학협력단 | An auto-adaptive controller tuning system for wastewater treatment process using data-driven smart decisions of offline reinforcement learning |
| CN114909706B (en) * | 2022-04-24 | 2024-05-07 | 常州英集动力科技有限公司 | Two-level network balance regulation and control method based on reinforcement learning algorithm and differential pressure control |
| CN116697445A (en) * | 2023-06-06 | 2023-09-05 | 巴云海 | Heat supply control method for deep learning |
-
2024
- 2024-05-10 CN CN202410578917.3A patent/CN118519340B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109631150A (en) * | 2018-12-19 | 2019-04-16 | 天津宏达瑞信科技有限公司 | A kind of accurate heat supply method of temperature based on secondary network, pressure and user's room temperature |
Non-Patent Citations (1)
| Title |
|---|
| 基于策略梯度的多热力站热量分配优化研究;谭梦媛;中国优秀硕士学位论文全文数据库工程科技II辑;20230331;第2023年(第03期);C038-894 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118519340A (en) | 2024-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN118519340B (en) | Intelligent heating control method based on deep reinforcement learning and adaptive control | |
| CN110458443B (en) | Smart home energy management method and system based on deep reinforcement learning | |
| Kang et al. | A novel approach of day-ahead cooling load prediction and optimal control for ice-based thermal energy storage (TES) system in commercial buildings | |
| CN111561732B (en) | Heat exchange station heat supply adjusting method and system based on artificial intelligence | |
| CN111580382B (en) | Unit-level heat supply adjusting method and system based on artificial intelligence | |
| Lu et al. | Data augmentation strategy for short-term heating load prediction model of residential building | |
| Yuan et al. | Analysis and evaluation of the operation data for achieving an on-demand heating consumption prediction model of district heating substation | |
| CN118316092B (en) | Intelligent control method, device, system, electronic device and storage medium for coordinated power consumption of source, grid, load and storage | |
| CN115333168B (en) | A field-level control strategy for offshore wind farms based on distributed rolling optimization | |
| CN116734424A (en) | Control method of indoor thermal environment based on RC model and deep reinforcement learning | |
| CN119713515B (en) | Optimal control method of central air-conditioning system for building load forecasting | |
| CN115455835A (en) | Optimized operation method of renewable energy-containing multi-heat-source networked heating system | |
| CN117913812B (en) | Deep learning-based interactive regulation and control method for wind-solar power supply and flexible load | |
| CN115471006A (en) | Power supply planning method and system considering wind power output uncertainty | |
| CN112560160A (en) | Model and data-driven heating ventilation air conditioner optimal set temperature obtaining method and equipment | |
| CN116995659A (en) | Flexible operation method of heat pump system considering renewable energy source consumption | |
| CN119382251B (en) | Cooperative optimization method of multiple types of power sources based on complementary characteristics | |
| CN119647672B (en) | A near-zero energy building energy supply method and system based on multi-objective optimization | |
| Li et al. | Enhancing demand response and heating performance of air source heat pump through optimal water temperature scheduling: Method and application | |
| CN119227998A (en) | Energy management strategy for electric-thermal integrated energy system based on DQN-CE algorithm with noise network and self-attention mechanism | |
| CN116485582A (en) | Heat supply optimization regulation and control method and device based on deep learning | |
| CN116454996A (en) | Hydropower station real-time load distribution method based on deep reinforcement learning | |
| CN114372645A (en) | Energy supply system optimization method and system based on multi-agent reinforcement learning | |
| EP4109200A1 (en) | Acquiring a user consumption pattern of domestic hot water and controlling domestic hot water production based thereon | |
| CN116107206A (en) | Greenhouse equipment control method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |