CN118836523A - Central air conditioner purifying method and system based on deep reinforcement learning - Google Patents

Abstract

The present invention relates to the field of energy management technology, specifically a central air conditioning purification method and system based on deep reinforcement learning. The specific implementation method is: obtaining indoor and outdoor environmental parameters and ASHRAE data sets, and preprocessing the obtained data; using the ASHRAE data set to train a feedforward neural network, and predicting the thermal comfort of individuals in the indoor environment through the trained neural network; accurately identifying and counting the distribution of people in different areas of the building through the population density detection method, and providing key crowd density information; defining the MDP model, including state space, action space, reward function, transfer function and discount factor; through the PRE-DDPG algorithm, the system dynamically adjusts the air conditioning operation parameters according to the real-time collected environmental data and personnel needs, so as to purify the indoor air quality and improve the comfort to the greatest extent, while effectively saving energy consumption, in line with the modern building management concept of energy conservation and emission reduction.

Description

A central air conditioning purification method and system based on deep reinforcement learning

Technical Field

The present invention relates to the technical field of energy management, and specifically to a central air-conditioning purification method and system based on deep reinforcement learning.

Background Art

With the acceleration of global urbanization and the increasing prominence of building energy consumption, the construction industry has an increasingly urgent need to improve energy efficiency and indoor environmental quality. As an important component of building energy consumption, central air-conditioning systems face many challenges in maintaining indoor comfort and providing air purification functions.

Traditional central air-conditioning systems often adopt fixed control strategies and preset operation modes, which cannot flexibly respond to different seasons, different weather conditions and frequently changing indoor air quality requirements, resulting in energy waste and poor air purification effects. At present, some air-conditioning optimization methods and systems based on model control have appeared on the market. Ambroziak automatically optimizes the proportional integral differential control settings of the air handling unit according to the current operating actuator information to improve its energy efficiency and trouble-free operation time (Ambroziak A. The PID controller optimisation module using fuzzy self-tuning PSO for air handling unit in continuous operation. Engineering Applications of Artificial Intelligence, 2023, 117: 105485). Xie used PID to control the fan speed and used fuzzy reasoning to adjust the PID parameters. The objective function designed according to the performance index was optimized through genetic algorithm to improve the heat transfer efficiency and energy saving effect of the terminal fan (Xie R.GA Optimized fuzzy PID control with modified smith predictor for HVACterminal fan system, IEEE, 2022: 1098-1104).

However, with the rapid advancement of science and technology, central air-conditioning systems are facing increasing complexity and dynamics. Therefore, it is urgent to introduce more intelligent and adaptive control methods to promote continuous updates in the field of building energy conservation. Zeng proposed an adaptive MPC architecture that regularly relearns building dynamics and unmeasured internal heat loads from data, and convexly approximates the original non-convex problem to maintain indoor temperature while reducing energy consumption (Zeng T. An Adaptive Model Predictive Control Scheme for Energy-Efficient Control of Building HVAC Systems. Journal of Engineering for Sustainable Buildings and Cities, 2021, 2 (3): 1-10). Tang established a model of the chiller, using the MPC method, and used the first-order exponential smoothing method in the prediction part to deal with the prediction errors caused by model simplification and inaccuracy, thereby improving the final control effect (Tang R. Model predictive control for thermal energy storage and thermal comfort optimization of building demand response in smart grids. Applied Energy, 2019, 242: 873-882). Although MPC control may be more effective, it is not easy to construct a simplified and sufficiently accurate building model in practice. This is because the indoor environment is affected by many factors, such as building structure, building layout, internal building heat, and outdoor environment. When the model cannot accurately describe the building thermal dynamics and there is a large deviation, the control performance may deviate from expectations.

The current central air conditioning air purification system and method lack flexibility and have low control accuracy. In order to reduce building energy consumption, improve indoor air quality and indoor comfort according to indoor environmental characteristics, improving flexibility and accuracy is one of the keys to solving the deficiencies and problems in existing related technologies. To this end, a central air conditioning purification system and method based on deep reinforcement learning is proposed.

Summary of the invention

The purpose of the present invention is to provide a central air-conditioning purification system and method based on deep reinforcement learning, which accurately predicts the thermal comfort of individuals in the indoor environment through a feedforward neural network, thereby realizing refined control and optimization of the central air-conditioning; secondly, through the population density calculation formula, it can accurately identify and count the distribution of people in different areas of the building, and can also provide key crowd density information; finally, a PRE-DDPG algorithm is proposed, and the system can dynamically adjust the air-conditioning operating parameters according to the real-time collected environmental data and personnel needs, so as to maximize the indoor air quality and comfort, while effectively reducing the building energy consumption, in line with the modern building management concept of energy conservation and emission reduction.

To achieve the above object, the present invention provides the following technical solutions:

A central air conditioning purification method based on deep reinforcement learning, comprising:

Step S1, data collection and collation, obtaining indoor and outdoor environmental parameters and ASHRAE data sets, performing data cleaning, missing value processing and data normalization on the indoor and outdoor environmental parameters and ASHRAE data sets, and dividing the indoor and outdoor environmental parameters and ASHRAE data sets into a training set and a test set;

Step S2, predicting the PMV value by a feedforward neural network, using the training set of the ASHRAE data set to train the feedforward neural network offline; importing the test set of the ASHRAE data set into the optimized PRE-DDPG model through iterative updating and multiple trainings to obtain an accurate feedforward neural network model; inputting the temperature, humidity and wind speed in the indoor and outdoor environmental parameters into the feedforward neural network to predict the PMV value in real time;

Step S3, a population density detection method, which calculates and identifies the distribution of people in different areas of the building through a population density detection formula, and provides key crowd density information;

Step S4, designing an MDP model, including a state space, an action space, a reward function, a transfer function and a discount factor, and inputting the PMV value into the state space;

Step S5, running the PRE-DDPG algorithm, using the population density detection method and the training set of the indoor and outdoor environmental parameters on the MDP model, dynamically adjusting the central air-conditioning operating parameters through iterative updates and multiple trainings, and obtaining the optimized PRE-DDPG model; importing the test set of the indoor and outdoor environmental parameters to obtain the optimal control strategy for indoor central air-conditioning purification of the building.

Further, the step S1 comprises:

The indoor and outdoor environmental parameters are collected in real time through sensors installed in the building, including temperature, humidity, _CO2 concentration, air supply rate, number of people and time in different areas, and data including environmental parameters and comfort indicators are obtained from the ASHRAE data set;

Perform the data cleaning, use the mean imputation method to process missing values, determine which values are missing in the data set, for each feature with the missing values, calculate the mean of the non-missing values of the feature, and replace the missing values with the mean of the feature;

Use the triple standard deviation method to detect and process outliers in the data. If the value of the feature exceeds three times the mean, it is marked as an outlier and the anomaly is replaced with the mean of the feature.

All data are normalized, and the normalization formula is:

Where X is the original data, X _norm is the normalized value, X _min and X _max are the minimum and maximum values of the feature respectively.

Further, the step S2 comprises:

Determine the dimensionality of the input layer, i.e. the number of features the network receives, including temperature, humidity, and wind speed;

Determine the number of hidden layers and the number of neurons in each layer;

Determine the structure of the output layer. The regression task only requires one neuron to output the predicted value.

Select the hidden layer activation function as ReLU;

Select the mean square error as the loss function to measure the gap between the predicted value and the true value, and select the Adam optimizer to optimize the weights and biases of the network;

Using the training set in the ASHRAE data to train the feedforward neural network, adjusting the hyperparameters of the model according to the performance, including the learning rate, batch size, and the number of neurons in the hidden layer;

Monitoring the performance of the feedforward neural network to check whether the feedforward neural network is overfitting or underfitting;

If the overfitting occurs, the number of layers and neurons of the neural network is reduced; if the underfitting occurs, the training time is increased, that is, the number of training rounds is increased; if the overfitting and underfitting do not occur, no further adjustment is required;

Loading the trained feedforward neural network, inputting the training set in the ASHRAE data, and calculating the accuracy of predicting the PMV value;

The temperature, humidity and wind speed among the indoor and outdoor environmental parameters are input into the feedforward neural network to predict the PMV value in real time.

Further, the step S3 comprises:

At each interval Δt, the indoor and outdoor _CO2 concentrations in the indoor and outdoor environment are obtained, and the increase in indoor _CO2 concentration ΔC is calculated, which is expressed as:

ΔC＝CC ₀ ;

Wherein, the indoor CO ₂ concentration is C, and the outdoor CO ₂ concentration is C ₀ ;

The _CO2 concentration increase ΔC is also expressed as:

Among them, I is the per capita CO ₂ production, K is the population density, and V is the indoor volume;

According to the above formula, the population density formula K is:

When the population density exceeds a maximum population threshold M, a warning is issued.

Furthermore, in step S4, a reinforcement learning MDP model is designed. The modeling process of the MDP model is based on the target problem, that is, purifying indoor air quality and improving comfort, while effectively saving energy consumption. The continuous state variables and action variables are read, so that the intelligent agent can continuously interact with the environment and take corresponding actions in different states to obtain reward values. After multiple iterations, the solution of how to obtain the highest reward is mastered, which is expressed as:

Among them, Φ is the MDP model, S is the state space, A is the action space that the agent can take, P is the transfer function, R is the reward function, and γ is the discount factor.

Further, the state space is expressed as:

Among them, S _t is the state space at time t, Temp _out,t is the outdoor temperature at time t, is the indoor temperature of area z at time t, Hum _out,t is the outdoor humidity at time t, is the indoor humidity in area z at time t, C _out,t is the CO ₂ concentration at time t, is the CO ₂ concentration in area z at time t, PMV _t ^z is the predicted value of thermal comfort at time t, is the number of people at time t.

Further, the action space is expressed as:

Among them, A _t is the action space at time t, represents the air supply rate of zone z at time t, and the set point of each zone is a continuous variable.

Furthermore, the reward function is expressed as:

Among them, R _t is the reward function at time t, is the thermal comfort reward of area z at time t, R _energy,t is the energy consumption reward at time t, is the air quality reward for area z at time t, a, b and d are the weighting factors, and n is the total amount of the area;

The thermal comfort reward is expressed as:

in, is the thermal comfort reward of area z at time t, PMV _t ^z is the PMV value of area z at time t, and PMV _penalty is the penalty item when the threshold M is exceeded. is the population density of area z at time t. When it is detected that the indoor population density is higher than the threshold M and higher than the maximum comfort threshold c, the reward is the absolute value of the current PMV value and the additional penalty term. When the indoor situation is other, the reward is the absolute value of the current PMV value.

The air quality reward is expressed as:

in, is the air quality reward for area z at time t, is the CO ₂ concentration in area z at time t, C _high is the highest threshold of CO ₂ concentration, C _penalty is the additional penalty item when the threshold C _high is exceeded, and when it is detected that the indoor CO ₂ concentration exceeds the fixed standard, the reward is C _penalty .

Further, in step S5, the PRE-DDPG algorithm runs, including:

Step S5-1, the agent observes the indoor environment state s _t ;

Step S5-2, the agent predicts the current PMV value of each area indoor according to the indoor environmental state;

Step S5-3: The agent calculates the current population density of each area

Step S5-4: The agent selects control actions for each area based on the current strategy and exploration noise That is, the air supply rate in the central air conditioning system in area z at time t;

Step S5-5: The agent performs the action in the environment Calculate the reward function r _t and observe the next state s _t ′ based on the PRE-DDPG model;

Step S5-6: Store it in the experience replay pool D, and let s _t = s _t ′;

Step S5-7: If the experience replay pool D is full, randomly sample N transitions from D. A small batch consisting of, where j is the sample number;

Step S5-8, update the Critic network by minimizing the loss function, and update the Actor network using the policy gradient;

Step S5-9, using soft update to update the target Actor network and the target Critic network;

Step S5-10: If the PRE-DDPG learning process is not effective, the network parameters and discount factors can be continuously adjusted based on the model operation status to achieve better convergence of the entire learning effect;

Step S5-11, repeat the above steps S5-1 to S5-10 for a total of n times until the optimal strategy with the maximum cumulative reward value is learned, that is, the optimal control operation strategy.

Step S5-12: import the test set of indoor and outdoor environmental parameters into the trained PRE-DDPG model to obtain the optimal control strategy for indoor central air-conditioning purification of the building.

Furthermore, a central air conditioning purification system based on deep reinforcement learning includes:

A data acquisition and processing unit, used for acquiring indoor and outdoor environmental parameters and ASHRAE data sets, performing data cleaning, missing value processing and data normalization on the indoor and outdoor environmental parameters and ASHRAE data sets, and dividing the indoor and outdoor environmental parameters and ASHRAE data sets into a training set and a test set;

A feedforward neural network unit is used to train the feedforward neural network offline using the training set of the ASHRAE data set; after the training is completed, the test set of the ASHRAE data set is imported into the trained PRE-DDPG model, and an accurate feedforward neural network model is obtained through iterative updates and multiple trainings; the temperature, humidity and wind speed in the indoor and outdoor environmental parameters are input into the feedforward neural network to predict the PMV value in real time;

Population density detection unit, which is used to calculate the population density detection formula and identify the distribution of people in different areas of the building, and provide key crowd density information;

A deep reinforcement learning unit, for involving an MDP model, including a state space, an action space, a reward function, a transfer function, and a discount factor, and inputting the PMV value into the state space;

The central air-conditioning control unit is used to run the PRE-DDPG algorithm, run the PRE-DDPG algorithm on the MDP model, use the population density detection method, through iterative update and multiple training, use the training set of indoor and outdoor environmental parameters to dynamically adjust the air-conditioning operation parameters to obtain the trained PRE-DDPG model; import the test set of indoor and outdoor environmental parameters into the trained PRE-DDPG model to obtain the optimal control strategy for indoor central air-conditioning purification of the building.

Furthermore, the population density detection unit responds according to the result of the data acquisition and processing unit and is applied to the deep reinforcement learning unit to provide an optimal decision-making method for the central air conditioning control unit.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention uses the ASHRAE data set to train a feedforward neural network model and predicts the PMV values of different areas through real-time environmental data, so that the central air conditioner can adjust the strategy in a personalized manner according to individual needs and real-time environmental changes in different areas, thereby achieving more flexible and precise control.

2. The present invention designs a population density detection method, which effectively manages and optimizes the density of people flow inside the building through accurate population density detection and real-time data analysis. In the event of an emergency or crowd gathering, the system can respond quickly and adjust the air conditioning operating parameters to ensure safety and comfort, thereby improving the accuracy and flexibility of central air conditioning control.

3. This paper introduces a method based on deep reinforcement learning and designs a PRE-DDPG algorithm. This algorithm does not rely on an accurate building model. Through intelligent air-conditioning system control and optimization, it can effectively improve the air quality and thermal comfort of the building, reduce energy consumption costs, and achieve more flexible and precise control.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a central air conditioning purification method based on deep reinforcement learning provided by an embodiment of the present invention;

FIG2 is a schematic diagram of a central air conditioning purification system for a multi-zone commercial building provided by an embodiment of the present invention;

FIG3 is a schematic diagram of a feedforward neural network for predicting indoor thermal comfort values according to an embodiment of the present invention;

FIG4 is a schematic diagram of a network structure of a PRE-DDPG algorithm provided in an embodiment of the present invention;

FIG5 is a schematic diagram of a central air conditioning purification system based on deep reinforcement learning provided in an embodiment of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The current central air conditioning air purification system and method lack flexibility and are not very accurate. Therefore, in order to reduce building energy consumption, improving indoor air quality and indoor comfort according to indoor environmental characteristics is one of the keys to solving the deficiencies and problems in existing related technologies.

Embodiment 1

As shown in FIG1 , the present invention provides a central air conditioning purification method based on deep reinforcement learning, and the technical solution is as follows:

Step S1, data collection and organization, obtaining indoor and outdoor environmental parameters and ASHRAE data sets, performing data cleaning, missing value processing and data normalization on the indoor and outdoor environmental parameters and ASHRAE data sets, and dividing the indoor and outdoor environmental parameters and ASHRAE data sets into a training set and a test set.

Specifically, when analyzing the factors affecting the indoor air environment and human thermal comfort, it is necessary to identify the control variables and controlled variables that are easy to implement. It is worth noting that the adjustment of the air supply rate will also directly affect energy consumption. Therefore, when optimizing the indoor environment and thermal comfort, energy efficiency must be considered at the same time.

As shown in Figure 2, suppose a commercial building has 4 areas, and the central air conditioner includes an air handling unit and a variable air volume box in each area. The air handling unit consists of a damper, a cooling/heating coil, and a variable frequency drive blower. The cooling/heating coil can cool/heat the mixed air, and the variable frequency drive blower delivers the cooled/heated mixed air to the variable air volume box in each area, and delivers the air through the air handling unit and exhausts the indoor air to purify the indoor air quality. In order to simplify the control operation and more clearly reflect the control effect, only the air supply rate of different areas is controlled.

The temperature, humidity, _CO2 concentration and number of people in different areas will change over time. These parameters are easy to collect and can therefore be used as key parameters of the indoor environment. By installing sensors in the building, the indoor and outdoor environmental parameters can be collected in real time and effectively used as reference factors for the indoor thermal environment.

The ASHRAE dataset contains detailed and extensive environmental parameters and comfort indicators. Data containing environmental parameters and comfort indicators are obtained from the ASHRAE dataset, including but not limited to temperature, humidity, wind speed, and comfort indicator PMV value.

The data cleaning is performed. In addition, missing values in the data may affect the results of subsequent analysis and modeling. Therefore, the missing values need to be processed to maintain the integrity and availability of the data. The mean imputation method is used to process the missing values, determine which values are missing in the ASHRAE data set and indoor and outdoor environmental parameters, and for each feature with the missing values, calculate the mean of the non-missing values of the feature, and replace the missing values with the mean of the feature.

The three-times standard deviation method is used to detect and process outliers in the data. If the value of the feature is beyond three times the mean, it is marked as an outlier and the anomaly is replaced with the mean of the feature.

All data are normalized to avoid training instability caused by large differences in eigenvalues. The normalization formula is:

Where X is the original data, X _norm is the normalized value, X _min and X _max are the minimum and maximum values of the feature respectively.

The preprocessed ASHRAE data set and indoor and outdoor environmental parameters were divided into 80% training sets and 20% test sets. Through comprehensive data preprocessing, the quality and reliability of the data were improved, providing a solid foundation for subsequent modeling and optimization, and ensuring that the model can more accurately predict and control the indoor environment.

Step S2, a feedforward neural network predicts the PMV value, using the training set of the ASHRAE data set to train the feedforward neural network offline; through iterative updates and multiple trainings, the test set of the ASHRAE data set is imported into the trained PRE-DDPG model to obtain an accurate feedforward neural network model; the temperature, humidity and wind speed in the indoor and outdoor environmental parameters are input into the feedforward neural network to predict the PMV value in real time.

Specifically, wind speed corresponds to the air supply rate, and the PMV value is an indicator for evaluating human thermal comfort, which comprehensively considers multiple factors that affect the human body. In addition to temperature, humidity and wind speed, PMV also includes radiant temperature, activity level and individual metabolic rate. Temperature, humidity and wind speed are direct factors affecting human thermal comfort and have a great impact on human comfort, so they are important considerations in the PMV model. Since other factors (such as radiant temperature, activity level and individual metabolic rate) are not easy to obtain, they are not considered for the time being.

The feedforward neural network has a strong nonlinear modeling capability and can effectively capture these complex relationships. As shown in Figure 3, the dimensions of the input layer include temperature, humidity, and wind speed, a total of 3 features. The number of hidden layers is 2, and each layer contains 20 neurons to enhance the network's ability to learn nonlinear relationships. If the learning effect is not good, it can be adjusted at any time. Since it is a regression task, the output layer only needs one neuron to output the predicted PMV value, which directly reflects the prediction of the comfort level of the human body under the current environmental conditions. The hidden layer activation function is selected as ReLU, which can effectively handle nonlinear relationships and alleviate the gradient disappearance problem, so that the network can better learn complex features. The loss function is selected as the mean square error to measure the gap between the predicted value and the true value, and the Adam optimizer is selected to optimize the weights and biases of the network.

The learning rate is set to 0.001 and the batch size is set to 64. After 500 rounds of training, the performance of the feedforward neural network on the training set is monitored, and hyperparameters such as the learning rate, batch size, and the number of neurons in the hidden layer are adjusted in a timely manner to ensure that the model has good generalization ability in practical applications and avoid overfitting or underfitting problems.

The trained feedforward neural network is loaded, the training set in the ASHRAE data is input, and the accuracy of the predicted PMV is calculated. Through the feedforward neural network, the accurate prediction of human thermal comfort is achieved, providing a solid data and model foundation for the intelligent control of the central air-conditioning system.

Step S3, a population density detection method, calculates and identifies the distribution of people in four areas of the building through a population density detection formula, and provides key crowd density information.

Specifically, in order to simulate this process, a discrete time step method is used, with half an hour as a time unit, represented by t = 0, 1, 2, .... Every half an hour, the indoor and outdoor _CO2 concentrations are measured, and the increase in indoor _CO2 concentration ΔC is calculated:

ΔC＝CC ₀ ;

Wherein, the indoor CO ₂ concentration is C, and the outdoor CO ₂ concentration is C ₀ ;

The increase in CO ₂ concentration is:

Among them, I is the per capita CO ₂ production, K is the population density, and V is the indoor volume;

From the above formula, we can deduce that the population density formula is:

Wherein, K is the population density, and a warning is issued when the population density exceeds the maximum population threshold M. Through accurate population density detection and real-time dynamic adjustment, the operation flexibility and accuracy of the central air-conditioning system are improved to ensure the comfort and health of the indoor environment.

Step S4: Taking the commercial building as the environment and the PRE-DDPG algorithm as the agent, the MDP model is designed, the PMV value is predicted with the help of a feedforward neural network, and the PMV value is input into the state space. The agent interacts in the environment and continuously optimizes its strategy through the learning process to achieve the goal of maximizing the cumulative reward.

Specifically, the constructed MDP model is expressed as:

Among them, Φ is the MDP model, S represents the state space, which refers to the set of environmental states that the agent can perceive and process; A represents the action space that the agent can take, which refers to the operations that the agent can perform; P represents the transfer function, which refers to the probability distribution of the system transferring from one state to another; R represents the reward function, which refers to the immediate feedback obtained after the agent takes an action in the environment; γ represents the discount factor, which is used to measure the importance of the agent to future rewards.

Further, the state space is expressed as:

Among them, S _t is the state space at time t, Temp _out,t is the outdoor temperature at time t, is the indoor temperature of area z at time t, Hum _out,t is the outdoor humidity at time t, is the indoor humidity in area z at time t, C _out,t is the CO ₂ concentration at time t, is the CO ₂ concentration in area z at time t, PMV _t ^z is the predicted value of thermal comfort at time t, is the number of people at time t. By using the state space definition method, the intelligent agent can more comprehensively grasp and respond to changes in the indoor and outdoor environments, and improve the real-time performance and flexibility of the central air-conditioning system.

Further, the action space _At is expressed as:

in, It represents the air supply rate of the z zone at time t. The set point of each zone is a continuous variable. By accurately controlling the air supply rate of different zones, we can better cope with different temperature, humidity and number of people. For example, according to the temperature difference of different zones, the air supply rate is adjusted so that the temperature of each zone can be maintained within a comfortable range to avoid overcooling or overheating.

Furthermore, the reward function is expressed as:

Among them, R _t is the reward function at time t, is the thermal comfort reward for area z at time t, R _energy,t is the energy consumption reward at time t, which is automatically calculated by the environment. is the air quality reward for area z at time t, n is the total area, a, b and d are trade-off factors, and the importance of different rewards is considered by dynamically adjusting the trade-off factors;

Further, the thermal comfort reward is expressed as:

in, is the thermal comfort reward of area z at time t, PMV _t ^z is the PMV value of area z predicted by the feedforward neural network at time t, PMV _penalty is the penalty term when the threshold M is exceeded, PMV _penalty is set to 20, M is set to 3, is the population density of area z at time t. When it is detected that the indoor population density is higher than the threshold M and higher than the maximum comfort threshold c, the reward is the absolute value of the current PMV value and an additional penalty term. The maximum comfort threshold c is set to 1. When other conditions are detected indoors, the reward is the absolute value of the current PMV value.

When it is detected that the indoor _CO2 concentration exceeds the maximum threshold, a negative reward is added. The air quality reward is specifically expressed as follows:

in, is the air quality reward for area z at time t, is the CO ₂ concentration in area z at time t, C _high is the maximum CO ₂ concentration threshold, C _penalty is the penalty term when the threshold C _high is exceeded, C _penalty is defined as 20, and C _high is defined as 1,200 ppm. By comprehensively considering the optimization goals of thermal comfort, energy consumption, and air quality, and dynamically adjusting the trade-off factors. For example, when the system detects that the PMV value of the current area deviates from the ideal value, the trade-off factor will increase the focus on thermal comfort, prompting the system to prioritize the adjustment of relevant parameters.

Furthermore, the discount factor is used to measure the importance the agent places on future rewards. In the commercial building system, the balance between comfort optimization and energy consumption is taken into account to achieve long-term system benefits and user satisfaction. Here, the discount factor is defined as 0.9, and the transfer function is determined by the environment.

Step S5, running the PRE-DDPG algorithm, the PRE-DDPG algorithm runs on the MDP model, uses the population density detection method, and dynamically adjusts the air-conditioning operation parameters using the training set of the indoor and outdoor environmental parameters, and obtains the trained PRE-DDPG model through iterative updates and multiple trainings; imports the test set of the indoor and outdoor environmental parameters into the trained PRE-DDPG model to obtain the optimal control strategy for indoor central air-conditioning purification of the building.

In this example, the state space required by the central air-conditioning system in the commercial building is very large, so the support of deep neural networks is particularly important. When running the PRE-DDPG algorithm, the combination of deep learning and reinforcement learning can automate control, reduce dependence on manual operation and monitoring, and simplify the management process.

Specifically, the PRE-DDPG algorithm, as shown in Figure 4, in terms of network structure, DDPG uses two types of networks, namely Actor networks and Critic networks. At the same time, it also continues the idea of using a fixed target network in DQN, and each network contains a target network and an estimation network. The traditional PG method adopts a random strategy, and each time an action is obtained, the distribution of the current optimal strategy needs to be sampled, while DDPG adopts a deterministic strategy. The input of the Actor network is the current state, and the output is a deterministic action. The Critic network is used to fit the state-action value function. Its input consists of the current state and the action generated by the Actor network, and the output is the Q value of the current state-action pair. This Q value will be further used to update the parameters of the Actor network.

Implementation steps include:

Step S5-1, randomly initialize the Actor network m(s|q ^u ) and the Critic network Q(s,a|q ^Q ), and the parameters of the two networks are represented by θ ^μ and θ ^Q respectively;

Step S5-2, initializing the target network: q ^u′ ←q ^u , q ^Q′ ←q ^Q , the parameters of the two target networks are represented by q ^u′ and q ^Q′ respectively, which are used to stabilize the target value estimation in the training process;

Step S5-3, initialize the experience replay pool. The experience replay mechanism enables the algorithm to learn from historical experience, improve sample efficiency and learning stability, especially perform well in non-stationary environments;

Step S5-4: perform iteration.

Further, the iterative process of step S5-4 is as follows:

Step S5-4-1, the agent observes the initial state {Temp _{out, t = 0} , Temp _{in, t = 0} , Hum _{out, t = 0} , Hum _{in, t = 0} };

Step S5-4-2: The agent predicts the initial thermal comfort value of each area indoors

Step S5-4-3: The agent calculates the current population density of each area represents the initial value of population density;

Step S5-4-4: The agent selects an action based on the current strategy and exploration noise That is, the air supply rate regulation in the central air conditioning system in area z at time t;

Step S5-4-5: The agent performs actions in the environment Calculate the reward r _t according to the reward function and observe the next state s _t ′ based on the PRE-DDPG model;

Step S5-4-6: Store it in the experience replay pool D for subsequent learning, and let s _t = s _t ′;

Step S5-4-7: If the experience replay pool D is full, randomly sample N transitions from D. The small batch composed of , let y _j = r(s _j , a _j ) + Q(s _j ′, m(s _j ′|q ^m′ )|q ^Q′ );

Step S5-4-8, update the Critic network Q(s,a|q ^Q ) by minimizing the loss function, and use the policy gradient to update the Actor network m(s|q ^u );

Step S5-4-9, use soft update to update the target Actor network and the target Critic network to stabilize the training process;

Step S5-4-10: Environmental simulation stops and the algorithm stops.

Step S5-4-11, import the test set of indoor and outdoor environmental parameters into the trained PRE-DDPG model to obtain the optimal control strategy for indoor central air-conditioning purification of the building.

In order to demonstrate the effectiveness of the method proposed in the present invention, two comparative schemes are introduced. Specifically, comparative scheme 1 does not use the feedforward neural network model, and uses the traditional on/off method to control the central air conditioner; comparative scheme 2 does not use the feedforward neural network model, and uses DDPG to control the central air conditioner.

Table 1 Performance comparison

PRE-DDPG Solution 1 Solution 2 Average energy consumption (kWh) 150 200 180 PMV value 0.8 1.1 1.2 Air quality (CO2ppm) 800 1000 900

Table 1 shows the performance of the proposed algorithm and the comparison scheme. The thermal comfort of individuals in the indoor environment is accurately predicted by the feedforward neural network. The thermal comfort PMV value of the proposed PRE-DDPG algorithm is closest to the ideal comfort value of 0, thereby realizing the refined control and optimization of the central air conditioner. Secondly, through the population density calculation formula, the distribution of people in different areas of the building can be identified and counted, and key crowd density information can also be provided. Finally, the PRE-DDPG algorithm is proposed. The system can dynamically adjust the air conditioning operation parameters according to the real-time collected environmental data and personnel needs. The proposed algorithm has the lowest energy consumption and the best air quality. It can maximize the improvement of indoor air quality and comfort, while effectively reducing building energy consumption, which is in line with the modern building management concept of energy conservation and emission reduction.

Embodiment 2

As shown in FIG5 , a central air conditioning purification system based on deep reinforcement learning includes:

A data acquisition and processing unit is used to obtain indoor and outdoor environmental parameters (including temperature, humidity, _CO2 content, etc.) and ASHRAE data sets, perform data cleaning, missing value processing and data normalization on the indoor and outdoor environmental parameters and ASHRAE data sets, and divide the indoor and outdoor environmental parameters and ASHRAE data sets into a training set and a test set. The data acquisition and processing unit can accurately prepare data and efficiently distribute data;

A feedforward neural network unit is used to train the feedforward neural network offline using the training set of the ASHRAE data set; after the training is completed, the test set of the ASHRAE data set is imported into the trained PRE-DDPG model, and an accurate feedforward neural network model is obtained through iterative updates and multiple trainings; the temperature, humidity and wind speed in the indoor and outdoor environmental parameters are input into the feedforward neural network, and the PMV value can be predicted in real time through the feedforward neural network unit;

A population density detection unit is used to calculate and identify the distribution of people in different areas of the building using a population density detection formula, and to provide key crowd density information. The population density detection unit can effectively improve the responsiveness of the central air conditioning system. For example, in areas with fewer people, the system will reduce the air supply rate, thereby reducing energy consumption; while in areas with dense population, the system will increase the air supply to ensure comfort and air quality;

A deep reinforcement learning unit is used to involve an MDP model, including a state space, an action space, a reward function, a transfer function and a discount factor, and input the PMV value into the state space, and improve the intelligence and adaptability of the system by dynamically adjusting the air-conditioning operating parameters, and can be flexibly adjusted according to real-time data;

A central air-conditioning control unit is used to run a PRE-DDPG algorithm, run the PRE-DDPG algorithm on the MDP model, use the population density detection method, and dynamically adjust the air-conditioning operation parameters using the training set of indoor and outdoor environmental parameters to obtain the trained PRE-DDPG model; and import the test set of indoor and outdoor environmental parameters into the trained PRE-DDPG model.

Furthermore, the population density detection unit calculates the air quality and energy consumption according to the result response of the data acquisition and processing unit, and uses the feedforward neural network unit to calculate the thermal comfort value, which is applied to the deep reinforcement learning unit, and provides a control method for the central air conditioning control unit according to the obtained reward function. After multiple iterations, the optimal decision of the central air conditioning is output. Through the collaborative work of multiple units, closed-loop management of data acquisition, processing, analysis, prediction, decision-making and control is realized, which improves the overall control accuracy and flexibility of the central air conditioning system.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims (10)

1. A central air conditioning purification method based on deep reinforcement learning, characterized by comprising: Step S1, data collection and collation, obtaining indoor and outdoor environmental parameters and ASHRAE data sets, performing data cleaning, missing value processing and data normalization on the indoor and outdoor environmental parameters and ASHRAE data sets, and dividing the indoor and outdoor environmental parameters and ASHRAE data sets into a training set and a test set; Step S2, predicting the PMV value by a feedforward neural network, using the training set of the ASHRAE data set to train the feedforward neural network offline; importing the test set of the ASHRAE data set into the trained feedforward neural network model through iterative updating and multiple trainings to obtain an accurate feedforward neural network model; inputting the indoor and outdoor environmental parameters into the feedforward neural network to predict the PMV value in real time; Step S3: Detecting population density, and identifying the distribution of people in different areas of the building by calculating a population density detection formula; Step S4, designing an MDP model, including a state space, an action space, a reward function, a transfer function and a discount factor, and inputting the PMV value into the state space; Step S5, running the PRE-DDPG algorithm, using the population density detection method and the indoor and outdoor environmental parameter training set on the MDP model, dynamically adjusting the central air-conditioning parameters through multiple iterative training, and obtaining an optimized PRE-DDPG model; importing the test set of the indoor and outdoor environmental parameters to obtain the optimal central air-conditioning control strategy. 2. According to a central air conditioning purification method based on deep reinforcement learning according to claim 1, it is characterized in that the step S1 comprises: Collect the indoor and outdoor environmental parameters in real time, and obtain data including environmental parameters and comfort indexes from the ASHRAE data set; Perform the data cleaning, use the mean filling method to deal with the missing values, and use the triple standard deviation method to detect and deal with outliers in the data; All data were normalized as described. 3. According to a central air conditioning purification method based on deep reinforcement learning according to claim 1, it is characterized in that the step S2 comprises: Determine the dimensionality of the input layer, i.e. the number of features the network receives, including temperature, humidity, and wind speed; Determine the number of hidden layers and the number of neurons in each layer; Determine the structure of the output layer; Select the hidden layer activation function as ReLU; Select the mean square error as the loss function and the Adam optimizer to optimize the weights and biases of the network; Training the feedforward neural network using a training set in the ASHRAE data, and adjusting the hyperparameters of the model based on performance; Monitor the performance of the feedforward neural network to check whether it is overfitting or underfitting. If overfitting occurs, reduce the number of layers and neurons of the neural network. If underfitting occurs, increase the number of training rounds. If overfitting or underfitting does not occur, no further adjustment is required. Load the trained feedforward neural network model, input the training set in the ASHRAE data, and calculate the accuracy; The temperature, humidity and wind speed among the indoor and outdoor environmental parameters are input into the feedforward neural network model to predict the PMV value in real time. 4. According to a central air conditioning purification method based on deep reinforcement learning according to claim 1, it is characterized in that the step S3 comprises: At each time interval Δt, the indoor and outdoor _CO2 concentrations in the indoor and outdoor environmental parameters are obtained, and the increase in indoor _CO2 concentration ΔC is calculated, which is expressed as: ΔC＝CC ₀ ; Wherein, the indoor CO ₂ concentration is C, and the outdoor CO ₂ concentration is C ₀ ; The _CO2 concentration increase ΔC is expressed as: Where I is the per capita _CO2 production, K is the population density, and V is the indoor volume; It is deduced that the population density K is expressed as: When the population density exceeds a maximum population threshold M, a warning is issued. 5. According to a central air conditioning purification method based on deep reinforcement learning according to claim 1, it is characterized in that the state space in step S4 is expressed as: Among them, S _t is the state space at time t, Temp _out,t is the outdoor temperature at time t, is the indoor temperature of area z at time t, Hum _out,t is the outdoor humidity at time t, is the indoor humidity in area z at time t, C _out,t is the CO ₂ concentration at time t, is the CO ₂ concentration in region z at time t, is the predicted value of thermal comfort at time t, is the number of people at time t. 6. According to a central air conditioning purification method based on deep reinforcement learning in claim 1, it is characterized in that the action space _At in step S4 is expressed as: in, represents the air supply rate of zone z at time t, and the set point of each zone is a continuous variable. 7. According to a central air conditioning purification method based on deep reinforcement learning according to claim 1, it is characterized in that the reward function in step S4 is expressed as: Among them, R _t is the reward function at time t, is the thermal comfort reward of area z at time t, R _energy,t is the energy consumption reward at time t, is the air quality reward for area z at time t, a, b and d are the weighting factors, and n is the total amount of the area; The thermal comfort reward is expressed as: in, is the thermal comfort bonus for area z at time t, is the PMV value of area z at time t, and PMV _penalty is the penalty term when the maximum population threshold M is exceeded. is the population density of area z at time t, and c is the maximum comfort threshold; The air quality reward is expressed as: in, is the air quality reward for area z at time t, is the CO ₂ concentration in area z at time t, C _high is the maximum threshold of CO ₂ concentration, and C _penalty is the additional penalty item when exceeding the threshold C _high . 8. According to a central air conditioning purification method based on deep reinforcement learning according to claim 1, it is characterized in that the step S5 comprises: Step S5-1, the agent observes the state s _t of each area; Step S5-2, the agent predicts the current PMV value of each area according to the indoor environment state; Step S5-3, the agent calculates the current population density of each area; Step S5-4: The agent selects control actions for each area based on the current strategy and exploration noise That is, the air supply rate in area z at time t; Step S5-5: The agent performs the action in the environment Calculate the reward function r _t to obtain the state s _t ′ at the next moment; Step S5-6: Store it in the experience replay pool D, and let the s _t = s _t ′; Step S5-7: When the experience replay pool D is full, randomly sample N A small batch consisting of, where j is the sample number; Step S5-8, update the Critic network by minimizing the loss function, and update the Actor network using the policy gradient; Step S5-9, using soft update to update the target Actor network and the target Critic network; Step S5-10: If the learning process of the PRE-DDPG algorithm is not effective, the network parameters and discount factors are continuously adjusted based on the model operation status to achieve better convergence of the entire learning effect; Step S5-11, importing the test set of indoor and outdoor environmental parameters into the optimized PRE-DDPG model to obtain the optimal control strategy for indoor central air-conditioning purification of the building. 9. A central air conditioning purification system based on deep reinforcement learning, characterized by comprising: A data acquisition and processing unit, used for acquiring indoor and outdoor environmental parameters and ASHRAE data sets, performing data cleaning, missing value processing and data normalization on the indoor and outdoor environmental parameters and ASHRAE data sets, and dividing the indoor and outdoor environmental parameters and ASHRAE data sets into a training set and a test set; A feedforward neural network unit is used to train the feedforward neural network offline using the training set of the ASHRAE data set; through iterative updates and multiple trainings, the test set of the ASHRAE data set is imported into the optimized PRE-DDPG model to obtain a feedforward neural network model with accurate prediction; the temperature, humidity and wind speed in the indoor and outdoor environmental parameters are input into the feedforward neural network model to predict the PMV value in real time; Population density detection unit, used to calculate population density and identify the distribution of people in different areas of the building, providing key crowd density information; A deep reinforcement learning unit, used for designing an MDP model, including a state space, an action space, a reward function, a transfer function and a discount factor, and inputting the PMV value into the state space; The central air-conditioning control unit is used to run the PRE-DDPG algorithm, use the population density detection method and the indoor and outdoor environmental parameter training set on the MDP model, dynamically adjust the air-conditioning parameters through multiple iterative training, and obtain the optimized PRE-DDPG model; import the indoor and outdoor environmental parameters of the test set to obtain the best central air-conditioning control strategy. 10. According to claim 9, a central air conditioning purification system based on deep reinforcement learning is characterized in that the population density detection unit responds according to the result of the data acquisition and processing unit and is applied to the deep reinforcement learning unit to provide an optimal decision-making method for the central air conditioning control unit.