CN117346285B - Indoor heating and ventilation control method, system and medium - Google Patents

Abstract

The invention discloses an indoor HVAC control method, system and medium. The method includes: acquiring indoor video data and environmental parameters; inputting the video data into a trained indoor window segmentation model IWS-Net to obtain a window segmentation mask image. And calculate the indoor window opening degree based on the window segmentation mask image; calculate the backlight flow sequence and human skeleton key point sequence corresponding to the video data, and combine the video data, backlight flow sequence and human skeleton key point sequence Sequentially input the trained indoor personnel behavior recognition model IDARM to obtain the indoor personnel thermal comfort behavior; based on the obtained indoor window opening degree, indoor personnel thermal comfort behavior and the environmental parameters, calculate the indoor HVAC control adjustment strategy. The invention controls and adjusts the HVAC system based on indoor environmental parameters, window opening degree and indoor personnel behavior to achieve energy saving and emission reduction and improve indoor thermal comfort.

Description

Indoor HVAC control method, system and medium

Technical Field

The present invention relates to an indoor HVAC control method, system and medium, and in particular to computer vision and HVAC control, belonging to the technical field of artificial intelligence and image processing.

Background Art

Energy consumption in the building sector accounts for about 40% of the total energy consumption in the world, of which about half is used for building air conditioning. Energy consumption in the building sector must be reduced. The behavior of people opening windows is the most important factor affecting HVAC control. Without the use of intelligent control, the frequent opening of windows by people in the building will sharply increase energy consumption and operating costs.

Summary of the invention

In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides an indoor HVAC control method, system and medium, which are used to control and adjust the HVAC system according to indoor environmental parameters, window opening degree and indoor occupant behavior, so as to achieve energy conservation and emission reduction and improve indoor thermal comfort.

In one aspect, the present invention provides an indoor HVAC control method, comprising:

Obtain indoor video data and environmental parameters;

Input the video data into the trained indoor window segmentation model IWS-Net to obtain a window segmentation mask image, and calculate the degree of opening of the indoor window according to the window segmentation mask image;

Calculating the inverse optical flow sequence and the human skeleton key point sequence corresponding to the video data, and inputting the video data, the inverse optical flow sequence and the human skeleton key point sequence into the trained indoor personnel behavior recognition model IDARM to obtain the thermal comfort behavior of the indoor personnel;

The indoor HVAC control adjustment strategy is determined based on the obtained indoor window opening degree, indoor occupant thermal comfort behavior and the environmental parameters.

Furthermore, the video data is obtained by a camera. The camera can be placed on the top of the room to ensure that it can clearly capture the overall indoor and window images, and measure the distance and angle between it and the detection window, as well as its own pitch angle, yaw angle and roll angle.

Specifically, the video acquired by the camera is sampled at a certain frequency to obtain corresponding video frames, and then the video frames are input into the indoor window segmentation model IWS-Net for window segmentation.

Furthermore, the environmental parameters are obtained by an indoor thermal comfort measuring instrument, which mainly obtains indoor air temperature, relative humidity, air flow rate and carbon dioxide concentration.

Optionally, a cyclic full-domain converter RAFT is used to calculate a corresponding inverse optical flow sequence from the RGB video frame sequence obtained from the camera, and the Mediapipe library is used to detect skeleton key points for each frame image in the video to form a human skeleton key point sequence.

Furthermore, the operation of calculating the inverse optical flow sequence is as follows:

(1) Select two adjacent frames in the video frame sequence and record them as and , and fill its width and height to multiples of 8;

(2) In and After the exchange, it is used as the input of the pre-trained cyclic full-domain converter RAFT to obtain the inverse optical flow map;

(3) Reverse the direction of the values in the inverse optical flow map and repeat (1) until the video ends.

Furthermore, the operation of calculating the key points of the human skeleton in each frame is as follows:

(1) Separate the video frame sequence frame by frame;

(2) Use Mediapipe in Python to detect the key points of the human skeleton for each frame of the image;

(3) Store the 33 human skeleton key points obtained in (2) and repeat (1) until the end.

As a further technical solution, the training of the indoor window segmentation model IWS-Net further includes:

Collecting images of indoor windows, preprocessing the images and forming a training set;

Inputting the training set into the constructed indoor window segmentation model IWS-Net for training to obtain model parameters that meet the accuracy requirements;

The model parameters are loaded into the indoor window segmentation model IWS-Net to obtain a trained indoor window segmentation model IWS-Net.

Furthermore, after collecting the images of indoor windows, the images are annotated using labelme, the walls, windows and window edges in the images are segmented, and the rest are classified as background. All data are divided into a test set and a training set according to the ratio.

As a further technical solution, the operation process of the indoor window segmentation model IWS-Net includes:

A backbone network consisting of five feature extraction modules is used to extract features from the input indoor window image, and the output of each feature extraction module is saved;

Using three attention modules connected sequentially, the output of the last feature extraction module is used to suppress background features and enhance the features of the required segmented parts;

The reconstruction module is used to reconstruct the output of the last attention module and the output of each feature extraction module to obtain mask images of walls, windows, window sills and background.

The above technical solution uses the IWS-Net deep learning model for indoor window segmentation, which has the following characteristics: (1) It adopts the depthwise separable convolution and dilated convolution calculation methods to expand the receptive field while reducing the operation parameters, thereby improving the calculation speed and segmentation accuracy; (2) It adopts the multi-channel convolution method and uses different convolution kernels to extract image features, which increases the diversity of features, improves the expression ability of the model, reduces the risk of model overfitting, and accelerates the model training and reasoning process; (3) It adopts the attention mechanism to suppress the background information in the image, enhance the feature information of the part to be segmented, and improve the attention of IWS-Net to the area of interest, thereby improving the accuracy and robustness of segmentation.

Furthermore, the reconstruction module includes five upsampling modules, wherein the first The input of each module is The output of the module and The output of the feature extraction module composition.

As a further technical solution, the method further includes:

After obtaining the mask images of the wall, window, window edge and background, the window image is extracted, and the extracted window image is filtered, threshold segmented and opened in sequence;

Obtaining several minimum inscribed rectangles on the processed window image and the vertex coordinates and area of each rectangle;

The sliding window opening ratio is calculated based on the number of rectangles obtained, the vertex coordinates and the area of each rectangle.

Specifically, the window mask is used to extract the window in the image, and Gaussian filtering is performed to remove noise; then threshold segmentation is used to generate a binary image; then the binary image is opened to remove the fuzzy spots on the binary image. The minimum inscribed rectangle is found on the obtained image, the number of rectangles is counted, and the coordinates of their four vertices are obtained. Finally, the opening ratio of the sliding window is calculated based on the number of rectangles, the coordinates of the four vertices, and the area.

As a further technical solution, the method further includes:

After obtaining the mask images of the wall, window, window edge and background, the straight line at the upper end of the window edge and the straight line at the lower end of the window edge are obtained by using Hough line detection;

Calculate the slope of two straight lines, and calculate the angle between the two straight lines based on the slope;

The deep neural network model DNN is used to combine the relative distance and angle between the camera and the window, as well as the pitch angle, yaw angle and roll angle of the camera to obtain the actual opening angle of the external casement window.

As a further technical solution, the training of the indoor personnel behavior recognition model IDARM further includes:

Obtain video data of indoor occupants’ thermal comfort behavior and construct a training set;

Input the training set into the constructed indoor personnel behavior recognition model IDARM for training to obtain model parameters that meet the accuracy requirements;

The model parameters are loaded into the indoor person behavior recognition model IDARM to obtain a trained indoor person behavior recognition model IDARM.

Furthermore, constructing the training set includes collecting videos of multiple subjects performing the following six actions: ① sitting; ② walking; ③ fanning with hands; ④ shaking clothes; ⑤ rubbing hands; ⑥ hugging shoulders, where each video is 3 to 5 seconds long and has a frame rate of 30FPS; the collected videos are divided into training set and test set in a 1:1 ratio.

As a further technical solution, the operation process of the indoor personnel behavior recognition model IDARM includes:

Extract features from the video frame sequence and the inverse optical flow sequence respectively to obtain video features and inverse optical flow features;

The video features, inverse optical flow features and position encoding are passed through the encoding module Encoder to perform data enhancement to obtain enhanced video features;

The enhanced video features, inverse optical flow features and human skeleton key point sequence are used as the input of the decoding module Decoder, and the optical flow query and content query are output;

The optical flow query and content query are subjected to multi-head linear transformation, and the confidence corresponding to the actions related to thermal comfort behavior is obtained through a fully connected network and a Softmax function.

As a further technical solution, the method further includes:

Constructing data tensors , where T is the air temperature, H is the relative humidity, V is the air velocity, C is the carbon dioxide concentration, L is the window opening degree, and A is the behavior of the indoor occupants;

According to the data tensor, an optimal control strategy is obtained by using an indoor temperature and humidity control algorithm;

According to the optimal control strategy, the indoor heating, ventilation and air conditioning control systems are adjusted.

In another aspect, the present invention provides an indoor HVAC control system, comprising:

An acquisition module is used to acquire indoor video data and environmental parameters;

A window opening degree calculation module is used to input the video data into a trained indoor window segmentation model IWS-Net to obtain a window segmentation mask image, and calculate the indoor window opening degree according to the window segmentation mask image;

An indoor occupant thermal comfort behavior recognition module is used to calculate the inverse optical flow sequence and the human skeleton key point sequence corresponding to the video data, and input the video data, the inverse optical flow sequence and the human skeleton key point sequence into the trained indoor occupant behavior recognition model IDARM to obtain the indoor occupant thermal comfort behavior;

The indoor HVAC control module is used to determine the indoor HVAC control adjustment strategy according to the obtained indoor window opening degree, indoor occupant thermal comfort behavior and the environmental parameters.

The present invention also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the indoor HVAC control method are implemented.

Compared with the prior art, the present invention has the following beneficial effects:

(1) The present invention uses indoor window image data to train the IWS-Net model to achieve the segmentation of walls, windows, window edges and backgrounds in indoor window images, and calculates the window opening degree through the sliding window opening ratio algorithm and the casement window opening angle algorithm.

(2) The present invention calculates the inverse optical flow sequence and the human skeleton key point sequence through the video frame sequence, and integrates the video frame sequence, the inverse optical flow sequence and the human skeleton key point sequence. Through the indoor occupant behavior recognition model IDARM, the end-to-end recognition of six behaviors related to thermal comfort performed by indoor occupants is achieved.

(3) The present invention controls the HVAC system according to the optimal control strategy through the indoor temperature and humidity adjustment control algorithm, balances the thermal comfort of personnel and the energy consumption of the building, avoids unnecessary energy waste, and thus achieves energy conservation. At the same time, it improves the production and living quality of the residents, improves the reliability and utilization efficiency of building equipment, reduces the failure rate and maintenance cost of equipment, and reduces the workload and cost of equipment maintenance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic flow chart of an indoor HVAC control method according to an embodiment of the present invention;

FIG2 is a flow chart of a window opening detection algorithm according to an embodiment of the present invention;

FIG3 is an overall network structure diagram of an indoor window segmentation model IWS-Net according to an embodiment of the present invention;

FIG4 is a structural diagram of a feature extraction module in an indoor window segmentation model IWS-Net according to an embodiment of the present invention;

FIG5 is a structural diagram of an attention module in an indoor window segmentation model IWS-Net according to an embodiment of the present invention;

FIG6 is a structural diagram of an upsampling module in an indoor window segmentation model IWS-Net according to an embodiment of the present invention;

FIG. 7 is an overall network structure diagram of an indoor personnel behavior recognition model IDARM according to an embodiment of the present invention.

Implementation

The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

As shown in Figure 1, it is a flow chart of an indoor HVAC control method provided by the present invention, which collects indoor thermal environment parameters through an indoor thermal comfort measuring instrument, and uses a camera to collect the global indoor image; uses a window opening detection algorithm and an indoor occupant behavior recognition algorithm to respectively obtain the degree of window opening and the thermal behavior related to thermal comfort performed by indoor occupants; finally, through the indoor temperature and humidity adjustment control algorithm, the optimal control strategy is calculated to control the indoor heating, ventilation, and air conditioning systems to balance the thermal comfort of indoor occupants and the energy consumption of the building.

The core technologies in the present invention include window opening detection algorithm, indoor personnel behavior recognition algorithm and indoor temperature and humidity adjustment control algorithm. The working principle and module functions will be explained in detail with reference to the illustrations.

1. Window Opening Detection Algorithm

The window opening detection algorithm is divided into four parts: (1) indoor window segmentation model IWS-Net; (2) sliding window opening ratio algorithm; (3) external casement window opening angle algorithm; (4) window opening degree algorithm. The process of the window opening detection algorithm is shown in Figure 2. The indoor window segmentation model IWS-Net is used to generate the required mask images of the wall, window, window edge and background; the sliding window opening ratio algorithm is used to calculate the opening area and proportion of the sliding window; the external casement window opening angle algorithm is used to calculate the actual opening angle of the external casement window; and finally, the actual opening degree of the two types of windows is calculated.

(1) Indoor window segmentation model IWS-Net

1) Working principle

The overall structure of the indoor window segmentation model IWS-Net is shown in Figure 3. Its input image is of fixed size (this method uses input size), IWS-Net consists of Backbone, attention module and reconstruction module.

The RGB image input by the network is first subjected to feature extraction by Backbone, which consists of five feature extraction modules. During the feature extraction process, the output of each module is saved and named . Then the final output of Backbone is passed through three attention modules. The function of the attention module is to suppress the background information in the feature map and enhance the feature information of the required segmented part. Finally, the feature tensor after the attention module is reconstructed through the reconstruction module to finally generate the required mask images of the wall, window, window edge and background. The reconstruction module consists of 5 upsampling modules and a depthwise separable convolution. The input of each upsampling module is the output of the previous upsampling module and the output of the corresponding feature extraction module of the backbone. The depthwise separable convolution reorganizes the channels of the last layer of upsampling modules so that the number of channels is equal to 4.

2) Feature extraction module

The specific structure of the feature extraction module is shown in Figure 4. Assume that the input feature tensor is The function of depthwise separable convolution 1 is to reorganize the channels of the input feature tensor. After depthwise separable convolution 1, the number of channels of the feature tensor remains unchanged. The depthwise separable convolution is recorded as , it can be expressed as the following formula:

,

in is the convolution kernel size; is the step length; is the filling pixel size; The number of channels after convolution output; The number of channels input for the feature map. is the feature tensor calculated by depth-wise separable convolution, is the height of the feature tensor, is the width of the feature tensor.

Then Different features are extracted through convolution calculations of different receptive fields. Since the hole convolution will change the size of the feature map, an upsampling operation is required to make the three feature maps of the same size. Finally, the three feature maps are spliced in the channel dimension.

;

in, is the ratio of the number of holes to the size of the convolution kernel, It is the output tensor of three feature tensors concatenated in the channel dimension; is the sampling function, is the concatenation function, , The receptive field size of depthwise separable convolution 1 is 3, the receptive field size of atrous convolution 1 is 5, and the receptive field size of atrous convolution 2 is 7.

Finally, The downsampling operation is performed, the size of the feature map becomes half of the original size, and the number of channels is changed to twice the original size through depthwise separable convolution 3, that is, .

,

in, is the feature tensor after downsampling, .

3) Backbone

The structure diagram of Backbone is shown in Figure 3. Backbone is composed of five feature extraction modules. Assume that the input RGB image is ,but , then The feature map output by the feature extraction module is denoted as ,in , , , is the height of the input image, is the width of the input image. The output of each feature extraction module is saved and recorded as , and the final output of Backbone is , A list of feature tensors extracted from each Backbone layer, used to store .

4) Attention Module

The structure diagram of the attention module is shown in Figure 5. Assume that the input feature map is , . First flatten it into a 1D tensor , and then perform a linear transformation on it, recorded as , it can be expressed as:

;

in, is the output feature tensor after linear transformation, .

For feature maps The coordinates are encoded, and the feature map The coordinates of each point can be expressed as , position encoding is to convert the coordinates into the number of channels The one-dimensional tensor of is as follows:

;

in, , , .

The principle of the multi-head self-attention mechanism is to divide the input data into multiple parts, each of which has an independent attention head. Each attention head calculates the weight associated with it in the input data, and then sums these weights to get the final output result. The attention mechanism requires three inputs, namely , , , which is abbreviated as , , .

The following is the calculation formula of the attention mechanism:

;

in, for The number of channels of the feature tensor.

In this module for ; , Both and The sum in the channel dimension can be expressed as follows:

.

The formula of the multi-head self-attention mechanism can be expressed as follows:

;

in, Both are transformation matrices that project tensors into another space. The output after the multi-head self-attention mechanism is recorded as .

Finally and After the sum of passes through the linear fully connected layer, the final output of the entire attention module can be expressed as follows:

.

5) Upsampling module

The structure diagram of the upsampling module is shown in Figure 6. The input of the upsampling module is the output of the previous layer module and the output feature tensor of the corresponding feature extraction module in Backbone. The output of the previous layer module is , the feature tensor output by the Backbone corresponding feature extraction module is recorded as .

Assumptions , , after splicing them and performing upsampling, it can be expressed as follows:

;

in, , Tensor concatenation is to concatenate two feature tensors in the channel dimension, so the size of the concatenated feature map remains unchanged, but the number of channels is doubled. The purpose of the upsampling operation is to double the size of the input feature map.

The role of depthwise separable convolution 1 is to reduce the number of channels by half, that is

;

in, .

Channel attention is composed of global average pooling and linear fully connected layers:

;

in, Represents global average pooling, which converts the original Perform global average pooling on the entire feature map. ,so .

Spatial attention is composed of channel average pooling and depth-wise separable convolution:

;

in, Represents channel average pooling, which converts the original Perform average pooling on the channels, ,so .

The function of depthwise separable convolution 2 is the same as 1, using represents the computation of depthwise separable convolutions 1 and 2, so after the upsampling operation, the output of the entire module is:

;

in, .

6) Reconstruction module

The specific structure of the reconstruction module is shown in Figure 3. The reconstruction module consists of 5 upsampling modules and 1 depthwise separable convolution. The input of each upsampling module is the output of the previous upsampling module and the output of the backbone corresponding feature extraction module. The depthwise separable convolution reorganizes the channels of the last upsampling module to make the number of channels equal to 4. Each channel corresponds to the wall, window, window edge and background class in the image.

(2) Sliding window opening ratio algorithm

The indoor window image is used as the input of the IWS-Net model to generate a mask image of the window and window edge. The window mask is used to extract the window in the image, and Gaussian filtering is performed to remove noise. Threshold segmentation is used to generate a binary image. The binary image is opened to remove the hair points on the binary image.

Find the minimum inscribed rectangle in the denoised image, count the number of rectangles, and get the coordinates of their four vertices , respectively represent the coordinates of the upper left, upper right, lower left, and lower right corners of the rectangle. The following divides the number of rectangles into X cases and discusses them separately:

1) The number of rectangles is 2, and for each rectangle, calculate and , , Respectively The height value of the coordinates of the midpoints of the top and bottom segments of the rectangle, where represents different matrices, in this case, .like or If the window is closed, the window is considered to be fully open, and the window opening ratio is 100%. This is a manually set threshold.

2) If the number of rectangles is 3, the window is considered to be fully open and the window opening ratio is 100%.

3) The number of rectangles is 4. Calculate the two rectangles on the left and right sides. and .if , then the window opening is judged to be on the left, and the window opening ratio is , otherwise, the window opening is judged to be on the right, and the window opening ratio is .in, , The heights of the coordinates of the midpoints of the top segments of the leftmost and rightmost rectangles respectively; , are the areas of the leftmost and rightmost rectangles respectively; For the The area of a matrix, in this case, .

(3) Calculation of the opening angle of the external casement window

The indoor window image is used as the input of the IWS-Net model to generate mask images of the window and window edge. The mask images of the window and window edge are extracted respectively, and the image opening operation is used to reduce the noise and burrs of the image. Then the threshold segmentation is performed, and the Hough line detection is used on the two images at the same time to obtain the straight line at the upper end of the window edge and the oblique line at the lower end of the window. The slopes of the two straight lines are obtained respectively. , and the intercept , .

The angle between two straight lines is calculated by the slope of the two straight lines. The calculation formula is as follows:

.

Establish a DNN model, set its input tensor length to 5, the number of hidden layers to 3, the number of hidden layer neurons to 10, and the number of output layer neurons to 1.

Open the windows in the image , the relative distance between the camera and the window , the angle between the camera and the front of the window , the camera's pitch angle , yaw angle and roll angle , concatenate to form a tensor .

Through DNN from Predict the actual window opening angle .

(4) Window opening degree algorithm

The sliding window opening ratio calculated in (2) can represent the degree of window opening. The actual opening angle of the casement window calculated in (3) cannot represent the degree of window opening. Therefore, to obtain the degree of window opening, the actual opening angle needs to be divided by the maximum opening angle.

,

in, is the window opening degree, and its value range is .

2. Indoor Person Behavior Recognition Algorithm

The indoor personnel behavior algorithm calculates the inverse optical flow sequence and the human skeleton key point sequence through the input video frame sequence, and uses it as the input of the indoor personnel behavior recognition model IDARM to perform end-to-end indoor personnel behavior recognition. It can mainly identify the six most common actions related to thermal comfort of indoor personnel, namely ① sitting; ② walking; ③ fanning with hands; ④ shaking clothes; ⑤ rubbing hands; ⑥ hugging shoulders. Among them, ① and ② represent thermal neutral behaviors, while the other 4 represent thermal discomfort behaviors. More specifically, ③ and ④ represent behaviors that feel hot, and ⑤ and ⑥ represent behaviors that feel cold.

Next, the generation of inverse optical flow sequences and human skeleton key point sequences and the indoor person behavior recognition model IDARM will be introduced in detail.

(1) Inverse optical flow sequence and human skeleton key point sequence

Assume that the input video frame sequence is ,common In order to obtain the human skeleton key point sequence, the Mediapipe library in Python is used to detect the human skeleton key points for each frame. 33 skeleton key points can be detected for each frame, and the coordinates of each point are ,in .

Therefore, the human skeleton key point sequence can be expressed as ,in Represents the The coordinates of all bone key points in the frame.

In order to obtain the inverse optical flow sequence, the pre-trained RAFT model is used to calculate the inverse optical flow. Select two adjacent frames in the video frame sequence and The normal optical flow calculation is as follows:

,

In order to obtain the inverse optical flow, the two adjacent input frames should be swapped and the speed direction of the optical flow should be reversed. The specific formula is as follows:

,

Therefore, the inverse optical flow sequence can be expressed as .

(2) Indoor Person Behavior Identification Model IDARM

In order to make the length of the inverse optical flow sequence, video frame sequence and skeleton key point sequence the same, the first frame of video image and its detected skeleton key point coordinates are removed. The overall network structure diagram of the indoor person behavior recognition model IDARM is shown in Figure 7.

First, the video frame sequence and the inverse optical flow sequence are extracted through two backbones to obtain the video features. and backlight flow features Next, the Encoder module and the Decoder module are used to identify human behavior.

The Encoder module will , And position encoding through the self-attention module to enhance video features After that, the enhanced video features Standardize and add unenhanced , and finally use FFN for dimension mapping. Stack N Encoder modules, where N is 6. The i-th Encoder module can be expressed as follows:

,

in, is the positional encoding, is the self-attention module, is the normalization function, is a fully connected layer, and the final output of the N Encoder layers stacked is recorded as .

The Decoder module further refines the general query tensor into optical flow queries and content query ,in . Use the skeleton key point sequence to provide the corresponding coordinates First, coordinate encoding is performed, and the encoding principle is the same as the position encoding principle to obtain .

First, the optical flow query is passed through the self-attention module, and then the deformable attention module is used to update the optical flow query. Therefore, the update process can be expressed as follows:

,

in, Represents deformable attention module.

The sum of the updated optical flow query and content query is updated through the self-attention module to obtain a mixed query, and finally the deformable attention module is used to update the content query. Therefore, the update process can be expressed as follows: , similarly, like the Encoder, the Decoder also needs to be stacked N times, and N is set to 6 in the present invention. The optical flow query and content query of the final output of the Decoder are recorded as and .

Finally, through the multi-head linear change and The confidence scores of the six behaviors are calculated by fusion, which can be expressed as follows:

;

in, Represents The confidence score of each behavior in the frame. In order to determine the behavior represented by the input video sequence, it is necessary to Add along the first dimension and map it to between 0 and 1. Therefore, the confidence score of each behavior needs to be calculated through the Softmax function, which can be specifically expressed as follows:

,

in, is the confidence score of the final output. Select the behavior with the largest confidence score as model output.

3. Indoor temperature and humidity control algorithm

Assumptions is the air temperature, is the relative humidity, is the air velocity, is the carbon dioxide concentration, is the degree of window opening, It is the behavior of the people inside. Divided into three categories, ①, ② are recorded as the first category, with an output score of 0; ③, ④ are recorded as the second category, with an output score of -1; ⑤, ⑥ are recorded as the third category, with an output score of 1. The output score is recorded as .

When there are people in the room, the indoor temperature is kept at 25℃, the humidity is kept at 60%, and the air flow rate is kept at 0.25m/s. For the control of temperature, humidity and air flow rate, the rated values will be used and adjusted according to the influencing factors.

First, the degree of window opening is divided into three categories: (1) Assuming that the window is closed; (2) Think that the indoor air flow rate needs to be increased, but the indoor temperature does not want to change; (3) It is considered to be adjusting the indoor temperature and humidity to the same as the outdoor temperature and humidity.

Therefore, the special condition is the degree of window opening When the temperature, humidity and air conditioning systems are turned off.

The indoor temperature control strategy is as follows:

,

in, is the rated value of temperature regulation, i.e., the change in indoor temperature regulation, which is set to 2 in the present invention; Output the value to which the current indoor temperature should be adjusted for the strategy.

The significance of this strategy is that when the window is closed, the temperature is adjusted according to the type of human behavior. When the window is open, it needs to be regulated according to the human behavior and the degree of window opening. The greater the degree of window opening, the more this variable needs to be increased in order to maintain the indoor temperature.

The control strategy of indoor air velocity is as follows:

,

in, is the rated value of air flow rate regulation, i.e., the change in indoor air flow rate regulation, which is set to 0.005 in the present invention; Output the value to which the current indoor air flow rate should be adjusted for the strategy; It is a logical function. When the content in the brackets is true, it outputs 1, otherwise it outputs 0.

The significance of this strategy is that when the window is closed, the air flow rate is increased when the carbon dioxide concentration is too high. When the window is open, the greater the window opening, the greater the air flow rate it can bring, so the change in air flow rate can be reduced.

The indoor humidity control strategy is as follows:

,

in, is the rated value of indoor humidity regulation, i.e., the change in indoor humidity regulation, which is set to 0.05 in the present invention; Output the value that the current indoor humidity should be adjusted to for the strategy; The variable is the indoor humidity, which is the humidity value measured at the previous moment minus the humidity value measured at the current moment.

The significance of this strategy is: when the window is closed, the indoor humidity is maintained at 0.6. When the window is opened, if the indoor humidity drops, it is increased on the basis of 0.6 to maintain the average indoor humidity at 0.6 over a period of time. Otherwise, it is reduced on the basis of 0.6.

The present invention also provides an indoor HVAC control system, comprising:

An acquisition module is used to acquire indoor video data and environmental parameters;

A window opening degree calculation module is used to input the video data into a trained indoor window segmentation model IWS-Net to obtain a window segmentation mask image, and calculate the indoor window opening degree according to the window segmentation mask image;

An indoor occupant thermal comfort behavior recognition module is used to calculate the inverse optical flow sequence and the human skeleton key point sequence corresponding to the video data, and input the video data, the inverse optical flow sequence and the human skeleton key point sequence into the trained indoor occupant behavior recognition model IDARM to obtain the indoor occupant thermal comfort behavior;

The indoor HVAC control module is used to calculate the indoor HVAC control adjustment strategy according to the obtained indoor window opening degree, indoor occupant thermal comfort behavior and the environmental parameters.

The system can be implemented with reference to the implementation of the aforementioned method.

Furthermore, the video data is obtained by a camera. The camera can be placed on the top of the room to ensure that it can clearly capture the overall indoor and window images, and measure the distance and angle between it and the detection window, as well as its own pitch angle, yaw angle and roll angle.

Specifically, the video acquired by the camera is sampled at a certain frequency to obtain corresponding video frames, and then the video frames are input into the indoor window segmentation model IWS-Net for window segmentation.

Furthermore, the environmental parameters are obtained by an indoor thermal comfort measuring instrument, which mainly obtains indoor air temperature, relative humidity, air flow rate and carbon dioxide concentration.

Optionally, a cyclic full-domain converter RAFT is used to calculate a corresponding inverse optical flow sequence from an RGB video frame sequence acquired from a camera, and a Mediapipe library is used to perform skeleton key point detection on each frame image in the video to form a human skeleton key point sequence.

Furthermore, the operation of calculating the inverse optical flow sequence is as follows:

(1) Select two adjacent frames in the video frame sequence and record them as and , and fill its width and height to multiples of 8;

(2) In and After the exchange, it is used as the input of the pre-trained cyclic full-domain converter RAFT to obtain the inverse optical flow map;

(3) Reverse the direction of the values in the inverse optical flow map and repeat (1) until the video ends.

Furthermore, the operation of calculating the key points of the human skeleton in each frame is as follows:

(1) Separate the video frame sequence frame by frame;

(2) Use Mediapipe in Python to detect the key points of the human skeleton for each frame of the image;

(3) Store the 33 human skeleton key points obtained in (2) and repeat (1) until the end.

The training of the indoor window segmentation model IWS-Net further includes:

Collecting images of indoor windows, preprocessing the images and forming a training set;

Inputting the training set into the constructed indoor window segmentation model IWS-Net for training to obtain model parameters that meet the accuracy requirements;

The model parameters are loaded into the indoor window segmentation model IWS-Net to obtain a trained indoor window segmentation model IWS-Net.

Furthermore, after collecting the images of indoor windows, the images are annotated using labelme, the walls, windows and window edges in the images are segmented, and the rest are classified as background. All data are divided into a test set and a training set according to the ratio.

The operation process of the indoor window segmentation model IWS-Net includes:

A backbone network consisting of five feature extraction modules is used to extract features from the input indoor window image, and the output of each feature extraction module is saved;

Using three attention modules connected sequentially, the output of the last feature extraction module is used to suppress background features and enhance the features of the required segmented parts;

The reconstruction module is used to reconstruct the output of the last attention module and the output of each feature extraction module to obtain mask images of walls, windows, window sills and background.

Furthermore, the reconstruction module includes five upsampling modules, wherein the first The input of each module is The output of the module and The output of the feature extraction module composition.

The window opening degree calculation module is also used to perform the following operations:

After obtaining the mask images of the wall, window, window edge and background, the window image is extracted, and the extracted window image is filtered, threshold segmented and opened in sequence;

Obtaining several minimum inscribed rectangles and vertex coordinates of each rectangle on the processed window image;

The sliding window opening ratio is calculated based on the number of rectangles obtained, the vertex coordinates and the area of each rectangle.

Specifically, the window mask is used to extract the window in the image, and Gaussian filtering is performed to remove noise; then threshold segmentation is used to generate a binary image; then the binary image is opened to remove the fuzzy spots on the binary image. The minimum inscribed rectangle is found on the obtained image, the number of rectangles is counted, and the coordinates of their four vertices are obtained. Finally, the opening ratio of the sliding window is calculated based on the number of rectangles, the coordinates of the four vertices, and the area.

The window opening degree calculation module is also used to perform the following operations:

After obtaining the mask images of the wall, window, window edge and background, the straight line at the upper end of the window edge and the straight line at the lower end of the window edge are obtained by using Hough line detection;

Calculate the slope of two straight lines, and calculate the angle between the two straight lines based on the slope;

The deep neural network model DNN is used to combine the relative distance and angle between the camera and the window, as well as the pitch angle, yaw angle and roll angle of the camera to obtain the actual opening angle of the external casement window.

The training of the indoor personnel behavior recognition model IDARM further includes:

Obtain video data of indoor occupants’ thermal comfort behavior and construct a training set;

Input the training set into the constructed indoor personnel behavior recognition model IDARM for training to obtain model parameters that meet the accuracy requirements;

The model parameters are loaded into the indoor person behavior recognition model IDARM to obtain a trained indoor person behavior recognition model IDARM.

Furthermore, constructing the training set includes collecting videos of multiple subjects performing the following six actions: ① sitting; ② walking; ③ fanning with hands; ④ shaking clothes; ⑤ rubbing hands; ⑥ hugging shoulders, where each video is 3 to 5 seconds long and has a frame rate of 30FPS; the collected videos are divided into training set and test set in a 1:1 ratio.

The operation process of the indoor personnel behavior recognition model IDARM includes:

Extract features from the video frame sequence and the inverse optical flow sequence respectively to obtain video features and inverse optical flow features;

The video features, inverse optical flow features and position encoding are passed through the encoding module Encoder to perform data enhancement to obtain enhanced video features;

The enhanced video features, inverse optical flow features and human skeleton key point sequence are used as the input of the decoding module Decoder, and the optical flow query and content query are output;

The optical flow query and content query are subjected to multi-head linear transformation, and the confidence corresponding to the actions related to thermal comfort behavior is obtained through a fully connected network and a Softmax function.

The indoor personnel thermal comfort behavior recognition module is also used to perform the following operations:

Constructing data tensors , where T is the air temperature, H is the relative humidity, V is the air velocity, C is the carbon dioxide concentration, L is the window opening degree, and A is the behavior of the indoor occupants;

According to the data tensor, an optimal control strategy is obtained by using an indoor temperature and humidity control algorithm;

According to the optimal control strategy, the indoor heating, ventilation and air conditioning control systems are adjusted.

According to one aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the indoor HVAC control method are implemented.

The present invention is described with reference to the flowcharts and/or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims (10)

1. An indoor heating and ventilation control method, characterized by comprising: Obtain indoor video data and environmental parameters. The video data is obtained through a shooting device. The shooting device is placed at the top of the room to ensure that it can clearly capture the overall indoor situation and the window, and measure the distance, angle, and own pitch angle, yaw angle and roll angle; Input the video data into the trained indoor window segmentation model IWS-Net to obtain the window segmentation mask image, and calculate the indoor window opening degree based on the window segmentation mask image; when calculating the indoor window opening degree, use The deep neural network model DNN combines the relative distance and angle between the shooting device and the window, as well as the pitch angle, yaw angle and roll angle of the shooting device, to obtain the actual opening angle of the outer casement window; Calculate the inverse optical flow sequence and human skeleton key point sequence corresponding to the video data, and input the video data, inverse optical flow sequence and human skeleton key point sequence into the trained indoor human behavior recognition model IDARM to obtain the thermal comfort of indoor personnel. Behavior; Based on the obtained indoor window opening degree, indoor occupant thermal comfort behavior and the environmental parameters, the indoor HVAC control adjustment strategy is determined. 2. An indoor HVAC control method according to claim 1, characterized in that the training of the indoor window segmentation model IWS-Net includes: Collect images of indoor windows, preprocess the images and form a training set; Input the training set into the constructed indoor window segmentation model IWS-Net for training, and obtain model parameters that meet the accuracy requirements; Load the model parameters into the indoor window segmentation model IWS-Net to obtain the trained indoor window segmentation model IWS-Net. 3. An indoor HVAC control method according to claim 2, characterized in that the calculation process of the indoor window segmentation model IWS-Net includes: Use a backbone network composed of five feature extraction modules in sequence to extract features from the input indoor window image, and save the output of each feature extraction module; Using three sequentially connected attention modules, the output of the last feature extraction module is used to suppress background features and enhance the features that need to be segmented; The reconstruction module is used to reconstruct the output of the last attention module and the output of each feature extraction module to obtain mask images of walls, windows, window edges and backgrounds. 4. An indoor HVAC control method according to claim 3, characterized in that the method further includes: After obtaining the mask images of the wall, window, window edge and background, extract the window image, and perform filtering, threshold segmentation and opening operations on the extracted window image in sequence; Obtain several minimum inscribed rectangles on the processed window image and the vertex coordinates of each rectangle; Based on the obtained number of rectangles, the vertex coordinates and area of each rectangle, the sliding window opening ratio is calculated. 5. An indoor HVAC control method according to claim 3, characterized in that the method further includes: After obtaining the mask image of the wall, window, window edge and background, use Hough line detection to obtain the straight line at the upper end of the window edge and the straight line at the lower end of the window edge; Calculate the slope of two straight lines, and calculate the angle between the two straight lines based on the slope; Using the deep neural network model DNN, combined with the relative distance and angle between the shooting device and the window, as well as the pitch angle, yaw angle and roll angle of the shooting device, the actual opening angle of the outer casement window is obtained. 6. An indoor HVAC control method according to claim 1, characterized in that the training of the indoor personnel behavior recognition model IDARM includes: Obtain video data of thermal comfort behavior of indoor people and construct a training set; Input the training set into the constructed indoor human behavior recognition model IDARM for training, and obtain model parameters that meet the accuracy requirements; Load the model parameters into the indoor human behavior recognition model IDARM to obtain the trained indoor human behavior recognition model IDARM. 7. An indoor HVAC control method according to claim 6, characterized in that the calculation process of the indoor personnel behavior recognition model IDARM includes: Extract features from the video frame sequence and inverse optical flow sequence respectively to obtain video features and inverse optical flow features; Pass the video features, inverse optical flow features and position coding through the encoding module Encoder to perform data enhancement to obtain enhanced video features; Use the enhanced video features, inverse optical flow features and human skeleton key point sequences as inputs to the decoding module Decoder to output optical flow queries and content queries; The optical flow query and content query are subjected to multi-head linear changes, and the confidence corresponding to the action related to thermal comfort behavior is obtained through the fully connected network and Softmax function. 8. An indoor HVAC control method according to claim 1, characterized in that the method further includes: Construct the data tensor d = [T, H, V, C, L, A], where T is the air temperature, H is the relative humidity, V is the air flow rate, C is the carbon dioxide concentration, L is the window opening degree, and A is actions performed by people in the room; According to the data tensor, the indoor temperature and humidity adjustment control algorithm is used to obtain the optimal control strategy; According to the optimal control strategy, the indoor heating, ventilation and air conditioning control system is adjusted. 9. An indoor heating and ventilation control system, characterized by including: The acquisition module is used to acquire indoor video data and environmental parameters. The video data is acquired through a shooting device. The shooting device is placed on the top of the room to ensure that it can clearly capture the overall indoor picture and the window, and measure and detect the window. distance, angle and its own pitch angle, yaw angle and roll angle; The window opening degree calculation module is used to input the video data into the trained indoor window segmentation model IWS-Net to obtain the window segmentation mask image, and calculate the indoor window opening degree based on the window segmentation mask image; in When calculating the opening degree of indoor windows, the deep neural network model DNN is used, combined with the relative distance and angle between the shooting device and the window, as well as the pitch angle, yaw angle and roll angle of the shooting device, to obtain the actual opening angle of the outer casement window; The thermal comfort behavior recognition module of indoor personnel is used to calculate the backlight flow sequence and human skeleton key point sequence corresponding to the video data, and input the video data, backlight flow sequence and human skeleton key point sequence into the trained indoor personnel Behavior recognition model IDARM can obtain the thermal comfort behavior of indoor personnel; The indoor HVAC control module is used to determine the indoor HVAC control adjustment strategy based on the obtained indoor window opening degree, thermal comfort behavior of indoor personnel and the environmental parameters. 10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program according to any one of claims 1 to 8 is implemented. The steps of the indoor heating and ventilation control method described above.