Lecture 4: Data-driven maintenance

Inverse Modeling + Benchmarking

Building Energy Management and Optimization
CIVE 5708

Araz Ashouri PhD

Adjunct Research Professor
Civil & Environmental Engineering
Agenda
Applied theory
• Multiple linear regression models (last week)
• Artificial neural networks
Matlab example
Conceptual background
• Benchmarking
Case study

2
Multiple linear regression model

3
Artificial neural network model

4
Why ANNs?
• Pros
• Learn and generalize data as long as the dataset is
comprehensive enough
• Fault and noise tolerant
• Computationally fast
• Very accurate
• Cons
• Hard to comprehend by non-technical decision makers
• Operators, energy managers can easily interpret linear
relationships with inputs / outputs
• ANNs parameter weights are not suitable to make direct
interpretations
• Hard to visualize

5
Conceptual example
• Say you want to predict the heating demand of a
building cluster. And, there are four variables that
affect your decision:
1. Outdoor temperature
2. Internal gains from occupants, equipment, and lighting
3. Wind speed
4. Solar irradiance

We assume that these are the only variables that

affect our prediction

6
 We write these quantities as input variables xi:
1. Outdoor temperature x1
2. Internal gains from occupants, equipment, and lighting x2
3. Wind speed x3
4. Solar irradiance x4

Hidden node with sigmoid activation function:

𝑥1 𝑏1
𝑤1
1
𝑤2 𝑥1 𝑤1 + 𝑥2 𝑤2 + 𝑥3 𝑤3 + 𝑥4 𝑤4 + 𝑏1 1 + 𝑒− 𝑥1 𝑤1 +𝑥2 𝑤2+𝑥3 𝑤3+𝑥4 𝑤4+𝑏1
𝑥2 ∑ sig

𝑤3
𝑥3 𝑤4
𝑥4 7
Sigmoid / Logistic Activation Function

http://blog.hackerearth.com/wp-content/uploads/2017/01/sigmoid.gif

8
https://qph.ec.quoracdn.net/main-qimg-01c26eabd976b027e49015428b7fcf01?convert_to_webp=true 9
 1

𝑥1 𝑏1
𝑤1

𝑤2  1 
𝑥2 ∑ sig lin w5    b2
 x1w1  x2 w2  x3w3  x4 w4 b 
 1 e 
𝑤3
𝑥3 𝑤4 𝑏2

𝑥4 1

10
 1. Feedforward
2. Four inputs
3. Seven parameters
4. One hidden and one output node
5. Hidden node with sigmoid activation
6. Output node with linear activation

𝑥1 𝑏1
𝑤1

𝑤2  1 
𝑥2 ∑ sig lin w5 
 x1w1  x2 w2  x3w3  x4 w4 b1 
  b2
 1 e 
𝑤3
𝑥3 𝑤4 𝑏2

𝑥4 1

11
e.g., six inputs, two hidden layers

Credit: Taylor B. Arnold

12
 13
Credit: Taylor B. Arnold
The effect of model complexity: MLR vs. ANN

Cooling Heating

Gunay, Burak, Weiming Shen, and Guy Newsham. "Inverse blackbox modeling of the heating and
cooling load in office buildings." Energy and Buildings 142 (2017): 200-210. 14
Gunay, Burak, Weiming Shen, and Guy Newsham. "Inverse blackbox modeling of the heating and
cooling load in office buildings." Energy and Buildings 142 (2017): 200-210. 15
Credit: Taylor B. Arnold

16
Credit: Taylor B. Arnold

17
 Training ANNs given a dataset
• We now have an architecture that describes a neural
network, but how do we learn the weights and bias terms in
the model given a set of training data?

19
Cost function
• The primary set-up for learning neural networks is to define
a cost function that measures how well the network
predicts outputs (Arnold, 2016)
• The goal is to then find a set of weights and biases that
minimizes the cost.

20
 y Measured heating
energy use
Objective:
Minimize the difference between these two
Predicted heating
ŷ energy use
1

𝑥1 𝑏1
𝑤1

𝑤2  
𝑥2 1
∑ sig lin w5 
 x1w1  x2 w2  x3w3  x4 w4 b1 
  b2
 1 e 
𝑤3
𝑥3 𝑤4 𝑏2

𝑥4 1

21
J   yˆ  y 
2

22
  dJ dJ dJ dJ dJ dJ 
J   , , , , , 
 dw1 dw2 dw3 dw4 db1 db2 

 Learning rate

24
https://rasbt.github.io/mlxtend/user_guide/general_concepts/gradient-
optimization_files/ball.png

26
Learning occupants’ light switch
preferences with online ANNs

27
https://www.analyticsvidhya.com/blog/2018/01/online-
learning-guide-text-classification-vowpal-wabbit-vw/
28
Gunay, H. B., O'Brien, W., Beausoleil-Morrison, I., & Gilani, S. (2017). Development and implementation of
an adaptive lighting and blinds control algorithm. Building and Environment, 113, 185-199.

29
Principles for model selection and
evaluation are the same for all models
1. Employ stepwise regression (e.g., systematically add or
subtract variables)
• For each input combination, split the data at least in two
groups (training and testing)
• Train the model with training set at different level of
complexity (e.g., number of hidden layers)
• Compute error metrics with the testing set (e.g., RMSE,
MAE, R2)
2. Select the model inputs and complexity that gives the
highest accuracy
3. Carry out residual analysis on the best model

30
Matlab Example

31
clc; clear; close all;

[num,txt,raw] =
xlsread('SpaceHeatingCoolingLoads.xlsx','Cooling Load');

time = datenum(txt(2:end,1));

tOut = num(:,1); % outdoor temperature (degC)

sWind = num(:,2); % wind speed (m/s)
qSol = num(:,3); % solar irradiance (W/m2)
qClg = num(:,end); % cooling energy use intensity (W/m2)

hiddenNodes = 5;

%% 0. Define the model structure first

net = fitnet(hiddenNodes); % example here is for 5 nodes and 1
hidden layer
view(net);

32
%% 1. Forward stepwise regression

% 1.1. first round

x = tOut';
y = qClg';
[net,tr] = train(net,x,y);
xtest = x(:,tr.testInd); % inputs retained for validation
ytest = y(:,tr.testInd); % inputs retained for validation
yp = net(xtest); % make predictions with the trained model
residuals = ytest - yp; % residuals
RMSE(1) = sqrt(mean(power(residuals,2))); % RMSE error
mdl1_predict = net(x);

x = sWind';
y = qClg';
[net,tr] = train(net,x,y);
xtest = x(:,tr.testInd); % inputs retained for validation
ytest = y(:,tr.testInd); % inputs retained for validation
yp = net(xtest); % make predictions with the trained model
residuals = ytest - yp; % residuals
RMSE(2) = sqrt(mean(power(residuals,2))); % RMSE error
mdl2_predict = net(x);

x = qSol';
y = qClg';
[net,tr] = train(net,x,y);
xtest = x(:,tr.testInd); % inputs retained for validation
ytest = y(:,tr.testInd); % inputs retained for validation
yp = net(xtest); % make predictions with the trained model
residuals = ytest - yp; % residuals
RMSE(3) = sqrt(mean(power(residuals,2))); % RMSE error 33
mdl3_predict = net(x);
% plot models for visual inspection
plot(time,qClg,'k'); % measured
hold on
plot(time,mdl1_predict,'b');
hold on
plot(time,mdl2_predict,'r');
hold on
plot(time,mdl3_predict,'g');
dateFormat = 3; %
https://www.mathworks.com/help/matlab/ref/datetick.html#
btpmlwj-1-dateFormat
datetick('x',dateFormat,'keepticks')
close all

34
% 1.2. second round
net = fitnet(hiddenNodes); % redefine structure
x = [tOut,qSol]';
y = qClg';
[net,tr] = train(net,x,y);
xtest = x(:, tr.testInd); % inputs retained for validation
ytest = y(:, tr.testInd); % inputs retained for validation
yp = net(xtest); % make predictions with the trained model
residuals = ytest - yp; % residuals
RMSE(4) = sqrt(mean(power(residuals,2))); % RMSE error
mdl4_predict = net(x);

x = [tOut,sWind]';
y = qClg';
[net,tr] = train(net,x,y);
xtest = x(:, tr.testInd); % inputs retained for validation
ytest = y(:, tr.testInd); % inputs retained for validation
yp = net(xtest); % make predictions with the trained model
residuals = ytest - yp; % residuals
RMSE(5) = sqrt(mean(power(residuals,2))); % RMSE error
mdl5_predict = net(x);

% plot models for visual inspection

plot(time,qClg,'k'); % measured
hold on
plot(time,mdl4_predict,'b');
hold on
plot(time,mdl5_predict,'r');
dateFormat = 3;
datetick('x',dateFormat,'keepticks')
close all 35
% 1.3. third round
net = fitnet(hiddenNodes); % redefine structure
x = [tOut,qSol,sWind]';
y = qClg';
[net,tr] = train(net,x,y);
xtest = x(:, tr.testInd); % inputs retained for validation
ytest = y(:, tr.testInd); % inputs retained for validation
yp = net(xtest); % make predictions with the trained model
residuals = ytest - yp; % residuals
RMSE(6) = sqrt(mean(power(residuals,2))); % RMSE error
mdl6_predict = net(x);

% plot models for visual inspection

plot(time,qClg,'k'); % measured
hold on
plot(time,mdl6_predict,'b');
dateFormat = 3; %
https://www.mathworks.com/help/matlab/ref/datetick.html#btpmlwj-1-
dateFormat
datetick('x',dateFormat,'keepticks')
close all

36
Benchmarking

37
 Objectives of benchmarking

• Establish standardized building performance metrics

• Establish procedures for determining these metrics
• Determine which performance metrics are of greatest value
to the building community
• Determine the most reliable ways to measure and report
them

If you ask 10 energy professionals to give you the energy performance of a building,
you will probably get 10 different answers. (Deru and Torcellini, 2005)

38
What is a metric?
• A metric is a standard definition of any measurable quantity
• A performance metric is a standard definition of a
measurable quantity that indicates some aspect of
performance
• Other terms used with a similar meaning
Performance indicator
Performance index
Benchmarking

39
 Characteristics of a performance metric
• Directly or indirectly measurable
• Have a clear definition
• Indicate progress toward a performance goal
• Answer specific questions about the performance

Performance objective: “minimize primary energy consumption”

Performance goal: “reduce primary energy consumption for building operations by
30% compared to that consumed in 1990.”

40
Who Uses Metrics and How Are They Used?

41
Performance metrics
• Thermal complaint frequencies
• Frequency of thermostat overrides
• Absenteeism Comfort, productivity,
satisfaction metrics
• Sick days
• Space utilization and occupancy
• Energy use Environmental
• Water use metrics

• Work-orders Economics
• Cost of operation
• …

42
ASHRAE PMP
• ASHRAE’s Performance Measurement Protocols (PMPs) for
Commercial Buildings
• Technical Committees (TCs 7.6, 7.9, 4.7) develop the content
• The protocols identify
• what to measure
• how it is to be measured (instrumentation and spatial resolution)
• how often it is to be measured for inclusion in the building’s
operation and maintenance plan

45
Hunn et al., 2012
46
Basic Level Benchmarking – energy / water use
• Compile annual whole-building energy / water use and cost
• 12 consecutive months of utility data.
• Calculate annual energy use and cost indices (per unit of
gross floor area), by site (or source as well), preferably
normalized for weather and occupancy
• Then, compare to peer buildings

47
Basic Level Benchmarking – IEQ
• Evaluate complaint logs
• Conduct occupant and operator surveys of satisfaction with overall thermal
comfort and the impact on self-reported job performance
• https://www.cbe.berkeley.edu/research/survey.htm
• Spot measurements for temperature, relative humidity, mean radiant
temperature, air speed, illuminance – sure you take measurements, then
what is good and how about temporal / spatial variations.

Hunn et al., 2012

48
 IAQ Survey Before and After Retrofits

50
Hunn et al., 2012
Acoustic Comfort Survey Before and After Retrofits

51
Hunn et al., 2012
• Spatial normalization
• Energy use intensity (kWh per m2 of rentable floor area)
• Temporal normalization
• Annual work-order intensity (Number of work-orders
per year)
• User-centric normalization
• Energy use per occupant (5 kWh per occupant)

53
 Metric per
occupant - yr
x

Occupant
Duration (yr)

Metric per
m2-yr
54
O'Brien, William, et al. "On occupant-centric building performance
metrics." Building and Environment 122 (2017): 373-385.

57
Occupant-centric metrics
O'Brien, William, et al. "On occupant-centric building performance
metrics." Building and Environment 122 (2017): 373-385.

60
Occupant-centric metrics
O'Brien, William, et al. "On occupant-centric building performance
metrics." Building and Environment 122 (2017): 373-385.

61
Occupant-centric metrics
O'Brien, William, et al. "On occupant-centric building performance
metrics." Building and Environment 122 (2017): 373-385.

62
O'Brien, William, et al. "On occupant-centric building performance metrics." Building and Environment 122 (2017): 373-385.

Room number Occupied time Time with lights on LUR

(hours) (hours)
1 153 63 0.41
2 610 111 0.18
3 786 1076 1.37
4 634 219 0.35
5 519 91 0.18
6 25 3 0.13
7 341 314 0.92
8 431 403 0.94

63
Carleton work-order study
• Computerized maintenance management system (CMMS)
databases were mined to extract performance benchmarks
• 87,865 work-order descriptions in Carleton University in 7
years
• A wide-range of activities
• HVAC maintenance
• Administrative services
• Carpentry and locksmith
• Furniture moving
• Plumbing
• Water treatment and vehicle maintenance
• Painting and custodial services
• Interesting information about top HVAC failure modes and
their occurrence frequencies are buried within work-orders

65
66
Per equipment breakdown of failures

67
68
Benchmarking failure rates

69
70
Dutta et al. (2019)

72
 73
Dutta et al. (2019)
Summary
• Continued our discussion on regression analysis
• Theory behind artificial neural network models
• Matlab example
• Benchmarking
• Data sources for benchmarking
• Common performance indices
• Case studies on benchmarking

76
Bibliography
Deru, Michael, and Paul Torcellini. Performance Metrics Research Project-Final Report. No.
NREL/TP-550-38700. National Renewable Energy Laboratory (NREL), Golden, CO., 2005.
Newsham, Guy R., Jennifer A. Veitch, and Yitian Hu. "Effect of green building certification on
organizational productivity metrics." Building Research & Information (2017): 1-12.
O'Brien, William, et al. "On occupant-centric building performance metrics." Building and
Environment 122 (2017): 373-385.
Hunn, Bruce, et al. Measuring Commercial Building Performance (2012) ASHRAE Journal
(2012)
Gunay, Burak, and Weiming Shen. "Connected and Distributed Sensing in Buildings:
Improving Operation and Maintenance." IEEE Systems, Man, and Cybernetics Magazine 3.4
(2017): 27-34.
Gunay, Burak, et al. “A preliminary study on text mining operator logbooks to develop a fault-
frequency model” ASHRAE Transactions 124 (2018).
Gunay, Burak, Weiming Shen, and Guy Newsham. "Inverse blackbox modeling of the heating
and cooling load in office buildings." Energy and Buildings 142 (2017): 200-210.

77