Decision tree regression.
Using Decision Trees for Regression: A Detailed Explanation
The goal of using a decision tree for regression is to predict a continuous numerical output (the
dependent variable) based on one or more input features (the independent variables). The tree
achieves this by recursively partitioning the data space into smaller regions and fitting a simple
prediction (typically the mean of the target variable within that region) within each region.
Here's a breakdown of the process:
1. Data Preparation:
Collect and Prepare Data: Gather your dataset with independent features and the continuous
dependent variable. Handle missing values and encode categorical features if necessary
(though tree-based methods can often handle categoricals directly).
Split the Data: Divide your dataset into training, validation (optional but recommended for
tuning), and test sets. The training set is used to build the tree, the validation set to tune
hyperparameters and prevent overfitting, and the test set to evaluate the final model's
performance on unseen data.
2. Choosing the Splitting Criterion:
At each internal node of the tree, the algorithm must decide which feature to split on and what
the split point should be. For regression trees, the goal of the split is to reduce the variance or
the Mean Squared Error (MSE) of the target variable in the resulting child nodes. Common
splitting criteria include:
Variance Reduction: The algorithm selects the split that leads to the largest decrease in the
variance of the target variable across the child nodes. The variance at a node measures the
spread of the target variable values within that node.
Mean Squared Error (MSE) Reduction: The algorithm chooses the split that minimizes the
weighted average of the MSE in the child nodes. The MSE at a node is the average of the
squared differences between the actual target values and the mean target value in that node.
Mean Absolute Error (MAE) Reduction: Similar to MSE, but uses the absolute differences
instead of squared differences.
3. Building the Tree (Recursive Partitioning):
The decision tree is built using a greedy and recursive process:
Start at the Root Node: All the training data is in the root node.
Find the Best Split: For the current node, the algorithm iterates through each feature and
considers all possible split points (thresholds for numerical features, subsets for categorical
Vipin V. Palande
NMPL
features). For each potential split, it calculates the reduction in the chosen impurity measure
(e.g., variance or MSE). The split that yields the greatest reduction is selected as the best split
for that node.
Create Child Nodes: The best split divides the data in the current node into two or more child
nodes based on the split condition.
Repeat the Process: The splitting process is recursively applied to each child node. This
continues until a stopping criterion is met.
4. Defining Stopping Criteria:
The recursive splitting must stop to prevent the tree from becoming overly complex and
overfitting the training data. Common stopping criteria include:
Maximum Depth of the Tree: Limits the number of levels in the tree.
Minimum Number of Samples at a Node: Stops splitting if a node contains fewer than a
specified number of data points.
Minimum Number of Samples in a Leaf Node: Ensures that each leaf node has a minimum
number of data points.
Minimum Impurity Reduction: Stops splitting if the improvement in the impurity measure is
below a certain threshold.
5. Making Predictions:
Once the regression tree is built, you can use it to predict the target variable for new, unseen
data points:
Traverse the Tree: Starting from the root node, for each feature in the input data point, follow
the branch that corresponds to the feature's value at the current node.
Reach a Leaf Node: Continue traversing until you reach a leaf node.
Output the Prediction: The predicted value for the input data point is the average (or sometimes
median) of the target variable values of the training samples that ended up in that leaf node.
Example
Using decision tree regression predict efficiency when engine temperature =135
Sr no. Engine Temperature (0C) Efficiency (%)
1 50 0.4
2 100 0.6
3 150 0.55
4 200 0.7
Vipin V. Palande
NMPL
let's predict the efficiency at a temperature of 135 °C using the best first split of a decision
tree based on the provided data. We'll use the variance reduction criterion to determine the
best split.
Vipin V. Palande
NMPL
Therefor using the first best split of a decision tree based on variance reduction, the predicted
0.6+0.55+0.7
efficiency at an engine temperature of 1350C = = 0.6167 = 61.67%
3
6. Further Splitting
We check if the tree can be further split. The left branch (Temperature <= 75°C) cannot be
split as it has only one data point. The right branch (Temperature > 75°C) can be split further.
Vipin V. Palande
NMPL
We calculate the variance reduction for potential splits within this branch and find that
splitting the right branch at 175°C results in a variance reduction of 0.002222
7. Second Split at 175°C
The right branch (Temperature > 75°C) is split at 175°C, resulting in the following tree:
If Engine Temperature (℃) <= 75:
Predict Efficiency = 0.4
Else (Engine Temperature (℃) > 75):
If Engine Temperature (℃) <= 175:
Predict Efficiency = 0.575
Else (Engine Temperature (℃) > 175):
Predict Efficiency = 0.7
8. Final Split at 125°C
The tree can be further split at 125°C.
If Engine Temperature (℃) <= 75:
Predict Efficiency = 0.4
Else (Engine Temperature (℃) > 75):
If Engine Temperature (℃) <= 125:
Predict Efficiency = 0.6
Else (Engine Temperature (℃) > 125):
If Engine Temperature (℃) <= 175:
Predict Efficiency = 0.55
Else (Engine Temperature (℃) > 175):
Predict Efficiency = 0.7
9. Prediction at 135°C
• After the first split at 75°C: The predicted efficiency at 135°C is 0.6167%.
• After the second split at 175°C: The predicted efficiency at 135°C is 0.575%.
• After the final split at 125°C: The predicted efficiency at 135°C is 0.55%.
References
• "Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow" by Aurélien
Géron.
• Gemini AI. (2025, April 10).
Vipin V. Palande
NMPL
