Decision Trees
Decision Trees are supervised learning algorithms that can be applied to both classification and
regression problems. They work by recursively splitting the dataset into subsets based on feature
values, forming a tree-like structure where each internal node corresponds to a feature test and
each leaf node corresponds to a prediction.
The main principle is to identify the feature and split point that best separates the data. Common
impurity measures used are: - Gini Index: Measures the probability of incorrect classification. -
Entropy / Information Gain: Based on information theory to measure purity. - Variance Reduction:
Used in regression trees to minimize prediction error.
Advantages: - Easy to visualize and interpret. - Works with both numerical and categorical data. -
Handles missing values relatively well.
Disadvantages: - High variance; sensitive to small changes in data (overfitting). - Can create overly
complex trees if not pruned. - Bias towards features with more levels.
Applications include medical diagnosis (decision support), customer segmentation, fraud detection,
and recommendation systems. Decision Trees also serve as the foundation for advanced ensemble
models like Random Forests and Gradient Boosted Trees.
