Artificial Intelligence: Model Evaluation
and Decision Tree Notes
1. Model Evaluation in AI
Model evaluation in AI is the process of assessing how well an AI model (e.g., machine
learning, deep learning) performs in terms of accuracy, reliability, and efficiency. It ensures
that the model is suitable for solving the intended problem and works well on unseen data.
Why Model Evaluation is Important:
- To determine how accurate the model predictions are.
- To check if the model is overfitting (too complex) or underfitting (too simple).
- To compare different AI models for the same task.
- To ensure fairness, robustness, and trustworthiness in AI systems.
Types of Evaluation Based on Task:
1. **Classification Tasks** (e.g., spam detection, disease prediction):
Common Metrics: Accuracy, Precision, Recall, F1-Score, Confusion Matrix.
2. **Regression Tasks** (e.g., predicting prices or temperature):
Common Metrics: Mean Absolute Error (MAE), Mean Squared Error (MSE), R² Score.
3. **Clustering / Unsupervised Tasks** (e.g., customer segmentation):
Common Metrics: Silhouette Score, Dunn Index, Davies–Bouldin Index.
Evaluation Techniques:
1. **Train-Test Split:** Dataset is split into training and testing (e.g., 80% train, 20% test).
2. **K-Fold Cross Validation:** Data is divided into ‘k’ parts. Model is trained on k-1 parts
and tested on the remaining part.
3. **Leave-One-Out Cross Validation (LOOCV):** Each data point is used once as a test set.
4. **Holdout Method:** A portion of the data is held back for final evaluation.
Other Evaluation Considerations in AI:
- **Bias and Fairness:** Does the model treat all groups fairly?
- **Robustness:** How does the model perform on noisy or adversarial data?
- **Interpretability:** Can humans understand how the model makes decisions?
- **Computational Efficiency:** Is it fast and memory-efficient?
2. Decision Tree in Machine Learning
A Decision Tree is a supervised machine learning algorithm used for classification and
regression tasks. It models data by splitting it into branches based on feature values,
resembling a tree structure. It is widely used due to its simplicity, interpretability, and
efficiency.
Structure of a Decision Tree:
- **Root Node:** The top node representing the entire dataset.
- **Internal Nodes:** Represent decision points based on a feature/attribute.
- **Leaf Nodes:** Represent the final classification or prediction result.
- **Edges/Branches:** Connections between nodes representing decisions or splits.
How Decision Trees Work:
1. **Splitting:** The dataset is divided into subsets based on a feature. The decision is made
by choosing the best feature to split the data.
2. **Stopping Criteria:** The tree stops splitting when one of the following happens:
- A predefined depth is reached.
- The number of samples in a node falls below a threshold.
- No further useful feature is available to split.
3. **Prediction:** Once the tree is built, a new data point is passed through the tree, starting
from the root and following the branches, until it reaches a leaf node, where the prediction
is made.
Key Concepts in Decision Trees:
- **Entropy:** A measure of uncertainty or impurity in the dataset.
- **Information Gain:** A metric to choose the best feature for splitting.
- **Gini Impurity:** A criterion to measure impurity in classification tasks.
- **Pruning:** The process of removing branches that have little or no predictive power.
Types of Decision Trees:
- **Classification Trees:** Used for classification tasks (e.g., spam vs. non-spam).
- **Regression Trees:** Used for regression tasks (e.g., predicting house prices).
Advantages of Decision Trees:
- Easy to interpret.
- Non-linear.
- Requires little data preprocessing.
- Can capture interactions between features.
Disadvantages of Decision Trees:
- **Overfitting:** Decision trees tend to create overly complex trees.
- **Instability:** A small change in the data can lead to a completely different tree.
- **Bias towards features with more levels.**
- **Greedy Algorithm:** The tree might not always find the optimal solution.
Applications of Decision Trees:
- Customer Segmentation.
- Medical Diagnosis.
- Financial Predictions.
- Game AI.
