Designing Machine Learning Systems – Interview Notes (Based on Chip Huyen's Book)
Chapter 1: Introduction to Machine Learning Systems
Notes:
• ML systems comprise data, models, and infrastructure.
• Start with understanding the problem domain deeply.
• Not all problems require ML; some can be solved using rule-based systems.
Important Points:
• Data quality and quantity are foundational.
• ML is suited for problems with patterns in data, not where logic alone suffices.
Interview Questions & Answers:
1. What are the key components of a machine learning system?
2. Answer: Data (source, collection, labeling), Models (training, evaluation), and Infrastructure (storage,
deployment, monitoring).
3. How do you determine when to use machine learning?
4. Answer: If the problem has no hard-coded logic and involves pattern recognition, historical data
availability, and probabilistic outcomes.
5. Can you give an example of a case where ML was unnecessary?
6. Answer: If you’re mapping user input to predefined rules, like a calculator app, rule-based logic
suffices.
Chapter 2: Data Engineering Fundamentals
Notes:
• ETL: Extract from source, Transform to usable format, Load into storage.
• Structured (tables), unstructured (text, images), semi-structured (JSON).
Important Points:
• Track data lineage for debugging and audits.
• Data monitoring ensures consistency and freshness.
1
Interview Questions & Answers:
1. What is the ETL process, and why is it important?
2. Answer: It prepares raw data into clean, usable data for ML models. Ensures data integrity, quality,
and proper schema.
3. How do you handle unstructured data?
4. Answer: Use NLP for text (e.g., tokenization), CNNs for images, parsing tools for JSON/XML, and
vector representations for downstream models.
5. What is data lineage and why is it important?
6. Answer: It tracks the origin and transformations of data. Helps in reproducibility and compliance.
Chapter 3: Training Data
Notes:
• Garbage in, garbage out — poor data quality degrades model performance.
• Weak supervision: using heuristic or programmatic labels when manual labeling is costly.
Important Points:
• Class imbalance, noisy labels, and incomplete data are major challenges.
• Sampling strategies like stratified or up/downsampling help with imbalance.
Interview Questions & Answers:
1. What are the challenges associated with training data?
2. Answer: Noisy/incomplete labels, class imbalance, overfitting to irrelevant patterns, and domain
shifts.
3. How can weak supervision improve the labeling process?
4. Answer: It reduces manual effort by using labeling functions or models to infer labels with
reasonable accuracy.
5. How do you handle class imbalance?
6. Answer: Resampling techniques, synthetic data (SMOTE), class weighting, or anomaly detection
framing.
2
Chapter 4: Feature Engineering
Notes:
• Transform raw data into meaningful inputs for models.
• Encode categories (one-hot, embeddings), normalize, impute missing data.
Important Points:
• Use domain knowledge for selecting informative features.
• Tools like SHAP/LIME help with interpretability.
Interview Questions & Answers:
1. What techniques do you use for feature engineering?
2. Answer: Handling missing values, encoding, binning, interaction terms, log transforms, and scaling.
3. How do you measure the importance of features?
4. Answer: Feature importance scores (Gini, gain), SHAP values, permutation importance, model
performance after removal.
5. What’s the trade-off between one-hot encoding and embeddings?
6. Answer: One-hot works for low-cardinality; embeddings scale better for high-cardinality with
learned dense vectors.
Chapter 5: Model Development
Notes:
• Iterative: define objectives, select model, train, evaluate, refine.
• Start simple — baseline models often provide insight.
Important Points:
• Decoupling objectives (e.g., ranking vs. classification) increases flexibility.
• Interpretability matters in regulated or sensitive domains.
Interview Questions & Answers:
1. How do you approach model selection?
2. Answer: Define task type (classification, regression), try baseline, compare metrics (AUC, F1), use
validation.
3
3. What are the advantages of decoupling objectives?
4. Answer: Easier debugging, optimization, and flexibility. For example, using ranking models post
classification.
5. How do you balance accuracy and interpretability?
6. Answer: Use interpretable models like decision trees, or apply post-hoc tools (SHAP) to complex
models.
Chapter 6: Deployment
Notes:
• Online vs. batch inference. Online needs low latency; batch is cost-effective.
• Use CI/CD pipelines and containerization.
Important Points:
• Monitor models for drift and performance.
• Autoscaling ensures availability and cost-efficiency.
Interview Questions & Answers:
1. What are key considerations when deploying a model?
2. Answer: Latency requirements, scaling needs, monitoring, versioning, and rollback strategies.
3. How do you monitor model performance post-deployment?
4. Answer: Use live metrics (accuracy, latency), input distribution tracking, drift detection, and alerting
systems.
5. What is the difference between batch and online inference?
6. Answer: Batch processes large data at intervals, good for non-urgent tasks. Online serves
predictions in real-time.
Chapter 7: Why ML Systems Fail in Production
Notes:
• Distribution shifts, stale data, dependency failures.
• Feedback loops may reinforce model biases.
4
Important Points:
• Differentiate ML failure (data/model) from system failure (infra/API).
• Re-training schedules, anomaly detection are preventive measures.
Interview Questions & Answers:
1. What are common reasons ML systems fail in production?
2. Answer: Data drift, code changes, infrastructure errors, feedback loops, poor monitoring.
3. How do you detect and address data distribution shifts?
4. Answer: Use statistical tests (KS-test), embedding comparisons, performance drop indicators, retrain
triggers.
5. What is a feedback loop in ML and how can it harm performance?
6. Answer: When model output affects future training data (e.g., recommendations), leading to biased
or overfit models.
End of Notes. Prepared for interviews in ML engineering, data science, and applied AI roles.
