Learning Foundations
Foundations hold the concepts that make every later branch easier. These notes are model-agnostic: they apply to a logistic regression and to a frontier LLM alike, because both are systems that learn a function from data and then have to behave on data they have never seen.
If you only internalize one thing here: machine learning is the discipline of generalizing from finite examples to unseen cases — everything else is detail in service of that.
Mental model
A learning system turns finite observations into behavior on unseen inputs. Its core contract is therefore not training fit but generalization under explicit assumptions about data, loss, inductive bias, and deployment distribution.
Roadmap: core concepts
- How learning works: loss, objective, and ERM
- Generalization, overfitting & the bias–variance tradeoff
- Inductive bias & the no-free-lunch idea
- Types of learning: supervised, unsupervised, self-supervised, RL
Data & uncertainty
- Train/validation/test splits & data leakage
- The data distribution & distribution shift
- Probability & uncertainty for ML
- Features, representations & the curse of dimensionality
The math you actually use
- Linear algebra intuition: the dot product as similarity
- Gradient descent: how models actually learn
- Information theory: entropy, cross-entropy & KL
Judging models
Systems view
Knowing the limits
- When not to use AI recognizes where rules, humans, or simpler software beat a model.
Connects to: Mathematics for AI · Statistical Machine Learning · Evaluation and Measurement
Core sources
- An Introduction to Statistical Learning — accessible statistical-learning theory with exercises and labs.
- Understanding Machine Learning — formal treatment of learnability, generalization, and optimization.
- Google Rules of ML — production-oriented decision rules and system failure patterns.
- The ML Test Score — an operational rubric for production readiness and technical debt.