⌘K
ENES
GitHub
conceptLearning Foundations~1 min readUpdated 2026-06-07#foundations#inductive-bias#generalization

Inductive bias & the no-free-lunch idea

Infinitely many functions fit any finite set of points. To pick one and expect it to generalize, a learner must prefer some explanations over others. That set of built-in preferences is its inductive bias.

No free lunch, briefly

The no-free-lunch theorems say that, averaged over all possible problems, no learning algorithm beats any other. The catch: real-world problems are not uniformly random — they have structure (smoothness, locality, compositionality). A model wins by having a bias that matches the structure of your data. There is no universally best model, only a best match for a problem.

Inductive bias is baked into the architecture

Model Inductive bias
Linear models the relationship is (roughly) linear
CNNs locality + translation invariance (nearby pixels relate)
RNNs sequential order matters; recent context dominates
Transformers any token can attend to any other; weak positional prior
Trees axis-aligned, piecewise-constant splits

This is why CNNs dominate images and transformers dominate language: their biases fit the domain. It is also why transformers are data-hungry — a weaker prior must be compensated with more examples.

Practical consequences

  • Choosing a model is choosing a bias. Match it to what you know about the data.
  • Strong, correct priors mean you need less data. Weak priors mean you need more (and more compute). This is the core trade behind "just scale it."
  • Feature engineering and data augmentation are ways to inject bias by hand.

Pitfall

A wrong-but-confident bias generalizes badly in a way no amount of tuning fixes — e.g., forcing a linear model onto a fundamentally nonlinear relationship.

Connects to: bias–variance · features & representations · deep learning

Related notes

Learning FoundationsGeneralization, overfitting & the bias–variance tradeoffA model is only useful if it works on data it never saw. Overfitting, underfitting, and the tradeoff that governs both.Learning FoundationsEvaluation metrics & what they hideAccuracy lies on imbalanced data. Precision, recall, F1, ROC-AUC — what each measures, and how to pick the metric that matches the cost of being wrong.Learning FoundationsFeatures, representations & the curse of dimensionalityModels don't see the world — they see features. How representation quality decides everything, and why high dimensions are weird.Learning FoundationsHow learning works: loss, objective, and ERMEvery model that "learns" is minimizing a loss function over data. Understand that loop and most of ML stops being magic.