⌘K
ENES
GitHub
conceptLearning Foundations~1 min readUpdated 2026-06-07#foundations#features#representations#embeddings

Features, representations & the curse of dimensionality

A model never sees raw reality; it sees a representation — the numeric features you feed it. The quality of that representation usually matters more than the choice of algorithm. "Garbage in, garbage out" is really a statement about features.

Features vs learned representations

  • Hand-engineered features — humans decide what's relevant (word counts, ratios, domain signals). Classical ML lives or dies here.
  • Learned representations — the model discovers useful features itself. This is what "deep" in deep learning means: each layer builds a richer representation than the last. Embeddings are the canonical example — dense vectors where geometric closeness encodes semantic similarity.

The shift from hand-engineered to learned features is the central story of modern AI, and it's why embeddings power retrieval.

The curse of dimensionality

As the number of features grows, intuition built in 2-D and 3-D breaks down:

  • Sparsity — data points become isolated; volume grows exponentially, so any fixed dataset becomes vanishingly sparse. You need exponentially more data to cover the space.
  • Distance concentration — in very high dimensions, the distance between the nearest and farthest points becomes nearly equal, so "nearest neighbor" loses meaning. (This is why naive distance metrics struggle, and why embeddings and approximate nearest-neighbor indexes are engineered carefully.)
  • Overfitting risk rises with more features relative to samples.

The blessing on the other side

Real high-dimensional data usually lives near a lower-dimensional manifold — images of faces don't fill all of pixel space, they cluster on a thin sheet within it. Representation learning works by finding that manifold. Dimensionality reduction (PCA, UMAP) exploits the same fact.

Practical takeaways

  • More features is not better; relevant features are. Prune, select, or learn.
  • When distances misbehave, suspect dimensionality before the algorithm.
  • A good representation makes a simple model strong; a bad one defeats a complex one.

Connects to: inductive bias · representation learning · embeddings & retrieval

Related notes

Learning FoundationsEvaluation metrics & what they hideAccuracy lies on imbalanced data. Precision, recall, F1, ROC-AUC — what each measures, and how to pick the metric that matches the cost of being wrong.Learning FoundationsGeneralization, overfitting & the bias–variance tradeoffA model is only useful if it works on data it never saw. Overfitting, underfitting, and the tradeoff that governs both.Learning FoundationsHow learning works: loss, objective, and ERMEvery model that "learns" is minimizing a loss function over data. Understand that loop and most of ML stops being magic.Learning FoundationsInductive bias & the no-free-lunch ideaLearning is impossible without assumptions. Inductive bias is the set of assumptions a model makes — and why architecture choices matter.