⌘K
ENES
GitHub
conceptStatistical Machine Learning~1 min readUpdated 2026-06-07#machine-learning#feature-engineering#preprocessing

Feature engineering

In classical ML, the features decide the ceiling and the algorithm just approaches it. Deep learning automates this for text and images, but for tabular data, thoughtful features still beat a fancier model. It's where domain knowledge enters the model.

The everyday toolkit

  • Scaling — standardize/normalize numeric features so distance- and gradient-based models behave (kNN, SVM, linear, neural nets). Trees don't care.
  • Encoding categoricals — one-hot for low cardinality; target/ordinal/hashing or embeddings for high cardinality.
  • Handling missing values — impute (mean/median/model) and often add a "was-missing" flag, since missingness itself can be signal.
  • Transforms — log/Box-Cox for skewed values; binning; date → (dayofweek, month, is_holiday).
  • Interactions & ratios — price_per_sqft, clicks/impressions. Linear models can't invent these; you must add them (a way to inject inductive bias by hand).

Good features share traits

  • Relevant to the target (carry signal), not just available.
  • Available at prediction time — if a feature is only known after the outcome, it's target leakage.
  • Stable — not prone to drift that breaks the model later.

The leakage trap

Any feature learned from data (scaler stats, target encoding, TF-IDF) must be fit on the training fold only, inside a pipeline. Target encoding is especially dangerous: computing category means over all rows leaks the label.

More features is not better. Relevant, leak-free, available-at-serving features are. Prune aggressively; each weak feature adds variance.

Connects to: pipelines & leakage · representations · regularization for selection

Related notes

Statistical Machine LearningClustering & PCA: learning without labelsThe two most useful unsupervised tools — finding groups with k-means, and compressing/visualizing data with PCA (and when to reach for UMAP).Statistical Machine LearningCross-validation done rightHow to estimate performance without wasting data or fooling yourself — k-fold, stratified, grouped, time-series, and the nested CV that tuning requires.Statistical Machine LearningDecision trees & ensembles (RF, gradient boosting)Why gradient-boosted trees are still the default winner on tabular data, and how bagging and boosting tame a single tree's instability.Statistical Machine LearningError analysis: reading your model's mistakesA single aggregate score hides where a model fails. Slicing, learning curves, and confusion patterns turn errors into a roadmap.