⌘K
ENES
GitHub
conceptStatistical Machine Learning~1 min readUpdated 2026-06-07#machine-learning#regression#classification#baseline

Linear & logistic regression

These are the first models to reach for and the last to fully retire. They're fast, interpretable, hard to overfit, and — crucially — a single neuron in a neural network is one of them. Understand these and deep learning is less mysterious.

Linear regression — predict a number

Fit a weighted sum of features to predict a continuous value: ŷ = w·x + b. Train by minimizing squared error via gradient descent (or a closed-form solution). Each weight is readable: "holding others fixed, +1 here moves the prediction by wᵢ."

Assumptions worth knowing: roughly linear relationship, errors not wildly heteroscedastic, features not perfectly collinear. Break them badly and the coefficients become unstable or misleading.

Logistic regression — predict a probability

For classification, wrap the same linear score in a sigmoid to squash it into [0, 1]: a calibrated-ish probability. Train with cross-entropy loss. Despite the name, it's a classifier. A threshold (default 0.5, but tune it — see metrics) turns the probability into a decision.

A logistic regression is exactly the output layer of a classification neural network. The "deep" part just learns better features to feed it.

Why they remain the default baseline

  • Fast to train and predict, even on large data.
  • Interpretable — coefficients are inspectable, which matters for trust and debugging.
  • Hard to overfit with regularization, great when data is limited.
  • A strong score here means a complex model must earn its added risk.

Pitfall

Linear models need sensible features: scale them, encode categoricals, and add interaction/nonlinear terms by hand, since the model can't discover them on its own (that's the linear inductive bias).

Connects to: regularization · gradient descent · the neuron inside a net

Related notes

Statistical Machine LearningThe supervised learning workflow, end to endThe repeatable loop from problem framing to a shipped model — and why a dumb baseline is the most valuable first model you build.Statistical Machine LearningClustering & PCA: learning without labelsThe two most useful unsupervised tools — finding groups with k-means, and compressing/visualizing data with PCA (and when to reach for UMAP).Statistical Machine LearningCross-validation done rightHow to estimate performance without wasting data or fooling yourself — k-fold, stratified, grouped, time-series, and the nested CV that tuning requires.Statistical Machine LearningDecision trees & ensembles (RF, gradient boosting)Why gradient-boosted trees are still the default winner on tabular data, and how bagging and boosting tame a single tree's instability.