Computation and Autodiff
A mathematical function does not train itself. It must be represented as array operations, executed in a defined order, differentiated, scheduled on hardware, and observed under finite precision.
Mental model
The forward pass creates values and dependencies. Reverse-mode automatic differentiation traverses those dependencies backward, multiplying local derivatives and accumulating every path into shared parameters. Tensor libraries add shape rules, vectorized kernels, device placement, memory management, and synchronization around that mechanism.
Current foundation notes
Candidate note roadmap
arrays-tensors-shapes-and-strides— storage, views, broadcasting, strides, and contiguity.vectorization-and-batched-computation— replace scalar loops with array programs and reason about batch axes.compute-graphs-and-execution-models— eager, traced, compiled, static, and dynamic execution.reverse-mode-autodiff-engine— nodes, local derivatives, topology, accumulation, and lifecycle.forward-mode-jacobians-and-jvps— when forward mode is the cheaper derivative program.floating-point-and-mixed-precision— formats, rounding, loss scaling, and numerical failure.randomness-seeds-and-reproducibility— RNG state, nondeterministic kernels, and environment capture.gpus-kernels-and-parallel-computation— threads, warps, memory hierarchy, launch overhead, and arithmetic intensity.
What frameworks hide
Autograd frameworks correctly automate derivative bookkeeping, but they cannot decide whether an objective is meaningful, a tensor shape is semantically correct, an in-place mutation invalidates a graph, or a nondeterministic kernel undermines an experiment. This branch makes those boundaries explicit.
Connects to: Mathematics for AI · Deep Learning · Inference Systems
Core sources
- PyTorch Autograd mechanics — precise behavior of the production reverse-mode engine.
- JAX automatic differentiation — transformations, Jacobian-vector products, and composable autodiff.
- micrograd — compact reference implementation to study, not copy blindly.
- What Every Computer Scientist Should Know About Floating-Point Arithmetic — foundational analysis of finite-precision behavior.