Tensor Kernels and Inference from Scratch

An inference runtime is built from representations, operator semantics, kernels, memory planning, scheduling, and backend dispatch. This branch makes the smallest useful slice of that machinery explicit by executing a pretrained model with code whose layouts, loops, intermediate buffers, and numerical tolerances are all inspectable.

Introduction and Mental Model

A dense tensor is not just a nested array. It is a typed buffer plus a rank, shape, strides, layout convention, and valid address range. An operator maps one or more such logical tensors to outputs according to mathematical and broadcasting rules. A kernel is one concrete loop organization that realizes those rules for a dtype and layout. A model forward pass schedules operators and manages the lifetimes of weights, activations, and scratch buffers.

Correctness has layers: shape correctness, memory safety, operator semantics, and numerical agreement. Floating-point implementations need not be bit-identical to be acceptably equivalent, but tolerance is not permission to ignore instability. Stable softmax, accumulation order, exceptional values, and explicit reference outputs turn “it looks plausible” into evidence.

The AI Atlas explains model architectures, training, and inference optimization at a general level. This branch does not duplicate that curriculum. It implements the execution substrate in Wasm so the later workbench can explain what a runtime performs beneath its API.

Why It Matters

Using a mature runtime is usually the right production decision, but a black-box runtime is a poor first mental model. Building a deliberately small engine exposes why layouts affect loops, why broadcasting can allocate or disappear into indexing, why weight format is part of the ABI, and why a fast wrong kernel is worse than a slow reference implementation. Those lessons make later runtime, WebGPU, WebNN, and quantization comparisons meaningful.

Questions This Branch Answers

• How do rank, shape, strides, dtype, and layout determine the address of a tensor element?
• Which broadcasting rules are valid, and how can incompatible shapes be rejected before execution?
• How do reference matrix multiplication and convolution kernels translate mathematical indices into linear-memory accesses?
• Where are bias and activation fused safely, and what does fusion change about observability and intermediate buffers?
• Why does naïve softmax overflow or underflow, and how does max subtraction improve stability?
• How are layer normalization axes, epsilon, scale, and bias represented and verified?
• How should pretrained weights be serialized, validated, loaded, and associated with expected shapes?
• What tolerance policy distinguishes expected floating-point drift from an indexing or semantic bug?

Scope

• Dense tensors with explicit FP32 dtype, rank, shapes, contiguous and selected strided layouts.
• Shape validation, offset calculation, views, transposition concepts, and NumPy-style broadcasting rules needed by chosen operators.
• Clear scalar reference kernels for matrix multiplication and a constrained 2D convolution.
• Bias, ReLU and selected activations, stable softmax, and layer normalization.
• Weight manifests, little-endian binary payloads, integrity checks, and deterministic loading into linear memory.
• A fixed, pretrained multilayer perceptron forward pass implemented without an external inference runtime.
• Differential testing, golden outputs, invariants, absolute/relative tolerances, and reproducible CPU/Wasm benchmarks.

Out of Scope

• Training, automatic differentiation, optimizers, backpropagation, or a general tensor compiler.
• Full ONNX parsing, complete operator coverage, dynamic control flow, sparse tensors, or production runtime compatibility.
• Claiming the reference kernels are competitive with optimized native, GPU, NPU, or mature Wasm runtimes.
• SIMD, threads, workers, WebGPU, and WebNN optimization; later branches will preserve these kernels as correctness baselines.
• Quantized arithmetic and model conversion beyond defining the FP32 baseline they must later match.
• Re-teaching neural-network theory already covered by the AI Atlas.

Expected Outcomes

After this branch, the reader should be able to:

• Specify and validate a dense tensor descriptor and compute element addresses from strides.
• Implement readable scalar matmul, constrained convolution, bias, activation, stable softmax, and layer normalization kernels.
• Load versioned pretrained weights without interpreting arbitrary bytes as trusted shapes or lengths.
• Schedule a small MLP forward pass with explicit intermediate-buffer lifetimes.
• Build a differential test suite that catches shape, layout, bounds, and numerical-stability defects.
• Explain which responsibilities belong to a model format, an inference runtime, a kernel backend, and the host application.

Candidate Note Roadmap

• tensor-descriptors-layout-strides-and-broadcasting — represent rank, shape, dtype, byte ranges, contiguous and strided layouts, NCHW/NHWC conventions, transpose choices, broadcast-compatible indexing, and trace element offsets by hand.
• matmul-as-a-correctness-kernel — implement the triple loop, define dimension contracts, choose accumulation precision, and test non-square edge cases.
• convolution-without-a-runtime — lower a constrained 2D convolution directly to loops and make padding, stride, channels, and layout explicit.
• bias-activations-and-fusion-boundaries — add bias and ReLU while preserving a separately testable unfused reference path.
• stable-softmax-layer-normalization-and-exceptional-values — implement max-shifted softmax plus layer-normalization mean, variance, epsilon, scale, and bias over documented axes, with invariants and adversarial infinities and NaN inputs.
• weight-manifests-and-binary-loading — version names, shapes, offsets, lengths, dtype, endianness, and hashes before copying weights into memory.
• a-pretrained-mlp-forward-pass — schedule dense, bias, ReLU, dense, and softmax stages with planned activation reuse and golden outputs.
• numerical-tolerances-and-differential-testing — combine elementwise tolerances, aggregate error, invariants, metamorphic cases, and failure triage.

Future Runnable Artifact

WasmAI Workbench v1 will execute a fixed pretrained FP32 MLP with 16 input features, one 32-unit hidden layer, and 4 output classes. Weights and biases will live in a little-endian binary file accompanied by a versioned manifest containing dtype, shapes, byte offsets, byte lengths, labels, and SHA-256 digest. The model will be trained or generated offline with a fixed seed; training will not occur in the browser.

The Wasm module will implement dense, bias_relu, and numerically stable softmax kernels, then expose one run_mlp(input_ptr, output_ptr) forward pass. TypeScript will validate the manifest, allocate immutable weight regions plus reusable activation buffers, load one input, invoke the model, and display logits, probabilities, predicted class, and the selected tolerance policy. A straightforward TypeScript reference and stored golden vectors will remain alongside the Wasm result.

The artifact will also include separately callable scalar matmul, constrained NCHW convolution, and layer-normalization reference kernels as learning and test fixtures. They will not be presented as unsupported operators in the MLP or as a general-purpose runtime.

How to Verify and Measure

• Unit-test descriptor validation, stride derivation, offset arithmetic, broadcasting decisions, and every kernel before composing a model.
• Differentially test small random shapes against an independent reference implementation, including non-square matrices, singleton dimensions, zero-sized policy cases, and invalid descriptors.
• Use identity, zero, one-hot, constant, and hand-computable tensors to localize indexing mistakes that random tests can obscure.
• Test stable softmax with large positive/negative logits, equal logits, infinities, and NaN; verify documented behavior, finite normal cases, non-negativity, and probability sums within tolerance.
• Verify weight manifest version, dtype, shape products, non-overlapping ranges, exact file length, and SHA-256 before instantiation or copying.
• Define combined absolute/relative comparison explicitly and report maximum absolute error, maximum relative error, mismatched element count, probability-sum error, and argmax agreement.
• Run memory canaries and repeated inference to detect writes outside activation/output regions and accidental mutation of weights.
• Benchmark kernels and end-to-end inference separately after correctness gates, recording cold/warm latency distributions, throughput at stated batch size, peak pages, allocation count, module bytes, engine version, and hardware.

Primary Sources

• ONNX Intermediate Representation specification — primary semantics for tensors, graphs, initializers, shapes, types, and versioning used as a vocabulary reference. ONNX 1.23.0 documentation; checked 2026-07-16.
• ONNX operator specifications — primary versioned operator signatures and semantics for comparison fixtures. ONNX 1.23.0 living documentation; checked 2026-07-16.
• WebAssembly numeric execution semantics — normative integer and floating-point behavior of the execution substrate. Core Specification Release 3.0; checked 2026-07-16.
• IEEE 754-2019 — primary floating-point arithmetic standard referenced by Wasm with specified WebAssembly-level qualifications. Published standard; status checked 2026-07-16.
• Netlib BLAS — primary reference interface and implementations for foundational dense linear-algebra operations used for conceptual comparison. Maintained reference collection; checked 2026-07-16.

Connects to: Linear Memory, ABI, and Host Interop · SIMD, Threads, and Workers · Model Formats, Conversion, and Quantization · Browser Inference Runtimes · WebGPU for AI