Preprocessing, Postprocessing, and Media Pipelines

Introduction and mental model

Preprocessing and postprocessing are part of the model contract. The graph may accept a tensor, but the application decides how pixels, samples, or text become that tensor and how raw outputs become labels, boxes, tokens, scores, or embeddings. A mismatch in channel order, color range, resize rule, sample rate, tokenizer version, padding side, or threshold can produce plausible but wrong results even when inference itself is exact.

The useful mental model is a typed pipeline of observable transformations. Each stage has an input representation, an output representation, parameters, ownership, and a verification point. Image decode is separate from orientation and color handling; resize is separate from crop; layout conversion is separate from normalization. Audio decode is separate from channel mixing, resampling, framing, and mel projection. Tokenization is separate from truncation, padding, special-token insertion, and attention-mask construction.

Postprocessing deserves the same rigor. Non-maximum suppression, score activation, label mapping, logit decoding, thresholding, and embedding normalization all encode semantics that must match the model family and reference implementation.

Why it matters

Boundary bugs often survive smoke tests because outputs still look numerically reasonable. They also distort performance results: a benchmark labeled "inference" may include decode, copies, resize, tokenization, readback, or JavaScript allocation in one implementation but exclude them in another.

Explicit media pipelines enable golden intermediate fixtures, deterministic regression tests, zero- or low-copy experiments, and honest end-to-end latency accounting. They also help separate browser API variation from model-runtime behavior.

Questions this branch answers

Which image orientation, color space, alpha, resize, crop, range, mean, and standard-deviation rules does a model require?
How do NHWC and NCHW layouts change indexing, copy costs, and tensor construction?
Where should browser image decoding end and model-specific preprocessing begin?
How should audio decoding, channel mixing, resampling, framing, windowing, and mel filters be specified?
Which tokenizer artifact, normalization rules, special tokens, truncation policy, padding side, and attention mask match a text model?
When are sigmoid, softmax, argmax, top-k, thresholding, or non-maximum suppression semantically appropriate?
How can intermediate buffers be compared with a Python reference without relying on screenshots or final labels alone?
Which stages can be fused or moved to Wasm or WebGPU without changing results beyond a declared tolerance?
How should nondeterministic ordering and equal-score ties be normalized for reproducible tests?

Scope

Browser image decode, orientation, color and alpha handling, resize, crop, normalization, and NHWC/NCHW conversion.
Audio decode, channel conversion, resampling, framing, windows, spectrograms, and mel feature extraction.
Text normalization, tokenization, vocabulary artifacts, special tokens, truncation, padding, and attention masks.
Output activation, label decoding, top-k, thresholding, box decoding, non-maximum suppression, and embedding normalization.
Typed-array and tensor allocation, strides, copies, buffer reuse, and stage ownership.
Golden intermediate fixtures, Python-reference parity, numerical tolerances, and deterministic ordering.
End-to-end measurement that distinguishes media preparation, inference, readback, and decoding.

Out of scope

Model training, augmentation policy design, or data-labeling workflows.
Codec implementation, general-purpose media editing, or a production streaming UI.
Assuming browser decoders yield identical pixels or samples without an explicit contract and fixture.
Runtime adapter implementations or accelerator kernels in this index.
Treating the final top-1 label as sufficient evidence that the pipeline is correct.

Expected outcomes

After completing this branch, a reader should be able to write an unambiguous preprocessing and postprocessing specification for an image, audio, or text model; identify every layout and copy boundary; and create golden tests for intermediate tensors as well as final outputs. They should also be able to profile stage-by-stage latency and safely evaluate whether a fused Wasm or WebGPU stage preserves the reference semantics.

Candidate note roadmap

Preprocessing is a versioned part of the model contract — capture transforms, parameters, assets, ordering, data types, layouts, and tolerances beside the model manifest.
Browser pixels to image tensors: geometry, layout, and copies — specify decode, EXIF orientation, color and alpha handling, resize interpolation, crop geometry, numeric range, NHWC/NCHW strides, contiguous representations, layout conversion, and measured copies rather than treating transpose as metadata.
Audio decode and resampling before feature extraction — make sample format, sample rate, channels, mixing rule, resampler, frame origin, and boundary padding explicit.
From PCM to mel features with inspectable intermediates — validate windows, FFT magnitude or power, mel filters, logarithm floor, normalization, and output layout.
Tokenization must ship the exact text contract — bind normalization, vocabulary, merges or model file, special tokens, truncation, padding side, IDs, and attention masks.
Decoding model outputs: logits, thresholds, boxes, and NMS — distinguish logits from probabilities; select sigmoid, softmax, top-k, and thresholds from model semantics; then specify box coordinate space, anchors, clipping, class handling, IoU, non-maximum suppression, and stable tie-breaking.
Embedding normalization and similarity parity — verify pooling, mask use, dimensionality, L2 normalization, distance function, and zero-vector handling.
Golden intermediates beat end-result guesswork — serialize representative stage outputs and compare them to a pinned Python reference with per-stage tolerances.

Future runnable artifact

Build three manual, inspectable pipelines around fixed, redistributable fixtures: an image-classification input and top-k decoder, an audio-to-mel feature pipeline, and a text-to-embedding input/output pipeline. Each browser pipeline will emit a manifest and selected intermediate arrays, then compare them with a pinned Python reference that implements the same written contract.

The image path will record decoded dimensions, orientation, crop rectangle, resize method, channel order, layout, and normalized tensor. The audio path will record decoded PCM, channel mix, resampled PCM, frame starts, windowed frames, and mel features. The text path will record normalized text, token IDs, special tokens, padding, attention mask, raw embedding, pooling, and normalized embedding. The artifact will show per-stage maximum absolute and relative error, shape and dtype mismatches, deterministic hashes where bitwise equality is expected, and final task agreement. It will also measure stage latency and bytes copied so a later Wasm or WebGPU optimization has an honest baseline.

How to verify and measure

Pin the media fixture bytes, model, tokenizer or vocabulary assets, preprocessing manifest, and Python environment.
Assert shape, data type, layout, stride assumptions, numeric range, and finite values at every stage boundary.
Save small golden intermediates and hash large fixtures; compare at the earliest stage that diverges.
Define absolute, relative, cosine, or task-level tolerances according to the transformation rather than using one global epsilon.
Exercise portrait orientation, alpha, grayscale, odd dimensions, extreme aspect ratios, silence, multiple sample rates, multichannel audio, empty text, Unicode normalization, and maximum sequence length.
Specify stable ordering for equal scores and boxes before asserting exact postprocessing output.
Measure decode, transform, allocation, copy, inference, readback, and output decoding separately and end to end.
Run repeated calls with buffer reuse and without reuse to expose allocation and garbage-collection effects.
Compare browser output with the reference on more than one fixture before generalizing about correctness.

Primary sources

WebCodecs specification — https://www.w3.org/TR/webcodecs/ — W3C Working Draft dated 2026-05-05, consulted 2026-07-16. It is changing and availability or codec support must be detected rather than assumed.
Web Audio API — https://www.w3.org/TR/webaudio-1.0/ — W3C Recommendation defining browser audio graph and buffer semantics. Consulted 2026-07-16.
ONNX NonMaxSuppression operator — https://onnx.ai/onnx/operators/onnx__NonMaxSuppression.html — Primary operator semantics; the model's imported opset must be checked. Consulted 2026-07-16.
SentencePiece repository — https://github.com/google/sentencepiece — Primary implementation and model-format source for SentencePiece tokenization. Consulted 2026-07-16.
Hugging Face Tokenizers documentation — https://huggingface.co/docs/tokenizers/ — Current tokenizer pipeline and component documentation; version-sensitive and consulted 2026-07-16.
MediaPipe Image Classifier for Web — https://ai.google.dev/edge/mediapipe/solutions/vision/image_classifier/web_js — Current task-specific image input and result guidance; version-sensitive and consulted 2026-07-16.
ONNX Runtime Web documentation — https://onnxruntime.ai/docs/tutorials/web/ — Current browser tensor and runtime guidance; version-sensitive and consulted 2026-07-16.
NumPy testing reference — https://numpy.org/doc/stable/reference/routines.testing.html — Primary documentation for explicit array comparison strategies used by the proposed reference harness. Consulted 2026-07-16.

Connects to: Linear Memory, ABI, and Host Interop · Browser Inference Runtimes · Local-First, Offline, Multimodal AI · Browser LLMs, Embeddings, and RAG · Wasm AI Performance, Security, and Craftsmanship