Model Formats, Conversion, and Quantization

Introduction and mental model

A model file is an execution contract, not merely a bag of weights with a different extension. That contract may describe a compute graph, operator versions, tensors, data types, shapes, metadata, and external weight locations. A converter acts more like a compiler than a file copier: it lowers one set of semantics into another, rewrites the graph, and may reject or approximate operations that the destination cannot express.

The major formats in this branch are deliberately not treated as interchangeable. ONNX represents a versioned graph and tensors. A LiteRT model packages a graph for the LiteRT ecosystem in a FlatBuffer. Safetensors is a bounded tensor container and does not by itself define an executable graph. GGUF combines tensors and metadata for runtimes in the GGML family and other consumers that explicitly implement its conventions. The useful question is therefore not "Which format is best?" but "Which representation preserves this model's semantics and is supported by this target runtime and backend?"

Quantization is another compilation decision. FP16, INT8, and 4-bit schemes reduce different costs under different hardware and runtime constraints. Their effect must be measured on the actual model, operator set, input distribution, and backend; a smaller artifact does not universally imply a faster or sufficiently accurate workload.

Why it matters

Format and conversion choices determine whether a model can load at all, which operators can execute, how many network requests are required, how much memory is occupied, and whether outputs remain useful. These effects are especially visible in browser and Wasm deployments, where download size, startup latency, linear-memory limits, cache behavior, and backend-specific operator coverage all matter.

A reproducible conversion record also turns opaque runtime errors into diagnosable compatibility failures. It preserves the source checkpoint identity, exporter and converter versions, graph transformations, calibration corpus, target operator set, and numerical acceptance criteria.

Questions this branch answers

What semantics live in a graph format, a tensor container, and runtime-specific metadata?
When are static shapes preferable, and which dynamic dimensions does the target runtime actually support?
How do PyTorch export paths preserve or specialize control flow and shapes?
Which operators, data types, and opset versions survive conversion to ONNX or LiteRT?
When should optimization happen before quantization, and when can graph rewriting make debugging harder?
What are the practical differences among FP32, FP16, INT8, and common 4-bit schemes?
How should a representative calibration set be selected without leaking evaluation data?
How can numerical drift be separated from a layout, preprocessing, or operator-semantics bug?
When should ONNX external data be used, and how does it alter packaging, fetching, and integrity checks?

Scope

Compute graphs, initializers, weights, metadata, operator sets, opsets, shapes, layouts, and data types.
ONNX and LiteRT model structure and the conversion paths that target them.
Safetensors and GGUF as specialized representations with different execution assumptions.
PyTorch export, graph inspection, simplification, constant folding, and target-specific rewriting.
Static and dynamic shapes, including explicit shape constraints and specialization decisions.
FP32, FP16, INT8, and 4-bit quantization families, with scheme and backend named explicitly.
Representative calibration, accuracy checks, numerical tolerances, and regression fixtures.
Large-model packaging with ONNX external data, deterministic manifests, hashes, and cache-aware delivery.

Out of scope

Training recipes, optimizer selection, distributed training, or dataset acquisition.
A universal interchange promise across formats or runtimes.
Runtime adapter implementations, browser UI, or backend-selection code.
Hand-written inference kernels and accelerator-specific shader optimization.
Claims that one precision or format is always smaller, faster, or more accurate on every target.

Expected outcomes

After completing this branch, a reader should be able to inspect a model's real contract, choose a destination format for a named runtime, predict likely compatibility failures, and produce a conversion manifest that another person can reproduce. They should also be able to design quantization experiments with explicit baselines and acceptance thresholds, diagnose output drift, and package split model data without losing provenance or integrity.

Candidate note roadmap

Model formats are execution contracts, not interchangeable containers — separate graph semantics, tensor storage, metadata, operator versions, and runtime conventions while contrasting ONNX and LiteRT executable graphs with Safetensors tensor storage and GGUF's specialized metadata-rich packaging.
Reading an ONNX graph from inputs to opsets and external data — inspect value information, initializers, symbolic dimensions, domains, versioning, and sidecar tensor files.
LiteRT FlatBuffers and the converter-to-runtime boundary — trace signatures, tensor layouts, supported operations, delegates, and conversion diagnostics without assuming ONNX equivalence.
Exporting PyTorch graphs with explicit shape constraints — record exporter choice, example inputs, dynamic dimensions, decompositions, and semantic checks against eager execution.
Operator coverage is a target matrix — derive a compatibility matrix from model graph, runtime release, backend, data type, and operator implementation rather than a single support badge.
Graph optimization without losing a debuggable baseline — stage constant folding, fusion, simplification, and layout rewriting while retaining inspectable intermediate artifacts.
FP16, INT8, and 4-bit quantization, calibration, and numerical drift — name weight/activation treatment, granularity, symmetry, zero points, scales, accumulators, representative calibration data, backend requirements, and layer/graph comparisons that separate acceptable approximation from conversion defects.
Packaging large ONNX models with external data — define relative paths, request and cache behavior, file hashes, atomic publication, and offline completeness.

Future runnable artifact

Build a reproducible conversion lab around one small, fixed image model and a checked-in set of non-sensitive test inputs. A pinned source checkpoint will be exported to ONNX FP32 and LiteRT FP32, then converted into one documented quantized variant for each format that the selected tools and target runtimes actually support. Every artifact will have a manifest containing source hash, tool versions, commands, input signature, operator inventory, precision scheme, file hashes, and any external-data files.

The lab will run the same fixtures through a Python reference and through each exported artifact. It will save raw outputs and report maximum absolute error, maximum relative error with a protected denominator, cosine similarity where meaningful, and task-level agreement such as top-k labels. It will also record total bytes, compressed transfer bytes, cold load time, warm inference time, and peak memory when the environment exposes a credible measurement. A failed export or unsupported operator will remain a named result, not be silently replaced by another model or precision.

How to verify and measure

Hash the source checkpoint, conversion inputs, produced model files, and external-data files.
Pin and record exporter, converter, runtime, and operator-set versions.
Inspect graph inputs, outputs, shapes, layouts, data types, operators, and domains before and after every transformation.
Compare exported FP32 outputs with the source framework before attributing drift to quantization.
Use fixed golden fixtures plus a representative evaluation set; publish the tolerance formula and threshold per output.
Measure quantized variants against their corresponding FP32 target-runtime baseline as well as the source model.
Run forced-backend compatibility checks so an implicit fallback cannot make an unsupported graph look supported.
Separate download, parsing, initialization, first inference, steady-state inference, and disposal measurements.
Verify that a fresh offline cache contains every model shard or external-data file and rejects a hash mismatch.

Primary sources

ONNX Intermediate Representation — https://onnx.ai/onnx/repo-docs/IR.html — Primary specification for graph, tensor, type, shape, and versioning semantics. Consulted 2026-07-16.
ONNX operator specifications — https://onnx.ai/onnx/operators/ — Versioned operator definitions; compatibility depends on the model's domains and opsets. Consulted 2026-07-16.
ONNX external data documentation — https://onnx.ai/onnx/repo-docs/ExternalData.html — Primary guidance for tensors stored outside the main protobuf. Consulted 2026-07-16.
PyTorch ONNX exporter documentation — https://docs.pytorch.org/docs/stable/onnx.html — Version-sensitive exporter behavior and APIs; consulted 2026-07-16 and should be rechecked when the toolchain is upgraded.
LiteRT model conversion overview — https://developers.google.com/edge/litert/conversion/overview — Current Google conversion guidance; version-sensitive and consulted 2026-07-16.
LiteRT post-training quantization — https://developers.google.com/edge/litert/conversion/tensorflow/quantization/post_training_quantization — Current scheme and tooling guidance; version-sensitive and consulted 2026-07-16.
ONNX Runtime quantization documentation — https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html — Runtime-specific quantization guidance whose supported paths can change; consulted 2026-07-16.
ONNX Runtime float16 documentation — https://onnxruntime.ai/docs/performance/model-optimizations/float16.html — Runtime-specific FP16 conversion and blocking guidance; consulted 2026-07-16.
Safetensors format repository — https://github.com/huggingface/safetensors — Primary implementation and format description. Consulted 2026-07-16.
GGUF specification — https://github.com/ggml-org/ggml/blob/master/docs/gguf.md — Primary format specification in the GGML project; evolving and consulted 2026-07-16.

Connects to: Toolchains and Language Targets · Tensor Kernels and Inference from Scratch · Browser Inference Runtimes · WASI, Components, and WASI-NN · Wasm AI Performance, Security, and Craftsmanship