Local-First, Offline & Multimodal AI

Introduction and mental model

Local AI is not merely remote inference moved into a browser tab. It is a delivery system in which the application shell, model artifacts, preprocessing code, runtime, and user data must remain usable under unreliable connectivity and finite device resources.

The useful mental model is an installable capability with a lifecycle. A model is a large, versioned local asset; inference is one stage in a media pipeline; offline is a property that must hold across loading, storage, updates, backend selection, and failure recovery. Every fallback must be visible rather than silently changing the privacy, quality, or hardware contract.

Why it matters

Running image, audio, and embedding workloads locally can reduce network latency, preserve sensitive inputs, and keep a product useful without a server round trip. It also transfers responsibilities normally hidden by an API to the application: download size, quota and eviction, memory pressure, model integrity, warm-up, cancellation, compatibility, and recovery from interrupted updates.

An honest local-first product must distinguish “works after installation,” “works on this device,” and “works with this backend.” Those are separate claims and require separate evidence.

Questions this branch answers

What must be cached for a feature to work after the network disappears?
Where should versioned model shards and metadata live, and what can be evicted?
How can interrupted downloads resume without accepting partial or corrupted weights?
How should capability checks, warm-up, progress, cancellation, and fallbacks appear in the UI?
Which image and audio workloads fit realistic browser memory and latency budgets?
How can a model update be activated atomically and rolled back safely?
Which privacy guarantees follow from local execution, and which claims still require proof?

Scope

Service Workers, Cache Storage, Web App Manifest, and offline application-shell behavior.
IndexedDB-backed model metadata, shards, checksums, versions, and resumable downloads.
Storage quota, persistence requests, eviction risk, and recovery from missing assets.
Capability detection, backend disclosure, warm-up, cancellation, and progress reporting.
Local image classification or segmentation and small speech-to-text pipelines.
Model activation, rollback, garbage collection, and update policy.
Mobile memory pressure, lifecycle cleanup, and graceful degradation.
Local handling of user media, with explicit boundaries around logs and persisted data.

Out of scope

Training or fine-tuning models in the browser.
Re-teaching computer-vision, speech, or generative-model architecture.
Making a local chatbot the default demonstration or requiring a large language model.
Hiding browser, operating-system, hardware, or backend compatibility differences.
General semantic retrieval design, which belongs to the browser LLMs, embeddings, and RAG branch.
Runtime implementation internals covered by the browser inference runtimes branch.

Expected outcomes

After this branch, a reader should be able to design an offline-capable AI feature as a complete state machine: install, acquire model, verify, activate, warm, infer, update, roll back, and reclaim storage. They should be able to state which user data stays local, identify every network dependency, expose the selected backend, and define device-aware failure behavior before choosing a model.

Candidate note roadmap

local-first-ai-as-a-delivery-contract — define online, installed, offline-ready, degraded, and unavailable states.
service-worker-shell-and-offline-boundaries — separate application-shell caching from model and user-data storage.
model-storage-indexeddb-quota-and-eviction — design durable metadata, shards, quota checks, and repair paths.
resumable-model-downloads-checksums-and-activation — stage, verify, and atomically activate versioned artifacts.
capability-detection-backend-disclosure-and-fallbacks — choose a compatible path without silently changing the contract.
local-image-inference-end-to-end — connect decode, preprocessing, inference, postprocessing, and golden outputs.
local-speech-to-text-under-device-constraints — handle audio decoding, resampling, chunking, cancellation, and memory.
model-updates-rollback-and-storage-reclamation — evolve local models without stranding an offline installation.

Future runnable artifact

Build a static, installable WasmAI Workbench milestone that offers local image classification and small-model speech-to-text after an explicit first-time model installation. The artifact will use a versioned model manifest, content hashes, resumable shards in IndexedDB, an application shell controlled by a Service Worker, atomic activation, rollback to the previous verified version, and a visible backend decision.

The workbench must remain functional after a full offline reload, show download and warm-up progress, support cancellation, expose storage usage, and compare at least one fixed media fixture with a checked-in golden output. A failed checksum, missing shard, quota failure, or unsupported backend must produce an explicit recoverable state.

How to verify and measure

Test first install, interrupted install, offline reload, update, rollback, eviction recovery, and cancellation as distinct scenarios.
Record application bytes, model bytes, stored bytes, peak memory, cold start, warm inference p50/p95, and backend used.
Verify every model shard before activation and record the active manifest plus output checksum.
Compare outputs against golden fixtures with a documented per-model numerical tolerance.
Confirm that media and derived tensors do not leave the device during the local path.
Exercise the scalar or CPU fallback and prove that the UI reports the downgrade.
Repeat on a declared browser, operating system, and hardware matrix; absence of support is a valid result.

Primary sources

Service Workers specification — primary definition of fetch interception, lifecycle, and offline application behavior; living specification, consulted 2026-07-16.
Indexed Database API 3.0 — primary storage API for structured metadata and large local artifacts; living specification, consulted 2026-07-16.
WHATWG Storage Standard — quota, storage buckets, persistence, and eviction model; living standard with implementation-sensitive behavior, consulted 2026-07-16.
Web Application Manifest — install metadata and application identity; evolving W3C specification, consulted 2026-07-16.
ONNX Runtime Web documentation — official browser-runtime loading and backend guidance; version-sensitive implementation documentation, consulted 2026-07-16.
Transformers.js documentation — official reference for supported browser pipelines and model loading; fast-moving implementation documentation, consulted 2026-07-16.
MediaPipe Tasks for Web — official local media-task implementation reference; API and platform support are version-sensitive, consulted 2026-07-16.

Connects to: Browser Inference Runtimes · Preprocessing, Postprocessing & Media Pipelines · Model Formats, Conversion & Quantization · WebNN & Adaptive Backends · Wasm AI Performance, Security & Craftsmanship