Browser LLMs, Embeddings & RAG

Introduction and mental model

Language intelligence in the browser should begin with the smallest pipeline that creates useful, inspectable results. That pipeline is usually retrieval before generation: import local documents, create embeddings, persist an index, retrieve ranked passages, and show evidence.

The browser is not a smaller cloud server. Model downloads, memory ceilings, storage eviction, main-thread responsiveness, backend availability, and tensor copies shape the architecture. Generation with a small language model is an optional later stage, enabled only when the device clears explicit capability and resource checks; semantic search must remain useful without it.

Why it matters

A retrieval-first design delivers private local search on a much wider range of devices than an on-device chatbot. It creates measurable intermediate outputs — chunks, embeddings, scores, and evidence — and avoids making the entire product dependent on autoregressive decoding performance or large model downloads.

When generation is added, the same retrieval layer provides bounded, attributable context. The product can compare retrieval-only and retrieval-plus-generation paths instead of treating a fluent answer as proof of correctness.

Questions this branch answers

How are documents parsed and chunked without sending their contents to a server?
Which small embedding model, tensor layout, and pooling behavior fit the target device?
When is exact cosine search sufficient, and when is a local index structure justified?
How should embeddings, document versions, and chunk metadata persist in IndexedDB?
How can retrieval quality be measured with golden queries and expected evidence?
What context can an optional generator consume within its memory and token budget?
How should prefill, decode, KV cache, streaming, and cancellation be reported?
When must the product remain retrieval-only or refuse local generation?

Scope

Browser tokenization, padding, attention masks, pooling, and embedding normalization.
Local document import, deterministic chunking, metadata, and version tracking.
Exact vector search over typed arrays and justified local indexing strategies.
IndexedDB persistence for documents, chunks, embeddings, and schema migrations.
Retrieval ranking, evidence display, golden queries, and retrieval-quality metrics.
Context assembly with explicit byte, token, and memory budgets.
Optional small-language-model prefill, autoregressive decode, KV cache, sampling, and streaming.
Capability gates, model download UX, cancellation, memory limits, and visible fallbacks.

Out of scope

Re-teaching transformer architecture, general RAG theory, or enterprise retrieval systems.
Training embedding models or language models in the browser.
Server-hosted vector databases, ingestion services, or mandatory cloud generation.
Presenting a chatbot as the initial or universal goal of the branch.
Assuming that one model, quantization, or backend works across all browsers and devices.
Hiding retrieval evidence behind a generated answer.

Expected outcomes

After this branch, a reader should be able to build and evaluate a fully local semantic search path, explain every representation from source document to ranked chunk, and persist that path across offline sessions. They should be able to decide whether local generation fits the device, add it without coupling retrieval to it, and report memory, download, latency, quality, and fallback behavior separately.

Candidate note roadmap

browser-tokenization-embeddings-pooling-and-normalization — make token IDs, masks, truncation, tensor shapes, embedding-model outputs, pooling, and normalization inspectable so local vectors are reproducible and comparable.
local-document-ingestion-and-deterministic-chunking — preserve source identity, offsets, and reproducible chunks.
exact-vector-search-with-typed-arrays — establish a correct cosine-similarity baseline before adding indexes.
persistent-embedding-indexes-in-indexeddb — version schemas, documents, embeddings, and rebuild state.
retrieval-quality-golden-queries-and-evidence-ui — evaluate recall, ranking, provenance, and user-visible evidence.
local-context-assembly-and-resource-budgets — bound retrieved text by tokens, bytes, memory, and task value.
optional-small-language-model-generation — add prefill, decode, KV cache, streaming, and sampling behind a capability gate.
browser-language-model-limits-and-fallback-policy — make download, memory, performance, and unsupported states explicit.

Future runnable artifact

Build the retrieval-only v6 milestone of WasmAI Workbench: a user imports local text documents, the browser creates deterministic chunks, runs a local embedding model, persists versioned vectors and metadata in IndexedDB, performs local similarity search, and returns ranked passages with source names and offsets. The artifact will include a small golden corpus and query set so retrieval output can be reproduced after reload, offline use, schema migration, and index rebuild.

A later optional adapter may pass selected evidence to a small local language model. That adapter must be disabled when capability, memory, or latency budgets fail; the retrieval-only experience remains complete. Its metrics and output verification must be reported separately from retrieval.

How to verify and measure

Compare chunk boundaries, embedding dimensions, normalized vectors, and persisted records against golden fixtures.
Measure ingestion latency, query latency p50/p95, index bytes, model bytes, peak memory, and rebuild time.
Evaluate retrieval with a versioned query set using recall at k, ranking position, and evidence provenance.
Verify identical corpus and model versions produce equivalent rankings within documented numerical tolerances.
Test offline reload, interrupted model acquisition, storage eviction, migration, cancellation, and fallback behavior.
For optional generation, record TTFT, prefill time, decode tokens per second, KV-cache growth, output checksum, and cited evidence separately.
Report browser, operating system, hardware, runtime, backend, model version, and quantization with every result.

Primary sources

ONNX Runtime Web documentation — official reference for sessions, tensors, execution providers, and browser constraints; version-sensitive, consulted 2026-07-16.
Transformers.js documentation — official reference for tokenizers, embeddings, and supported model pipelines; version-sensitive, consulted 2026-07-16.
Indexed Database API 3.0 — primary persistence API for local corpora and vector records; living specification, consulted 2026-07-16.
WebGPU specification — primary browser GPU interface used by some inference backends; evolving specification and implementation matrix, consulted 2026-07-16.
WebNN specification — primary graph API for hardware-backed neural inference; implementation support remains non-universal, consulted 2026-07-16.
Lewis et al., Retrieval-Augmented Generation — primary RAG paper for the retriever-generator decomposition; conceptual source, not a browser implementation guide.
Hugging Face Tokenizers documentation — official tokenizer behavior and model-input reference; bindings and browser packaging are version-sensitive, consulted 2026-07-16.

Connects to: Browser Inference Runtimes · Preprocessing, Postprocessing & Media Pipelines · Model Formats, Conversion & Quantization · Local-First, Offline & Multimodal AI · Wasm AI Performance, Security & Craftsmanship