The AI application stack
Mechanism: request → context → model/tools → controlled outcome → evidence
layers = ["identity", "context", "model", "tools", "policy", "trace"]
assert layers[-1] == "trace"
print("stack is observable")
Run with python3; expected output is stack is observable. Debug from outcome toward the earliest failed layer; add layers only when an eval identifies a missing capability or control.
Sources
- NIST AI RMF — lifecycle system controls.
- OpenTelemetry specification — trace semantics across layers.
Most of this atlas describes one layer at a time. This note assembles them. A production LLM app is not "call the API" — it's a stack of layers, each with its own failure modes and evals. Knowing the stack tells you where a problem lives and where to add the next improvement.
The layers (request flow, top to bottom)
| Layer | Job | Atlas branch |
|---|---|---|
| Interface / UX | streaming, trust, error handling, human review | product |
| Orchestration | workflow vs agent, routing, retries | agents |
| Context assembly | prompt + retrieved docs + tools + memory | prompting |
| Retrieval | fetch relevant knowledge | RAG |
| Tools | act on the world | agents |
| Model | the LLM (base / instruct / reasoning) | LLMs |
| Guardrails | input/output checks, policy | safety |
| Serving | latency, throughput, cost | inference |
| Eval & observability | tracing, offline + online quality | eval / MLOps |
The model is one box among many. Most product quality comes from the layers around it — context, retrieval, guardrails, and evaluation.
Two cross-cutting planes
- Evaluation runs through every layer: you eval the product, not just the model, and you eval each component (retriever, agent).
- Observability (tracing) makes the whole stack inspectable — without it, you can't tell which layer failed.
How to use the map
- Debugging: trace a bad output down the stack — wrong answer → retrieval? prompt? model? guardrail? Fix the earliest broken layer.
- Building: start with the smallest stack that works (prompt + model), add layers only when evals show you need them. Don't build RAG + agents on day one.
- Cost/latency: each layer adds both; the triangle is a stack-wide budget.
Pitfall
Treating the LLM as the whole system. The model is a probabilistic component; the engineering around it — grounding, validation, fallbacks, evals — is what makes a product reliable. Design the system, not the prompt.
Connects to: mental models for AI systems · product engineering · product evals