Threat modeling LLM apps
Mental model: threat modeling asks what can cross a trust boundary, who can influence it, what authority it reaches, and how harm is prevented or recovered. An LLM app is not just a prompt: it includes identities, retrieval, tools, secrets, logs, UI, vendors, and human approvals.
Mechanism: asset → boundary → attacker path → control → regression
Inventory assets (personal data, credentials, money, reputation), actors, entry points, and side effects. Draw the path from untrusted input through model/context/tool execution to the asset. For each path, add a preventive control, detection signal, owner, recovery action, and a reproducible attack fixture.
path = {"source":"web page", "asset":"customer data", "action":"send_email", "gate":False}
print("block release" if not path["gate"] else "test control")
Run with python3; expected output is block release. The artifact expresses a rule: an untrusted-content-to-external-action path needs an independent authorization boundary.
| Boundary | Typical threat | Control |
|---|---|---|
| User → model | direct injection, data exposure | input policy, minimization |
| Retrieval → context | indirect injection, cross-tenant data | ACL before retrieval, source labels |
| Model → tool | excessive agency, malformed action | narrow schema, semantic validation, approval |
| Output → UI | XSS, misleading action | context-aware encoding, confirmation |
| Trace → operator | PII leakage | redaction, retention, access audit |
Failure modes and decision rule
Do not model only model behavior while ignoring credentials, logs, or side effects. A prompt refusal is not a control against a compromised tool path. Release only when each high-impact path has a testable preventive control, detection, owner, and rollback or incident path.
Exercises
- Model a retrieved support ticket that asks an agent to export data.
- Add a tenant boundary and prove that the execution layer rejects a cross-tenant request.
Connects to: indirect injection · least privilege · red-team playbook · approval gates
Sources
- NIST AI RMF — lifecycle risk-management framework.
- MITRE ATLAS — adversarial ML tactics and techniques.
- OWASP LLM Top 10 — application-security threat taxonomy.