Defense in depth and least privilege
Mental model: an AI model is an untrusted proposal generator. Defense in depth assumes a prompt, classifier, provider, or reviewer will eventually fail; independent controls must prevent one failure from becoming unauthorized disclosure or action. Least privilege gives every identity, tool, and workflow only the authority required for its current task.
Mechanism: identity → scoped capability → verified side effect
Resolve the user and tenant before retrieval, issue a service identity with a narrow allowlist, validate arguments and current state at execution, and require approval for consequential actions. Log the decision and keep a kill switch. A model cannot upgrade its authority because retrieved text says so.
policy = {"support-agent": {"search_orders", "draft_reply"}}
def authorize(role, tool): return tool in policy.get(role, set())
print(authorize("support-agent", "issue_refund"))
assert not authorize("support-agent", "issue_refund")
Run with python3; expected output is False. The executor must also enforce tenant scope, amount limits, idempotency, and approval policy; an allowlist alone is not a full control.
Layers that should fail independently
| Layer | Control | Failure it contains |
|---|---|---|
| Identity | separate service principals, tenant scoping | cross-user access |
| Data | authorization before retrieval, minimization | private context exposure |
| Tools | narrow schemas and semantic checks | malformed or excessive action |
| Network | egress allowlists and secret isolation | exfiltration |
| Workflow | budgets, approvals, idempotency | runaway or irreversible action |
| Operations | traces, alerts, kill switch, rollback | delayed incident response |
Failure modes and decision rule
Prompt rules and post-hoc output filters are not authorization. A broad database token, shell access, or shared admin credential defeats the whole design. Remove unnecessary capabilities first; then add compensating controls for the remaining high-impact paths. Release only when each external action has a named identity, policy, audit record, and recovery route.
Exercises
- Add a tenant identifier to the artifact and test a cross-tenant denial.
- Classify five tools by reversibility and decide which requires human approval.
Connects to: autonomy control · approval gates · indirect injection · tracing
Sources
- OWASP LLM06: Excessive Agency — excessive functionality, permissions, and autonomy risks.
- NIST AI RMF — secure, resilient lifecycle controls.
- MITRE ATLAS — adversarial ML tactics and mitigations.