Run an AI red team lite

Mental model: test attacker goals against the assembled system—data, retrieval, tools, identity, UI, and logging—not a prompt in isolation.

Mechanism: attacker goal → trace → exploitability → regression

Use this playbook when a small team needs a focused adversarial pass before releasing or expanding an AI feature. It is not a full security assessment, but it catches many obvious failures early.

Inputs

• Product workflow, system prompt, retrieval sources, tool list, guardrails, and threat model.
• Test account with realistic permissions.
• Logging enabled for prompts, context, tool calls, guardrail decisions, and outputs.

Procedure

1. Pick 5-10 attacker goals: reveal secrets, bypass policy, misuse tools, exfiltrate data, poison context, or burn resources.
2. Test direct jailbreaks and prompt injection against normal user input.
3. Test indirect injection inside documents, webpages, support tickets, or tool output.
4. Test data leakage with cross-tenant, unauthorized, and hidden-context attempts.
5. Test tool misuse with unsafe destinations, malformed arguments, and excessive actions.
6. Test output handling with generated HTML, links, code, commands, and SQL-like payloads.
7. Record whether the system blocks, refuses, escalates, or fails silently.
8. Rank findings by impact, exploitability, and ease of mitigation.
9. Convert confirmed failures into regression tests.

Report template

Pitfall

Do not stop at model-only jailbreaks. The riskiest failures often involve retrieval, tools, permissions, caches, logs, and UI decisions.

Connects to: red teaming AI systems · input and output guardrails · regression testing

Executable severity triage

impact, exploitability = 4, 3
print("priority", impact * exploitability)

Run with python3; expected output is priority 12. Confirmed failures need an owner, mitigation, retest, and regression fixture.

Sources

• OWASP LLM Top 10 — application-risk taxonomy.
• MITRE ATLAS — adversarial-ML techniques.
• NIST AI RMF — risk-management process.