AI Safety and Security

AI safety and security is the discipline of making AI systems resistant to misuse, accidents, data exposure, and model-mediated attacks. The core move is to treat the model as an unreliable component inside a security boundary, not as the boundary.

Mental model

An AI application crosses trust boundaries whenever untrusted data can influence model output and that output can reach data, code, money, or people. Security therefore constrains authority and validates effects outside the model; a prompt is never the sole enforcement layer.

Roadmap: threat landscape to assurance

• OWASP LLM Top 10 overview gives the shared vocabulary for LLM application risk.
• Direct prompt injection covers user-provided instructions that try to override the system.
• Indirect prompt injection covers malicious instructions hidden in retrieved data, webpages, documents, and tool output.
• Jailbreaks explains adversarial prompts that try to bypass safety behavior.

Data and action risk

• Data and PII leakage focuses on sensitive information exposure through prompts, logs, retrieval, and outputs.
• Excessive agency covers agents with too much tool access, autonomy, or permission.
• Insecure output handling treats model output as untrusted input to downstream systems.

Controls and assurance

• Threat modeling LLM apps maps assets, trust boundaries, attackers, and failure modes.
• Input and output guardrails places checks before and after the model.
• Red teaming AI systems stress-tests prompts, retrieval, tools, and policies.
• Defense in depth and least privilege explains why prompts are only one layer.
• Privacy and data governance defines data minimization, retention, access, and audit controls.

Connects to: Autonomy and Control · Evaluation · AI Ethics and Governance

Core sources

• OWASP Top 10 for LLM Applications — application threat taxonomy and mitigations.
• MITRE ATLAS — adversarial ML tactics, techniques, case studies, and mitigations.
• NIST Adversarial Machine Learning Taxonomy — standardized attack and mitigation terminology.
• Indirect Prompt Injection — early systematic treatment of attacks delivered through external data.