Applies toLLM chat assistants, RAG applications and agents on any provider; Azure AI Content Safety Prompt Shields, Bedrock Guardrails and similar input filters
Recognise the incident
| Report | Type |
|---|---|
| Screenshot of your system prompt on social media | Prompt leakage (direct injection) |
| Assistant gave answers outside policy after "ignore previous instructions" | Jailbreak (direct injection) |
| Assistant followed instructions hidden in an email/document/web page it read | Indirect injection |
| Agent sent data somewhere, or called tools unexpectedly | Indirect injection with tool abuse (highest severity) |
Step 1: contain (first 30 minutes)
Match the containment to the impact:
- Data or tool abuse: disable the affected tools or switch the agent to read-only (tool kill switch); block the source of the malicious content (RAG data source blocklist, or stop ingesting that mailbox/site).
- Policy bypass / harmful output: safe mode level 2–3 for the assistant.
- Specific attacker: block the user/tenant/key.
- System prompt leak only: usually no emergency containment. But check the prompt for secrets (step 3).
Step 2: scope (which conversations?)
You need logs of prompts, retrieved content, tool calls and responses (with access controls). Search for the attack pattern:
// Example if conversations are logged to Log Analytics / App Insights as traces
AppTraces
| where TimeGenerated > ago(14d)
| where Message has_any ("ignore previous instructions", "system prompt", "you are now", "<!--")
or tostring(Properties.retrieved_sources) has "suspicious-doc-id"
| summarize turns = count(), first = min(TimeGenerated), last = max(TimeGenerated) by user = tostring(Properties.user_id), conversation = tostring(Properties.conversation_id)
Also search by:
- the document/email/web page that carried the instructions (indirect),
- tool calls with unusual arguments (external URLs, email addresses, large data exports),
- input shield / guardrail detections that were logged but not blocked.
Step 3: assess impact
| Question | Evidence |
|---|---|
| What data was in context? | Retrieved chunks and tool results in affected conversations |
| Did any data leave? | Tool calls to external endpoints, emails sent, links rendered (data in URL parameters, e.g. Markdown images) |
| Were actions taken? | Write tool calls; target systems' audit logs |
| Did the system prompt contain secrets? | Review the prompt: API keys, internal URLs, names |
If secrets were in the prompt: rotate them now. Prompts should never hold secrets, but they sometimes do.
Involve security and data protection if personal or confidential data may have been exposed. They decide on notification.
Step 4: fix the specific hole
| Hole | Fix |
|---|---|
| Malicious content in a data source | Remove it; review who could write to that source |
| Markdown image/link exfiltration | Don't render external images/links from model output, or allowlist domains |
| Tool with too much reach | Narrow scope (allowed recipients/domains, read-only), require approval |
| No input/document shield | Add Prompt Shields / Guardrails on user input and retrieved documents |
| System prompt revealed | Accept it's not secret; remove anything sensitive from it |
Step 5: fix the class of problem
- Treat all retrieved and tool-returned content as untrusted data: separate it clearly in the prompt and tell the model never to follow instructions in it. That helps, but it is not a guarantee.
- Design so that a fully compromised model can't do serious harm: least-privilege tools, human approval for sensitive actions, egress restrictions, output filtering.
- Add the attack examples to your red-team test set and run it on every release.
Caution
There's no prompt that fully prevents prompt injection. Defences must assume the model can be manipulated and limit what a manipulated model can reach.
Step 6: close out
- Restore normal mode in stages after fixes are tested against the attack examples.
- Post-incident review: detection time, containment time, missing logs.
- Update the prompt injection defence checklist and threat model.
Prevent it next time
- Conversation, retrieval and tool-call logging with retention and access control.
- Input and document shields in front of the model, with detections monitored.
- Tool kill switches and safe mode ready (see the kill switch series).
- Regular red-team exercises including indirect injection via documents and email.