Applies toRAG apps, chat assistants and agents on Azure, AWS, OCI or self-hosted infrastructure
1. Inventory: state versus rebuildable
| Component | Holds state? | DR approach | Typical RPO |
|---|---|---|---|
| App code, infrastructure, gateway policies | No (in Git) | Redeploy with IaC to the DR region | 0 |
| Prompts and model configuration | In Git or a prompt registry | Git; replicate the registry | 0 |
| Source documents | Yes | Storage geo-replication / object replication | Minutes |
| Vector index | Derived | Rebuild from source, or snapshot and restore (see the vector index DR post) | Hours (rebuild) / snapshot age |
| Conversation history, user feedback | Yes | Database geo-replication (Cosmos DB multi-region, Aurora Global, etc.) | Seconds to minutes |
| Agent state, memory, threads | Yes | Same as conversation storage | Seconds to minutes |
| Evaluation sets and results | Yes | Git or replicated storage | 0–1 day |
| Fine-tuned models / adapters | Yes | Keep the training data and recipe; copy model artifacts to the DR region | Last training run |
| Model endpoints (managed) | Provider-side | A deployment in the DR region, created in advance (quota!) | n/a |
| Secrets and keys | Yes | Key Vault / Secrets Manager replication, or recreate from the source | n/a |
| Logs and traces | Yes (audit) | Ship to a central store; retention policy | Minutes |
2. Pre-provision what can't be created quickly
- Model quota and deployments in the DR region. Quota for popular models can't always be obtained during an incident.
- GPU capacity for self-hosted models (reservations or an approved fallback to a managed model).
- Private endpoints, DNS zones and firewall rules for the DR resources.
- Content filter and guardrail policies matching production.
3. Recovery order
- Network, identity, secrets.
- Data stores: source documents, conversation and agent state (fail over or promote).
- Vector index (restore a snapshot, or start a rebuild; serve a reduced corpus first if needed).
- Model endpoints (the DR deployments) and the gateway configuration pointing to them.
- The application tier.
- Smoke tests with the golden evaluation set: quality must be close to production.
- DNS / Front Door switch.
4. RTO drivers
- Index rebuild time = number of chunks ÷ embedding throughput (TPM quota in DR). For 5 million chunks, this can be many hours, which is why snapshots or a warm standby index exist.
- Self-hosted model start time (downloading weights, GPU capacity).
5. Decide per tier
| Tier | Pattern |
|---|---|
| Internal pilot | Redeploy and rebuild in DR (RTO 1–2 days) |
| Business-important | Warm: DR model deployments + replicated data + an index snapshot every few hours (RTO hours) |
| Critical customer-facing | Active-active or hot standby in two regions with replicated indexes (RTO minutes) |
6. Test
- A yearly DR drill: run the app in the DR region from scratch using only the runbook.
- Measure the index rebuild or restore time, and the eval score in DR.
Tip
Conversation history and agent memory are often forgotten in GenAI DR plans. Users notice immediately when "the assistant forgot everything" after a failover.