Applies toSQL Server, Oracle, PostgreSQL, MySQL, MongoDB DR setups on-premises or in the cloud
1. Choose the test type
| Type | What happens | Risk | Frequency |
|---|---|---|---|
| Tabletop | Walk through the runbook on paper | None | Quarterly |
| Isolated test | Bring up a DR copy in an isolated network (snapshot, clone, or a paused standby) | Low | Quarterly |
| Planned switchover | Move production to DR with no data loss, run there, then switch back | Medium | Once or twice a year for Tier 1 |
| Full failover simulation | Cut off the primary, force failover | High | Yearly, if the business agrees |
2. Plan
- Scope: which databases, which apps, which tier.
- Success criteria: RTO ≤ X, RPO ≤ Y, app smoke tests pass, monitoring and backups work in DR.
- Window, change ticket, and approvals from app owners.
- Roles: DR lead, DBA per engine, network/DNS, app testers, and someone keeping notes and timestamps.
- Go/no-go criteria and an abort plan.
3. Pre-checks (T-1 day and T-0)
- Replication healthy, with lag close to zero (AG dashboard,
pg_stat_replication,v$dataguard_stats,SHOW REPLICA STATUS). - A recent backup on both sides.
- DR servers patched to the same level, with enough capacity (CPU, memory, licences).
- Logins, jobs, certificates, TDE keys and wallets present in DR.
- DNS TTLs lowered (for example to 60 s) a day before.
4. Execute and record times
| Step | Timestamp |
|---|---|
| T0: test starts / primary declared down | |
| Apps stopped or drained | |
| Role change issued | |
| Database open read-write in DR | |
| DNS or listener switched | |
| Apps started against DR | |
| Smoke tests passed: service restored |
RTO achieved = service restored − T0. RPO achieved = the last committed transaction on the old primary − the last transaction present in DR (use a heartbeat table that writes a timestamp every 10 seconds).
5. Validate in DR
- Application smoke tests (log in, read, write, a batch job).
- Integrations: interfaces, ETL, reporting.
- Monitoring shows the DR databases as the primaries; alerts work.
- Backups run successfully in DR.
- Run in DR for the agreed time (for example 4 hours, or a full business day).
6. Failback
- Make sure the old primary is resynchronised as a standby of DR (reinstate, reseed, or rewind).
- Wait for zero lag.
- Do a planned switchover back (no data loss) in the window.
- Repoint DNS, validate the apps again, and restore the normal backup schedule.
- Put DNS TTLs back.
7. Report and follow-up
- Measured RTO and RPO against the targets.
- Issues found (missing logins, hard-coded host names, firewall rules, capacity).
- Actions with owners and dates.
- Evidence attached: timestamps, screenshots or logs, sign-off from the app owner.
Tip
The most common drill finding is an application that can't reach DR because of a hard-coded host name, a firewall rule or a certificate name. Run a connectivity-only test from the app servers to the DR endpoints a week before the drill.