Sev1: OCI Base Database primary lost — Data Guard switchover or failover to the standby

#Decide: switchover or failover

Get the decision from the incident lead before failing over: it ends the current primary's role.

#1. Check Data Guard status

DB=ocid1.database.oc1..primary
oci db data-guard-association list --database-id $DB \
  --query 'data[].{id:id, role:role, peerRole:"peer-role", mode:"protection-mode", transport:"transport-type", apply:"apply-lag", rate:"apply-rate"}'

If you can reach the standby host, DGMGRL gives the clearest view:

dgmgrl /
DGMGRL> SHOW CONFIGURATION;
DGMGRL> SHOW DATABASE 'orcl_stby';
DGMGRL> VALIDATE DATABASE 'orcl_stby';

Note Apply Lag and Transport Lag. With Maximum Performance (ASYNC), transport lag is your likely data loss.

#2. Switchover (primary still available)

oci db data-guard-association switchover --database-id $DB \
  --data-guard-association-id <dg-assoc-ocid> \
  --database-admin-password '<sys-password>'

Or in DGMGRL: SWITCHOVER TO 'orcl_stby';

#3. Failover (primary lost)

Run it against the standby database's Data Guard association:

STBY=ocid1.database.oc1..standby
oci db data-guard-association failover --database-id $STBY \
  --data-guard-association-id <dg-assoc-ocid> \
  --database-admin-password '<sys-password>'

Or in DGMGRL on the standby:

DGMGRL> FAILOVER TO 'orcl_stby';

Caution

Never fail over while the old primary may still accept writes (for example, a network split where only your monitoring lost sight of it). Two primaries means diverged data. Make sure the old primary is down or isolated first.

#4. Repoint applications

• If apps use a connect string listing both hosts with role-based services (a service started only on the primary role), they reconnect automatically.
• Otherwise update DNS or connection strings to the new primary's scan/host.
• Check the service is running on the new primary:

SELECT name, open_mode, database_role FROM v$database;
SELECT name FROM v$active_services;

Example of a resilient connect descriptor:

(DESCRIPTION=(CONNECT_TIMEOUT=5)(RETRY_COUNT=3)(FAILOVER=ON)
  (ADDRESS_LIST=(LOAD_BALANCE=OFF)
    (ADDRESS=(PROTOCOL=TCP)(HOST=db-prim.sub.vcn.oraclevcn.com)(PORT=1521))
    (ADDRESS=(PROTOCOL=TCP)(HOST=db-stby.sub.vcn.oraclevcn.com)(PORT=1521)))
  (CONNECT_DATA=(SERVICE_NAME=orders_rw.sub.vcn.oraclevcn.com)))

#5. Reinstate the old primary

When the old primary is back (Flashback Database must have been on):

oci db data-guard-association reinstate --database-id <old-primary-ocid> \
  --data-guard-association-id <dg-assoc-ocid> --database-admin-password '<sys-password>'

Or REINSTATE DATABASE 'orcl_prim'; in DGMGRL. Without Flashback Database, you must re-create the standby.

#6. Afterwards

• Confirm redo transport and apply are running toward the new standby.
• Check backups are configured on the new primary.
• Record actual RTO and data loss.

#Prevent it next time

• Turn on Flashback Database on both sides so reinstate works.
• Use role-based services and multi-address connect strings in every app.
• Practice switchover quarterly.
• Alarm on apply lag and transport lag.