Oracle Data Guard Fast-Start Failover readiness checklist: observers, thresholds, lag limits, flashback and reinstatement

#Why a checklist

Fast-Start Failover (FSFO) lets a Data Guard observer fail over to the standby without a human. When it is set up well, a primary site loss turns into a short outage with automatic reinstatement. When it is set up badly, you get failovers on network blips, data loss you did not plan for, or an old primary that cannot rejoin. Every item below is something we have seen go wrong. Work through it before enabling FSFO and again after any topology change.

Commands assume DGMGRL is connected with a wallet (dgmgrl /@prim_dgmgrl) and the broker is running (dg_broker_start = TRUE) on all members.

#1. The broker configuration is healthy

• SHOW CONFIGURATION returns SUCCESS with no warnings.
• VALIDATE DATABASE 'stby' ; reports Ready for Switchover: Yes and Ready for Failover: Yes.
• VALIDATE DATABASE VERBOSE 'stby'; shows no standby redo log gaps: one more standby redo log group per thread than online groups, same size.
• A switchover and switchback have been done in this configuration in the last quarter.

DGMGRL> SHOW CONFIGURATION VERBOSE;
DGMGRL> VALIDATE DATABASE VERBOSE 'stby';
DGMGRL> VALIDATE NETWORK CONFIGURATION FOR ALL;

VALIDATE NETWORK CONFIGURATION exists from 18c. On 12.2, test connectivity with tnsping and the StaticConnectIdentifier from each host instead.

#2. Protection mode matches the data loss you accept

• The mode is a written decision with an RPO behind it, not the default.
• For MaxPerformance, FastStartFailoverLagLimit (seconds) matches that RPO.
• For SYNC, the commit latency added by the round trip to the standby has been measured under peak load.

DGMGRL> EDIT DATABASE 'stby' SET PROPERTY LogXptMode = 'SYNC';
DGMGRL> EDIT DATABASE 'prim' SET PROPERTY LogXptMode = 'SYNC';
DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MaxAvailability;
-- or, for MaxPerformance:
DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverLagLimit = 30;

#3. Flashback Database is on everywhere

Automatic reinstatement of the failed primary uses Flashback Database. Without it, every failover ends with a manual rebuild.

• SELECT flashback_on FROM v$database; returns YES on the primary and every FSFO target.
• db_flashback_retention_target and the fast recovery area are sized so the old primary can flash back far enough after a long outage.
• FRA usage is monitored (v$recovery_area_usage). A full FRA stops flashback logging and can block reinstatement.

ALTER DATABASE FLASHBACK ON;
ALTER SYSTEM SET db_flashback_retention_target = 1440 SCOPE=BOTH;  -- minutes

#4. Targets are explicit in both directions

• Each database names its FSFO target, so after a role change the new primary knows where to fail over.

DGMGRL> EDIT DATABASE 'prim' SET PROPERTY FastStartFailoverTarget = 'stby';
DGMGRL> EDIT DATABASE 'stby' SET PROPERTY FastStartFailoverTarget = 'prim';

• With several standbys, the list is ordered by preference (comma-separated) and the far-DR standby is not the first target.

#5. Thresholds fit your network

DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverThreshold = 45;
DGMGRL> EDIT CONFIGURATION SET PROPERTY ObserverReconnect = 30;

Warning

A threshold set lower than the time a RAC primary needs to evict a node and reconfigure can turn a single-node failure into a full site failover. Measure an eviction in test before you go below the default.

#6. Failover conditions are chosen on purpose

• SHOW FAST_START FAILOVER reviewed: the health conditions listed (for example Corrupted Controlfile, Corrupted Dictionary, Inaccessible Logfile, Stuck Archiver, Datafile Write Errors) are the ones you want.
• Any ORA- errors added as conditions are rare and fatal, not errors your application raises in normal operation.
• If the application should be able to request a failover, DBMS_DG.INITIATE_FS_FAILOVER is documented and restricted.

DGMGRL> SHOW FAST_START FAILOVER;
DGMGRL> ENABLE FAST_START FAILOVER CONDITION "Stuck Archiver";
DGMGRL> DISABLE FAST_START FAILOVER CONDITION "Inaccessible Logfile";

#7. Observers are placed and run like production

• The master observer runs at a third site, not with the primary or the standby. An observer next to the primary fails with it; next to the standby, a link failure between sites looks like a primary loss.
• Up to three observers (12.2+) are configured, with backups in different failure domains.
• Each observer host has an Oracle Client install that includes DGMGRL, at a version supported with your database release.
• Credentials come from a wallet, not a password in a script.
• The observer is started by systemd or a scheduler and restarts after a host reboot.

DGMGRL> START OBSERVER obs_site3 IN BACKGROUND
          FILE IS '/u01/app/oracle/dg/fsfo_obs_site3.dat'
          LOGFILE IS '/u01/app/oracle/dg/obs_site3.log'
          CONNECT IDENTIFIER IS prim_dgmgrl;
DGMGRL> SHOW OBSERVER;
DGMGRL> SET MASTEROBSERVER TO obs_site3;

#8. Clients follow the primary

FSFO is only half the job if applications keep connecting to the old host.

• Applications use a role-based service that starts only on the primary (srvctl add service ... -role PRIMARY with Oracle Restart or RAC, or a startup trigger on single instance).
• Connect strings list both sites with FAILOVER=ON, sensible CONNECT_TIMEOUT, TRANSPORT_CONNECT_TIMEOUT, RETRY_COUNT and RETRY_DELAY.
• FAN, Application Continuity or Transparent Application Continuity is configured where the driver supports it.
• DNS, firewall rules and connection pools on the application side have been tested after a role change.

#9. Monitoring sees FSFO state

SELECT database_role, fs_failover_mode, fs_failover_status,
       fs_failover_current_target, fs_failover_threshold,
       fs_failover_observer_present, fs_failover_observer_host
FROM   v$database;

• Alert when fs_failover_observer_present = 'NO', or the status is anything other than SYNCHRONIZED (MaxAvailability) or TARGET UNDER LAG LIMIT (MaxPerformance).
• Alert on any role change, read from the broker log or the alert log.
• Apply and transport lag from v$dataguard_stats is graphed.

#10. Rehearse before trusting it

• Observe-only first (19c+). Run with ENABLE FAST_START FAILOVER OBSERVE ONLY for a few weeks. The observer logs when it would have failed over, without doing it. Every logged event is either a real outage or a threshold to tune.
• Then a real test in a window: SHUTDOWN ABORT the primary (a NORMAL, IMMEDIATE or TRANSACTIONAL shutdown does not trigger FSFO). Time detection, failover, application reconnection and automatic reinstatement.
• Record the result: seconds of outage, transactions lost (if any), and anything done by hand.

DGMGRL> ENABLE FAST_START FAILOVER OBSERVE ONLY;
-- after the trial period
DGMGRL> DISABLE FAST_START FAILOVER;
DGMGRL> ENABLE FAST_START FAILOVER;
DGMGRL> SHOW CONFIGURATION;

Caution

With FSFO enabled, killing the primary instance, pulling its network or stopping its storage during maintenance will cause a real failover. Disable FSFO (DISABLE FAST_START FAILOVER) before unplanned-looking maintenance, and put it back afterwards as a checklist step.

#Prevent drift

• Re-run this checklist after adding a standby, changing a network path, moving observers or upgrading the database or client.
• Keep the DGMGRL property settings in version control as a script (SHOW CONFIGURATION VERBOSE output plus the EDIT commands) so a rebuilt configuration ends up the same.
• Review the observer log monthly. Repeated "primary unreachable" events that did not reach the threshold are early warnings about the network.