OCI DNS failover with a steering policy and Health Checks: build it, drill it and fail over on purpose

#When to use this

You have the same public endpoint in two regions, for example a load balancer in each, and you want DNS to send clients to the secondary when the primary fails, without someone editing records at 03:00. Steering policies only work on public zones. Attachments to private zones are rejected, so internal failover needs a different design.

Expectations to set up front:

• DNS failover is only as fast as the health check interval plus the TTL, plus any resolver that ignores the TTL. Plan for minutes, not seconds.
• Long-lived connections (database pools, gRPC, WebSockets) stay on the old endpoint until they reconnect.
• The steering policy only moves clients to the secondary. Data replication, state and capacity in the secondary region are your problem.

#Step 1: Create the health check monitor

The monitor needs to probe something that reflects real application health, such as a /healthz that checks downstream dependencies. A TCP port that is always open doesn't tell you anything.

oci health-checks http-monitor create \
  -c <compartment-ocid> \
  --display-name app-prod-failover \
  --targets '["203.0.113.10","198.51.100.20"]' \
  --protocol HTTPS --port 443 --path /healthz --method GET \
  --interval-in-seconds 30 --timeout-in-seconds 10 \
  --headers '{"Host":"app.example.com"}'

• Interval must be 10, 30 or 60 seconds, and the timeout must not exceed the interval.
• The Host header matters when the load balancer routes by hostname or SNI. Without it, the probe can hit the default listener and report healthy for the wrong application.
• --vantage-point-names is optional, and OCI picks vantage points if you leave it out. List them with oci health-checks vantage-point list if you need probes from particular geographies.
• Your WAF and the load balancer's NSG must allow the vantage points. Health check probes that are blocked look like an outage and trigger a failover.

oci health-checks http-probe-result list --probe-configuration-id <monitor-ocid> \
  --query 'data[0:10].{t:"start-time",target:target,healthy:"is-healthy",status:"status-code",vp:"vantage-point-name"}' \
  --output table

Wait until both targets report healthy from several vantage points before you continue.

#Step 2: Create the FAILOVER steering policy

Templates other than CUSTOM must use the rule order FILTER → HEALTH → PRIORITY → LIMIT, or the API returns 400. PRIORITY uses the answer pool, and each pool gets a unique value.

cat > answers.json <<'EOF'
[
  {"name":"primary-lb",   "rtype":"A","rdata":"203.0.113.10", "pool":"primary",  "isDisabled":false},
  {"name":"secondary-lb", "rtype":"A","rdata":"198.51.100.20","pool":"secondary","isDisabled":false}
]
EOF

cat > rules.json <<'EOF'
[
  {"ruleType":"FILTER","defaultAnswerData":[{"answerCondition":"answer.isDisabled != true","shouldKeep":true}]},
  {"ruleType":"HEALTH"},
  {"ruleType":"PRIORITY","defaultAnswerData":[
     {"answerCondition":"answer.pool == 'primary'","value":1},
     {"answerCondition":"answer.pool == 'secondary'","value":99}]},
  {"ruleType":"LIMIT","defaultCount":1}
]
EOF

oci dns steering-policy create -c <compartment-ocid> \
  --display-name app-prod-failover --template FAILOVER --ttl 60 \
  --health-check-monitor-id <monitor-ocid> \
  --answers file://answers.json --rules file://rules.json \
  --wait-for-state ACTIVE

Warning

Answers whose rdata doesn't match a monitored target are treated as healthy. If you change the load balancer IP and forget to update the monitor's targets, the policy keeps serving a dead address and the health check never fails.

TTL trade-off: 30 to 60 seconds keeps failover fast but raises query volume and cost. Above 300 seconds, a failover can take longer than many outages last. Choose the TTL based on your recovery time objective, not by default.

#Step 3: Attach the policy to the zone

oci dns steering-policy-attachment create \
  --steering-policy-id <policy-ocid> \
  --zone-id <public-zone-ocid> \
  --domain-name app.example.com \
  --display-name app-prod-failover

Caution

The attachment takes over the domain name. Any existing A record for app.example.com in the zone stops being served. Lower the old record's TTL a day ahead, attach during a quiet window, and check the result straight away.

dig +short app.example.com @<zone-nameserver>    # authoritative answer, no cache
dig app.example.com | grep -A1 "ANSWER SECTION"  # recursive, shows remaining TTL

#Step 4: Drill an automatic failover

Make the primary fail health checks without touching the secondary. Break the health endpoint, not the network, so the test stays reversible:

1. Switch the primary application's /healthz to return 503, or take the primary backend set's members out of service.
2. Watch the probe results turn unhealthy for the primary target:

watch -n 15 "oci health-checks http-probe-result list --probe-configuration-id <monitor-ocid> \
  --query 'data[0:6].{target:target,healthy:\"is-healthy\",vp:\"vantage-point-name\"}' --output table"

3. Query the authoritative nameserver until it returns the secondary IP. Record how long it took.
4. Query through a public resolver and from your clients' networks, then compare the times. The gap between them is TTL and resolver caching.
5. Check the application through the secondary: sign-in, writes and background jobs.

#Step 5: Planned failover and failback (manual override)

For maintenance or a region evacuation, disable the primary answer rather than waiting for health checks. The FILTER rule drops disabled answers.

oci dns steering-policy get --steering-policy-id <policy-ocid> \
  --query 'data.answers' > current-answers.json
# edit: set "isDisabled": true on primary-lb (keys may be kebab-case in output; convert to the input format)
oci dns steering-policy update --steering-policy-id <policy-ocid> \
  --answers file://answers-primary-disabled.json --rules file://rules.json --force

Update replaces the answers and rules, so always send the complete set of both. If you send only the answers, the policy ends up without the rules you expected.

Failback order:

1. Make sure data has caught up in the primary region (database role, replication lag and queues drained).
2. Set isDisabled: false on the primary answer, then send the update with the full answers and rules.
3. Expect clients to move back over one TTL plus caching. Old connections drain on the secondary.

Tip

If the database fails over separately, for example through Data Guard, plan the DNS move and the database role change in a single runbook. Without coordination, applications in the secondary can end up writing to the old primary, or the reverse.

#Symptom → cause → fix

#Prevent it next time

• Keep answers.json, rules.json and the monitor definition in Git, or use the Terraform oci_dns_steering_policy resource. Don't edit the policy by hand in the Console.
• Alarm on probe failures through Monitoring and Notifications, so a failover pages someone instead of happening unnoticed.
• Drill every quarter, and record detection time, authoritative switch time and client switch time.
• Whenever a load balancer IP changes, update the answers and the monitor targets in the same change.