Oracle RAC for high availability: services, connection failover and day-to-day operations

#What RAC gives you

Multiple instances on different servers open the same database on shared storage (ASM). If one node fails, the surviving instances keep serving and sessions can fail over. RAC does not protect against storage or site loss. For that, add Data Guard.

#1. Use services, never instance names

srvctl add service -db SALES -service sales_oltp -preferred SALES1,SALES2 \
  -failovertype TRANSACTION -commit_outcome TRUE -failoverretry 30 -failoverdelay 10 \
  -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE -role PRIMARY
srvctl add service -db SALES -service sales_batch -preferred SALES2 -available SALES1
srvctl start service -db SALES -service sales_oltp

• sales_oltp runs on both nodes (load balanced). sales_batch runs on node 2 and moves to node 1 if node 2 fails.
• -failovertype TRANSACTION plus -commit_outcome TRUE turns on Application Continuity, which replays in-flight transactions. AUTO gives Transparent Application Continuity (TAC) in 19c+.

#2. Client connection string

SALES_OLTP =
 (DESCRIPTION =
   (CONNECT_TIMEOUT=90)(RETRY_COUNT=20)(RETRY_DELAY=3)(TRANSPORT_CONNECT_TIMEOUT=3)
   (ADDRESS_LIST = (LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=sales-scan.corp.local)(PORT=1521)))
   (CONNECT_DATA = (SERVICE_NAME = sales_oltp)))

Connect through the SCAN name (three IPs in DNS). Use connection pools with FAN enabled (UCP, ODP.NET, OCI session pool) so dead connections are cleaned up immediately.

#3. Health checks

crsctl stat res -t                    # all cluster resources
crsctl check cluster -all
srvctl status database -db SALES -v
srvctl status service -db SALES
srvctl status scan_listener
olsnodes -n -s
asmcmd lsdg                           # disk groups: free space, offline disks

SELECT inst_id, instance_name, status FROM gv$instance;
SELECT inst_id, service_name, count(*) FROM gv$session GROUP BY inst_id, service_name;

#4. Rolling maintenance

srvctl stop instance -db SALES -node racnode1 -drain_timeout 300 -stopoption IMMEDIATE
# services move to node 2, sessions drain
# patch with opatchauto (rolling) or do OS maintenance
srvctl start instance -db SALES -node racnode1
srvctl relocate service -db SALES -service sales_batch -oldinst SALES2 -newinst SALES1   # if needed

#5. Common failure scenarios

#6. Performance considerations

• Avoid "hot blocks" shared across nodes: sequences with CACHE 1000 (and NOORDER), and reverse key or hash-partitioned indexes for right-growing keys.
• Watch gc wait events in AWR (gc buffer busy, gc cr request).
• Use services to keep each workload on its own nodes (affinity) where it helps.

Tip

RAC only gives seamless HA if the application uses services, FAN-aware pools and (ideally) Application Continuity. A JDBC thin URL with a fixed host name gets none of it.