Home / Databases / Oracle

DatabasesRunbookAdvanced

Sev1: Oracle RAC node evicted and Clusterware won't start — interconnect, voting disks and CSS logs

A RAC node rebooted itself or was evicted and its instance won't come back. Read the CSS and alert logs, check the private interconnect and voting disks, restart CRS, and keep the surviving nodes healthy.

Deepak Bhardwaj3 min read

Applies toOracle Grid Infrastructure and RAC 19c / 23ai on Linux

Symptoms

  • One node rebooted unexpectedly; the surviving nodes logged a node eviction.
  • On the evicted node, crsctl check crs shows CSS or CRS not running; the database instance is down.
  • Services failed over to surviving instances, so capacity is reduced. If more nodes are lost, the whole database goes down.

1. Check cluster state from a surviving node

crsctl check cluster -all
crsctl stat res -t
olsnodes -n -s -t
srvctl status database -d ORCL

2. Find why the node was evicted

Clusterware alert log and CSS trace on both the evicted node and a survivor:

less $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/alert.log
grep -iE "evict|heartbeat|missed|voting|reboot" $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/ocssd.trc | tail -50
Log text Cause
network heartbeat misses / has a disk HB, but no network HB Private interconnect problem
voting file ... is not accessible / disk timeouts Shared storage / multipath issue
CSSD ... missed checkins / OS hung CPU or memory starvation, swap storm
rebootless restart messages Clusterware tried to restart the stack without reboot

Also check the OS log (/var/log/messages or journalctl -k) around the eviction time for NIC resets, multipath errors and OOM kills. Oracle's Cluster Health Monitor data (via oclumon) shows CPU and memory just before the event.

3. Check the interconnect

oifcfg getif
ip -s link show <private-interface>
ping -c 20 -s 8972 -M do <other-node-private-ip>    # jumbo frames end-to-end check (if MTU 9000)

Packet loss, errors on the interface or a jumbo frame mismatch (MTU 9000 on hosts, 1500 on a switch) are common causes after network changes.

4. Check voting disks and OCR

crsctl query css votedisk
ocrcheck
ls -l /dev/mapper/ | head      # or your ASM disk paths
asmcmd lsdsk -k

All voting files should be online. If a path is missing on one node, fix multipath/ASM disk discovery before restarting the stack.

5. Restart the stack on the evicted node

As root:

crsctl start crs
crsctl check crs
crsctl stat res -t -init

Then start the instance and relocate services back if they don't move automatically:

srvctl start instance -d ORCL -i ORCL2
srvctl relocate service -d ORCL -s orders_svc -i ORCL1 -t ORCL2

Warning

Don't repeatedly force the stack to start while the interconnect or storage problem is still there. The node will be evicted again, and in a two-node cluster, a misbehaving node can cause disruption for the survivor too. Fix the cause first.

6. If CRS won't start

crsctl stat res -t -init

Look at which init resource fails (ora.cssd, ora.asm, ora.crsd) and its trace. Common fixes: correct permissions on Grid home after an OS patch, remount the GPnP profile path, or fix /etc/hosts / DNS for the private names.

7. After recovery

  • Confirm both instances are open and services are balanced.
  • Raise a Support SR with the TFA collection for the eviction time:
tfactl diagcollect -from "Oct/31/2026 07:30:00" -to "Oct/31/2026 08:30:00"

Prevent it next time

  • Redundant, bonded private interconnect NICs on separate switches; consistent MTU end to end.
  • Three voting disks on separate failure groups.
  • Enough memory headroom: HugePages for the SGA and no swap storms.
  • Monitor interconnect errors and CSS messages, not just instance status.

Related fixes