Oracle RAC global cache waits cheat sheet: gc buffer busy, gc cr/current block waits, lost blocks and hot objects

#When to use this

A RAC database is slower than the same workload on one node, and gc wait events dominate AWR or ASH. The work is always the same: decide whether the interconnect is slow or the application makes instances fight over the same blocks. The two look alike in top-event lists and have opposite fixes.

#The wait events that matter

Rule of thumb: high average times on 2-way/3-way events with no busy or congested point to the network or LMS. High busy counts with normal 2-way times point to the application.

#Average block receive times per instance

SELECT inst_id,
       SUM(DECODE(name, 'gc cr blocks received', value))           AS cr_blocks,
       ROUND(10 * SUM(DECODE(name, 'gc cr block receive time', value)) /
             NULLIF(SUM(DECODE(name, 'gc cr blocks received', value)), 0), 2) AS avg_cr_ms,
       SUM(DECODE(name, 'gc current blocks received', value))      AS cur_blocks,
       ROUND(10 * SUM(DECODE(name, 'gc current block receive time', value)) /
             NULLIF(SUM(DECODE(name, 'gc current blocks received', value)), 0), 2) AS avg_cur_ms
FROM   gv$sysstat
WHERE  name IN ('gc cr blocks received', 'gc cr block receive time',
                'gc current blocks received', 'gc current block receive time')
GROUP  BY inst_id
ORDER  BY inst_id;

Receive times are in centiseconds, hence the * 10 for milliseconds. Values are cumulative since startup; for an incident use AWR deltas or sample twice. On a healthy private interconnect, averages are typically well under a millisecond; Exadata with RDMA is lower still. Compare against your own baseline.

#Lost and congested blocks

SELECT inst_id, name, value
FROM   gv$sysstat
WHERE  name IN ('gc blocks lost', 'gc blocks corrupt')
ORDER  BY inst_id, name;

Transfer quality by peer instance and block class:

SELECT inst_id, instance AS from_inst, class,
       cr_block, cr_busy, cr_congested,
       current_block, current_busy, current_congested
FROM   gv$instance_cache_transfer
WHERE  cr_block + current_block > 0
ORDER  BY cr_busy + current_busy DESC
FETCH FIRST 20 ROWS ONLY;

busy concentrated on data block points at hot table blocks; on undo header or undo block it points at hot rows being read while modified elsewhere.

#Which sessions are waiting right now

SELECT inst_id, event, COUNT(*) AS sessions,
       ROUND(AVG(seconds_in_wait), 1) AS avg_sec
FROM   gv$session
WHERE  wait_class = 'Cluster'
  AND  state = 'WAITING'
GROUP  BY inst_id, event
ORDER  BY sessions DESC;

#Hot objects and blocks from ASH

Last 30 minutes, top objects behind Cluster waits:

SELECT ash.current_obj#, o.owner, o.object_name, o.subobject_name, o.object_type,
       ash.event, COUNT(*) AS samples
FROM   gv$active_session_history ash
LEFT JOIN dba_objects o ON o.object_id = ash.current_obj#
WHERE  ash.sample_time > SYSTIMESTAMP - INTERVAL '30' MINUTE
  AND  ash.wait_class = 'Cluster'
GROUP  BY ash.current_obj#, o.owner, o.object_name, o.subobject_name, o.object_type, ash.event
ORDER  BY samples DESC
FETCH FIRST 15 ROWS ONLY;

Single hot blocks:

SELECT current_obj#, current_file#, current_block#, event, COUNT(*) AS samples
FROM   gv$active_session_history
WHERE  sample_time > SYSTIMESTAMP - INTERVAL '30' MINUTE
  AND  event LIKE 'gc buffer busy%'
GROUP  BY current_obj#, current_file#, current_block#, event
ORDER  BY samples DESC
FETCH FIRST 10 ROWS ONLY;

If a few blocks have most of the samples, it is a hot block problem. Look at the block to see whether it is the right edge of an index, a segment header or a single popular row.

Top SQL behind the waits:

SELECT sql_id, event, COUNT(*) AS samples
FROM   gv$active_session_history
WHERE  sample_time > SYSTIMESTAMP - INTERVAL '30' MINUTE
  AND  wait_class = 'Cluster'
GROUP  BY sql_id, event
ORDER  BY samples DESC
FETCH FIRST 10 ROWS ONLY;

For history beyond the ASH buffer, use dba_hist_active_sess_history with the same columns and sample_time filter.

#Segment-level statistics

SELECT owner, object_name, subobject_name, statistic_name, SUM(value) AS total
FROM   gv$segment_statistics
WHERE  statistic_name IN ('gc buffer busy', 'gc cr blocks received', 'gc current blocks received')
GROUP  BY owner, object_name, subobject_name, statistic_name
ORDER  BY total DESC
FETCH FIRST 20 ROWS ONLY;

This catches objects that are steadily expensive even outside an incident.

#Interconnect checks

SELECT inst_id, name, ip_address, is_public, source
FROM   gv$cluster_interconnects;

Confirm every instance uses the private network you expect, not the public one. Then on each node, as the Grid owner:

oifcfg getif
netstat -s | grep -iE 'reassembl|fragments|packet receive errors|buffer errors'
ip -s link show <private_nic>

Rising reassembly failures, receive buffer errors or interface drops line up with gc ... lost. Common causes: MTU mismatch along the path (jumbo frames on hosts but not the switch), undersized socket buffers (net.core.rmem_max), a bad NIC or cable, or a saturated bonded link. Check the LMS processes are not starved:

ps -eo pid,pri,ni,pcpu,comm | grep -E 'ora_lms|ora_lm'

Warning

Lost blocks can escalate to node evictions if the interconnect degrades further. Treat any non-zero, growing gc blocks lost as a network incident, not a tuning task.

#Symptom → cause → fix

#Prevent it next time

• Design services so each workload's writes land on one instance where possible, with others as available instances for failover.
• Baseline avg CR and current receive times per instance and alert on a sustained doubling or any growing lost-block count.
• Review AWR's "Global Cache Load Profile" and "Global Cache Efficiency" sections monthly.
• Avoid ORDER sequences and right-growing indexes on tables written from every node; test inserts from all instances before go-live.
• Keep interconnect MTU, NIC firmware and kernel network settings identical across nodes and check them after any network change.