Home / OCI

OCIRunbookAdvanced

Sev1: OCI compute instance hung and unreachable — serial console, diagnostic reboot and safe recovery

A production instance stops answering SSH/RDP and its apps are down. Check the metrics, get onto the serial console, collect evidence, then reset or move it to new hardware.

Deepak Bhardwaj3 min read

Applies toOCI Compute (VM shapes, Oracle Linux, Ubuntu, Windows), OCI CLI 3.x

Symptoms

  • SSH or RDP times out; the instance still shows RUNNING.
  • Monitoring agent metrics stop or CPU is pinned at 100%.
  • Load balancer marks the backend critical.

1. Check what OCI sees

INST=ocid1.instance.oc1..aaaa
oci compute instance get --instance-id $INST --query 'data.{state:"lifecycle-state", shape:shape, ad:"availability-domain"}'

Look at the instance metrics (CpuUtilization, MemoryUtilization, DiskBytesRead) in Monitoring for the last hour:

oci monitoring metric-data summarize-metrics-data --compartment-id $COMP --namespace oci_computeagent \
  --query-text "MemoryUtilization[1m]{resourceId = \"$INST\"}.mean()" \
  --start-time "$(date -u -d '-1 hour' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)"

Memory at 100% followed by silence usually means the OS is thrashing or the OOM killer is busy. Also check for infrastructure maintenance or hardware events on the instance in the console.

2. Connect to the serial console

Create a console connection with your SSH public key:

oci compute instance-console-connection create --instance-id $INST \
  --ssh-public-key-file ~/.ssh/id_rsa.pub --wait-for-state ACTIVE
oci compute instance-console-connection list --compartment-id $COMP --instance-id $INST \
  --query 'data[0]."connection-string"' --raw-output

Run the returned SSH command. Press Enter a few times. You may see:

  • Kernel messages such as Out of memory: Killed process or hung_task_timeout_secs (I/O stalls).
  • A login prompt: log in with a local password user if one exists.
  • Nothing at all: the kernel is wedged.

Tip

Set a local emergency password user and enable the serial console in the OS (it is on by default on OCI platform images) before you need it. Without it, the console shows output but you can't log in.

3. Capture evidence if you can

From the console, before rebooting:

dmesg -T | tail -100
free -m; df -h; uptime
ps aux --sort=-%mem | head -15
journalctl -p err -b --no-pager | tail -50

Save the console output for the post-incident review.

4. Reset the instance

Try a graceful reboot first, then a hard reset:

oci compute instance action --instance-id $INST --action SOFTRESET --wait-for-state RUNNING
# if it doesn't come back within a few minutes
oci compute instance action --instance-id $INST --action RESET --wait-for-state RUNNING

If you need a crash dump for root cause, OCI supports sending a diagnostic interrupt (SENDDIAGNOSTICINTERRUPT) to trigger a kernel panic and kdump, if kdump is configured.

oci compute instance action --instance-id $INST --action SENDDIAGNOSTICINTERRUPT

5. If reset doesn't help: move to new hardware

A stop and start (not reset) can place the VM on a different host:

oci compute instance action --instance-id $INST --action SOFTSTOP --wait-for-state STOPPED
oci compute instance action --instance-id $INST --action START --wait-for-state RUNNING

Warning

Instances with an ephemeral public IP may get a new address after stop/start. Reserved public IPs and private IPs are kept. Check before telling users to reconnect.

If the boot volume is damaged, follow the rescue-instance repair process: stop, detach the boot volume, attach it to a helper instance, fix, reattach.

6. After recovery

  • Confirm apps, mounts and agents are back; check the load balancer marks the backend OK.
  • Look at why: memory leak, runaway process, full disk, kernel bug. Add the fix (limits, swap policy, patch).

Prevent it next time

  • Alarms on memory, disk and agent heartbeat, not just CPU.
  • Keep kdump configured on critical Linux instances.
  • Run critical services on at least two instances behind a load balancer across fault domains.
  • Keep a documented console-access user with a vaulted password.

Related fixes