Home / OCI

OCIRunbookAdvanced

Sev1: OCI instance won't boot — repair the boot volume from a rescue instance

Recover an OCI compute instance that fails to boot after a bad fstab entry, kernel update, full disk or broken network config. Detach the boot volume, attach it to a rescue instance, fix it and put it back, with a backup first.

Deepak Bhardwaj3 min read

Applies toOCI Compute (VM shapes), Oracle Linux / Ubuntu / RHEL boot volumes, OCI CLI 3.x

Symptoms

  • Instance is RUNNING but unreachable; the serial console shows emergency mode, a kernel panic, or grub>.
  • The console connection and instance console history show boot errors after a change or patch.

0. Look first, then back up

oci compute console-history capture --instance-id $INST
oci compute console-history get-content --instance-console-history-id <id> --length 100000 --file - | tail -60

Often the serial console connection lets you fix it in place (single-user mode). If not, continue — and take a backup of the boot volume before touching it:

BV=$(oci compute boot-volume-attachment list --availability-domain $AD --compartment-id $C --instance-id $INST \
  --query 'data[0]."boot-volume-id"' --raw-output)
oci bv boot-volume-backup create --boot-volume-id $BV --display-name pre-repair-$(date +%F) --type FULL

1. Stop and detach the boot volume

oci compute instance action --instance-id $INST --action STOP --wait-for-state STOPPED
BVA=$(oci compute boot-volume-attachment list --availability-domain $AD --compartment-id $C --instance-id $INST \
  --query 'data[0].id' --raw-output)
oci compute boot-volume-attachment detach --boot-volume-attachment-id $BVA --force --wait-for-state DETACHED

2. Attach it to a rescue instance (same AD)

oci compute volume-attachment attach --instance-id $RESCUE --volume-id $BV --type paravirtualized \
  --wait-for-state ATTACHED

Use a rescue instance with the same OS family; XFS needs nouuid when the rescue host's own disk was cloned from the same image.

lsblk
sudo mkdir -p /mnt/broken
sudo mount -o nouuid /dev/sdb3 /mnt/broken      # check partition layout with lsblk first

On Oracle Linux images the root may be on LVM: run sudo vgscan && sudo vgchange -ay and mount the logical volume.

3. Fix the cause

Symptom Fix on the mounted volume
fstab entry for a missing block volume Edit /mnt/broken/etc/fstab: add _netdev,nofail or comment the line
New kernel panics chroot and set the previous kernel as default (grubby --set-default)
Disk full Clear /mnt/broken/var/log, package caches, core dumps
Network config broken Fix files under etc/sysconfig/network-scripts or etc/netplan
SELinux relabel loop Remove /mnt/broken/.autorelabel, or set SELINUX=permissive temporarily in etc/selinux/config 
for d in dev proc sys; do sudo mount --bind /$d /mnt/broken/$d; done
sudo chroot /mnt/broken grubby --info=ALL | grep -E '^(index|kernel)'
sudo chroot /mnt/broken grubby --set-default /boot/vmlinuz-<previous-version>
for d in sys proc dev; do sudo umount /mnt/broken/$d; done
sudo umount /mnt/broken

4. Detach from rescue and reattach as boot volume

VA=$(oci compute volume-attachment list --compartment-id $C --instance-id $RESCUE --volume-id $BV --query 'data[0].id' --raw-output)
oci compute volume-attachment detach --volume-attachment-id $VA --force --wait-for-state DETACHED
oci compute boot-volume-attachment attach --boot-volume-id $BV --instance-id $INST --wait-for-state ATTACHED
oci compute instance action --instance-id $INST --action START --wait-for-state RUNNING

5. If the volume can't be fixed

Restore from the latest boot volume backup into a new boot volume and launch (or attach) from it:

oci bv boot-volume create --boot-volume-backup-id <backup-ocid> --availability-domain $AD --compartment-id $C

Prevent it next time

  • nofail on every non-root mount; paravirtualized attachments where possible.
  • A boot volume backup policy (Bronze or better) on every production instance.
  • Create a local user with a password for serial console emergencies, stored in Vault.

Related fixes