Sev1: OCI Linux instance boots into emergency mode after a reboot. Bad fstab or missing volume

#Symptoms

• Instance shows Running in the console, but SSH times out and the application is down.
• Console history or serial console shows: You are in emergency mode. After logging in, type "journalctl -xb" to view system logs... or A start job is running for dev-disk-by\x2duuid... followed by Dependency failed for /data.
• Often after a reboot that followed patching, a block volume detach, or an edit of /etc/fstab.

#Step 1: confirm from console history (2 minutes)

HIST=$(oci compute console-history capture --instance-id $INSTANCE --query 'data.id' --raw-output)
sleep 20
oci compute console-history get-content --instance-console-history-id $HIST --file - | tail -60

Look for the mount unit that failed and the device it waited for.

#Step 2: get a serial console session

oci compute instance-console-connection create --instance-id $INSTANCE --ssh-public-key-file ~/.ssh/id_ed25519.pub
oci compute instance-console-connection list --compartment-id $COMP --instance-id $INSTANCE \
  --query 'data[0]."connection-string"' --raw-output

Run the printed SSH command (it goes through the console proxy). Press Enter to get the emergency shell. You need the root password, or boot into single-user mode as below.

#No root password

Reboot from the console connection and interrupt GRUB:

1. oci compute instance action --instance-id $INSTANCE --action RESET
2. In the serial console, press Esc/arrow at the GRUB menu, press e on the default entry.
3. At the end of the linux line add rd.break (RHEL family) or init=/bin/bash (generic), then Ctrl-X.
4. Remount the root filesystem read-write:

mount -o remount,rw /sysroot && chroot /sysroot     # after rd.break
# or with init=/bin/bash:
mount -o remount,rw /

#Step 3: fix the mount

cat /etc/fstab
lsblk -f
blkid

Make a backup and fix it:

cp /etc/fstab /etc/fstab.bak.$(date +%F)
vi /etc/fstab                       # add nofail, or comment the failing line with #
# example line after fix:
# UUID=3f1c...  /data  xfs  defaults,_netdev,nofail  0 2

On SELinux systems booted with rd.break, trigger a relabel before rebooting if you edited files: touch /.autorelabel.

Then:

systemctl daemon-reload 2>/dev/null; exit   # leave chroot / emergency shell
reboot -f

#Step 4: if you cannot get a console shell

Use the boot volume on a rescue instance:

1. Stop the instance.
2. Detach its boot volume and attach it as a block volume to a working instance in the same AD.
3. Mount the root partition (for XFS add -o nouuid if the rescue VM has the same image), fix etc/fstab on it.
4. Unmount, detach, reattach as boot volume, start.

oci compute boot-volume-attachment list --availability-domain "$AD" --compartment-id $COMP --instance-id $INSTANCE
oci compute boot-volume-attachment detach --boot-volume-attachment-id $BVA --force --wait-for-state DETACHED
oci compute volume-attachment attach --type paravirtualized --instance-id $RESCUE --volume-id $BOOT_VOLUME

#Step 5: bring the data volume back

If the volume was legitimately detached, reattach it and mount:

oci compute volume-attachment attach --type paravirtualized --instance-id $INSTANCE --volume-id $DATA_VOLUME \
  --device /dev/oracleoci/oraclevdb --wait-for-state ATTACHED
sudo mount -a && df -h /data

#Step 6: confirm

• SSH works; the application is up; systemctl --failed shows no failed mount units.
• findmnt --verify reports no errors.

sudo findmnt --verify --verbose

#Prevent it next time

• Every non-root fstab entry has nofail; iSCSI entries also _netdev.
• Run findmnt --verify and mount -a after any fstab change, before rebooting.
• Keep a root password or console-access procedure documented for each production image.
• Patch with a rolling reboot and check the first server's console before continuing.