Home / OCI

OCIRunbookIntermediate

OCI instance unreachable over SSH: recovering with the serial console

Use an instance console connection to get into a Linux instance that won't boot or accept SSH, then fix fstab, sshd or firewall problems from maintenance mode.

Deepak Bhardwaj3 min read

Applies toOCI Compute (VM shapes), Oracle Linux 8/9, Ubuntu

1. Rule out the network first

  • The instance is RUNNING, and the console screenshot (Console → Instance → Console connection → Capture console history) shows a login prompt or a boot error.
  • The security list or NSG allows TCP 22 from the client's IP. The subnet routes to an Internet Gateway (public) or you connect through a bastion or VPN.
  • A host firewall (firewalld on Oracle Linux) still allows SSH.

The OCI Bastion service gives a managed SSH path to private instances without public IPs.

2. Create a console connection

oci compute instance-console-connection create \
  --instance-id ocid1.instance.oc1..xxxx \
  --ssh-public-key-file ~/.ssh/id_rsa.pub
oci compute instance-console-connection get --instance-console-connection-id <id> \
  --query 'data."connection-string"' --raw-output

Run the returned SSH command (it goes through a console proxy). You get the serial console of the instance. Press Enter to see a login prompt.

Note

Delete the console connection when you're done. It's an access path into the instance.

3. Boot into maintenance mode (Linux)

If you can't log in at the prompt (lost key, broken PAM, sshd not starting):

  1. Reboot the instance (Console → Reboot), keeping the serial console session open.
  2. When the GRUB menu appears, press e on the default entry.
  3. On the line starting with linux, append: init=/bin/bash, then boot with Ctrl+X.
  4. Remount root as writable and fix the problem:
mount -o remount,rw /
# Typical fixes:
vi /etc/fstab                              # comment out a missing volume, or add 'nofail'
vi /etc/ssh/sshd_config && sshd -t         # fix syntax errors
cat ~/.ssh/id_rsa.pub >> /home/opc/.ssh/authorized_keys   # (paste key) re-add a key
chmod 700 /home/opc/.ssh; chmod 600 /home/opc/.ssh/authorized_keys; chown -R opc:opc /home/opc/.ssh
touch /.autorelabel                        # SELinux systems, after editing files this way
exec /sbin/init                            # or reboot -f

4. Common boot failures

Console shows Cause Fix
Dependency failed for /data, emergency mode Block volume detached or renamed, still in fstab Use _netdev,nofail and consistent device paths (/dev/oracleoci/oraclevdb)
Kernel panic - not syncing after update Bad kernel or initramfs Pick the previous kernel in GRUB, then fix or remove the new one
SSH works, then disconnects Disk full, or user's shell missing df -h, then clean up or grow the boot volume

5. Can't get in at all

Stop the instance, detach the boot volume, attach it to a healthy instance as a block volume, mount it, fix the files, then reattach it as the boot volume. Take a boot volume backup before you start.

Prevent it next time

  • Keep the Bastion service set up for private access, and test it regularly.
  • Use nofail for every non-root mount.
  • Snapshot (boot volume backup) before OS upgrades.

Related fixes