Home / OCI

OCITroubleshootingIntermediate

Sev1: OCI block volume missing after reboot — iSCSI not reconnected, app data gone from the mount

Fix applications that start with an empty data directory because an iSCSI-attached OCI block volume didn't reconnect after a reboot. Reconnect the target, make it automatic, harden fstab, and consider paravirtualized attachments.

Deepak Bhardwaj2 min read

Applies toOCI Block Volume (iSCSI and paravirtualized attachments), Oracle Linux 7/8/9, Ubuntu, Oracle Cloud Agent Block Volume Management plugin

Symptoms

  • After patching or a reboot, the app starts but its data is missing, or the database fails because its files aren't there.
  • lsblk doesn't show the data disk; /u01 or /data is empty — or the instance dropped into emergency mode because of a hard fstab entry.

Warning

If the app started and wrote to the empty mount point, it wrote to the root disk. Stop the app before remounting, or you'll hide those writes and confuse recovery.

1. Is the volume attached in OCI?

oci compute volume-attachment list --compartment-id $C --instance-id $INST \
  --query 'data[].{vol:"volume-id", type:"attachment-type", state:"lifecycle-state", iqn:iqn, ip:ipv4, port:port, device:device}' --output table

ATTACHED with type iscsi means OCI side is fine — the guest just isn't logged in to the target.

2. Reconnect the iSCSI target

Use the IQN, IP and port from step 1 (the console's iSCSI commands & information shows the same commands):

sudo iscsiadm -m node -o new -T iqn.2015-12.com.oracleiaas:abcd... -p 169.254.2.2:3260
sudo iscsiadm -m node -o update -T iqn.2015-12.com.oracleiaas:abcd... -n node.startup -v automatic
sudo iscsiadm -m node -T iqn.2015-12.com.oracleiaas:abcd... -p 169.254.2.2:3260 -l

lsblk
ls -l /dev/oracleoci/

3. Mount and start the app

sudo mount -a
df -h /data
sudo systemctl start myapp

If the app wrote to the empty mount point while the volume was missing, move those files aside first (mount --bind / /mnt/root lets you see what's under the mount point).

4. Make it survive the next reboot

  • node.startup set to automatic (above).
  • iscsid / iscsi services enabled: sudo systemctl enable --now iscsid iscsi.
  • fstab with consistent device paths and network-safe options:
/dev/oracleoci/oraclevdb1  /data  xfs  defaults,_netdev,nofail  0 2

_netdev waits for the network (needed for iSCSI); nofail lets the OS boot even if the volume is missing, so you can still SSH in.

5. Let the agent do it, or avoid iSCSI

  • Enable the Block Volume Management plugin of the Oracle Cloud Agent: it logs in to iSCSI targets automatically for attached volumes.
  • For new attachments, prefer paravirtualized where supported: no iSCSI login needed, volumes appear at boot. iSCSI can give higher performance for some high-performance volume configurations — check which you need.

6. Protect the app

Add a startup check so the app refuses to start if its data mount isn't present:

# /etc/systemd/system/myapp.service.d/requires-mount.conf
[Unit]
RequiresMountsFor=/data

Prevent it next time

  • Patch runbook step: after reboot, verify lsblk and mounts before declaring success.
  • Monitor: alert when a mount point used by the app is not a separate filesystem (findmnt /data).

Related fixes