OKE PVC stuck Pending or pod won't attach: Block Volume CSI and availability domain conflicts

#Symptom

A pod that needs storage won't start:

$ kubectl get pvc,pod -n app
NAME                              STATUS    VOLUME   CAPACITY   STORAGECLASS
persistentvolumeclaim/data-db-0   Pending                       oci-bv

NAME       READY   STATUS    RESTARTS   AGE
pod/db-0   0/1     Pending   0          19m

Storage has several stages, and each fails in its own way. First, work out which one you're at.

#Step 1: find the failing stage

kubectl describe pvc data-db-0 -n app
kubectl describe pod db-0 -n app
kubectl get events -n app --sort-by=.lastTimestamp | tail -20

#Step 2: check the storage class

kubectl get storageclass
kubectl get storageclass oci-bv -o yaml

Class names and which one's the default depend on the cluster version, so check your output. oci-bv normally uses volumeBindingMode: WaitForFirstConsumer. The volume isn't created until a pod that uses it is scheduled, so it can be placed in that node's availability domain (AD).

So a Pending PVC with waiting for first consumer isn't the fault. The question is why the pod isn't scheduling. Look at the pod's events for reasons such as insufficient CPU, or see nodes not ready.

A working PVC looks like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-db-0
  namespace: app
spec:
  storageClassName: oci-bv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

Block volumes are ReadWriteOnce, meaning one node at a time. Asking for ReadWriteMany won't work. For shared storage, use File Storage with its own CSI driver. The minimum block volume size is 50 GB, so a smaller request gets you a larger volume than you asked for. Check your output with kubectl get pv.

#Step 3: fix availability domain conflicts

A block volume lives in one AD, and it can only attach to a node in that AD. This is the most common failure after a node replacement or a pool change.

Compare the two sides:

kubectl get pvc data-db-0 -n app -o jsonpath='{.spec.volumeName}{"\n"}'

kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity}{"\n"}'

kubectl get nodes -L topology.kubernetes.io/zone -L failure-domain.beta.kubernetes.io/zone

Label names vary by cluster version, so use whichever one the PV's nodeAffinity mentions. Pod events will read:

1 node(s) had volume node affinity conflict

How it usually happens:

Fixes, starting with the lowest risk:

1. Add capacity in the volume's AD. Scale up a node pool placed there. Nothing about your data changes.

oci ce node-pool get --node-pool-id "$POOL" \
  --query 'data."node-config-details"."placement-configs"[]."availability-domain"'

2. Remove a conflicting nodeSelector or affinity rule from the workload.
3. Move the data to another AD. Create a backup of the volume, then restore it in the target AD. See Block Volume backup and restore. You then have to create a new PV and PVC for the restored volume.

Caution

Don't delete the PVC to "reset" it. With the Delete reclaim policy, which is the default for dynamic volumes, Kubernetes also deletes the block volume and your data. Before any storage changes, check kubectl get pv -o custom-columns=NAME:.metadata.name,POLICY:.spec.persistentVolumeReclaimPolicy. If you need to, set it to Retain first.

kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

#Step 4: fix provisioning failures

ProvisioningFailed means that the volume couldn't be created. The event usually says why.

Check what was created, and whether the volume is healthy:

oci bv volume list --compartment-id "$C" --lifecycle-state AVAILABLE \
  --query 'data[].{name:"display-name", ad:"availability-domain", gb:"size-in-gbs", vpus:"vpus-per-gb"}' --output table

#Step 5: fix attach failures

Multi-Attach error for volume "csi-..." Volume is already exclusively attached to one node

This often follows a node failure, when the old attachment hasn't been released. Find where the volume is:

kubectl get volumeattachment | grep <pv-name>
oci compute volume-attachment list --compartment-id "$C" --volume-id "$VOL_ID" \
  --query 'data[].{instance:"instance-id", state:"lifecycle-state"}' --output table

If the old node really is gone, Kubernetes usually sorts it out after several minutes. A stuck case can follow a node that's NotReady but still registered. Confirm the node's state, drain it or delete the node object:

kubectl get node <node>
kubectl delete node <node>

Warning

Make sure the old node is really down before you force anything. Attaching one volume to two running nodes, or running two writers, can corrupt the file system.

Other causes are hitting the shape's volume attachment limit, and the node's own problems. Check the limits for your shape.

#Step 6: fix mount failures

The volume is attached, but the mount fails. Look at the CSI node plugin and the node:

kubectl get pods -n kube-system -o wide | grep -i csi
kubectl logs -n kube-system <csi-node-pod-on-that-node> --all-containers --tail=100

#Grow a volume

If the class allows it, you can expand a volume in place:

kubectl get storageclass oci-bv -o jsonpath='{.allowVolumeExpansion}{"\n"}'

kubectl patch pvc data-db-0 -n app \
  -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

kubectl get pvc data-db-0 -n app -o yaml | grep -A5 conditions

Sizes can only go up. Back up first, and check the conditions for a pending file system resize.

#Prevent it next time

• Spread node pools across ADs, and keep capacity in each AD that holds a volume.
• Use topologySpreadConstraints for stateful replicas, so one AD failure doesn't take all of them.
• Use the Retain reclaim policy for data you can't recreate.
• Schedule volume backups, and test a restore. Snapshots don't replace application backups.
• Before you change node pools, list each PV's AD. See the HA guide and the OKE baseline.