Home / OCI

OCIRunbookAdvanced

OCI instance pool rolling replacement: roll out a new image or instance configuration without downtime

Updating an instance pool's configuration changes nothing for running instances. Use surge-and-detach to replace them in batches behind a load balancer, with health gates, drain and a rollback path.

Deepak Bhardwaj7 min read

Applies toOCI Compute instance pools and instance configurations, OCI Flexible Load Balancer, OCI CLI 3.x

Goal

You have a new custom image (OS patching, a new app build) or a changed shape, and the instance pool behind a load balancer has to run it, with no drop in serving capacity.

What catches people out:

  • Launch details in an instance configuration can't be edited. You create a new configuration and point the pool at it.
  • Pointing the pool at a new configuration only affects instances launched afterwards. Existing instances keep running the old image until you replace them.
  • When you shrink a pool by lowering size, you don't choose which instances it terminates. To remove specific old instances, detach them.

This runbook uses surge-and-detach: grow the pool by a batch, wait until the new instances are healthy in the backend set, then detach and terminate the same number of old ones.

Before you start

Check Why
Service limits and capacity for the surge A surge of N instances needs N extra cores of that shape in each placement AD
Autoscaling disabled for the duration An autoscaling policy can resize the pool mid-rollout
Backend set health check matches app readiness A health check that passes before the app is warm sends traffic to cold nodes
Boot volume and app data are disposable Detach with auto-terminate deletes the instance and its boot volume
Old configuration OCID recorded It's your rollback 
export C=ocid1.compartment.oc1..aaaa
export POOL=ocid1.instancepool.oc1..aaaa
export LB=ocid1.loadbalancer.oc1..aaaa
export BS=bs-app

oci compute-management instance-pool get --instance-pool-id $POOL \
  --query 'data.{size:size,config:"instance-configuration-id",state:"lifecycle-state",lbs:"load-balancers"}'

Caution

instance-pool-instance detach --is-auto-terminate true permanently deletes the instance and its boot volume. If anything on the instance isn't reproducible from the image and user data (logs not shipped, local state), copy it off first.

Step 1: Create the new instance configuration

The simplest way is to build one reference instance from the new image, validate it, and capture its configuration:

NEWCFG=$(oci compute-management instance-configuration create-from-instance \
  -c $C --instance-id $REF_INSTANCE --display-name "app-cfg-2026-11-09" \
  --query 'data.id' --raw-output)

Alternatively, export the current one, edit imageId or shapeConfig and create a new one:

OLDCFG=$(oci compute-management instance-pool get --instance-pool-id $POOL \
  --query 'data."instance-configuration-id"' --raw-output)

oci compute-management instance-configuration get --instance-configuration-id $OLDCFG \
  --query 'data."instance-details"' > details.json
# edit details.json: launchDetails.sourceDetails.imageId, shapeConfig, metadata ...
oci compute-management instance-configuration create -c $C \
  --instance-details file://details.json --display-name "app-cfg-2026-11-09"

Check that the new configuration doesn't still reference the reference instance's private IP, hostname label or fixed VNIC settings. Pools need configurations without per-instance identity.

Step 2: Pause autoscaling

ASC=$(oci autoscaling configuration list -c $C \
  --query "data[?resource.id=='$POOL'].id | [0]" --raw-output)
oci autoscaling configuration update --autoscaling-configuration-id $ASC --is-enabled false

Step 3: Point the pool at the new configuration

oci compute-management instance-pool update --instance-pool-id $POOL \
  --instance-configuration-id $NEWCFG --wait-for-state RUNNING

Nothing restarts. From now on, every launch (surge, replacement, autoscale) uses the new image.

Step 4: Record the old instances

oci compute-management instance-pool list-instances -c $C --instance-pool-id $POOL --all \
  --query 'data[].{id:id,ad:"availability-domain",fd:"fault-domain",created:"time-created",state:state}' \
  --output table

oci compute-management instance-pool list-instances -c $C --instance-pool-id $POOL --all \
  --query 'data[].id' | jq -r '.[]' > old-instances.txt

Step 5: Replace in batches

Batch size is a trade-off: larger batches finish faster but need more surge capacity and expose more traffic to the new build at once. Start with one instance as a canary, then batch at around 25% of the pool.

5a. Surge

SIZE=$(oci compute-management instance-pool get --instance-pool-id $POOL --query 'data.size' --raw-output)
BATCH=2
oci compute-management instance-pool update --instance-pool-id $POOL \
  --size $((SIZE + BATCH)) --wait-for-state RUNNING

The pool reports RUNNING when the instances are launched, which is not the same as healthy. New instances are added to the attached backend sets automatically.

5b. Gate on load balancer health

oci lb backend-set-health get --load-balancer-id $LB --backend-set-name $BS
# Wait until "status" is OK and total-backend-count = SIZE + BATCH

Per-backend detail (backend name is ip:port):

oci lb backend-health get --load-balancer-id $LB --backend-set-name $BS --backend-name 10.0.2.41:8080

Also check your application signals: error rate and latency for the new instances, not just the LB health check.

5c. Drain the old instances being removed

Draining stops new connections while existing ones complete:

IP=$(oci compute instance list-vnics --instance-id $OLD --query 'data[0]."private-ip"' --raw-output)
oci lb backend update --load-balancer-id $LB --backend-set-name $BS \
  --backend-name "$IP:8080" --weight 1 --backup false --offline false --drain true \
  --wait-for-state SUCCEEDED

Wait for longer than your longest normal request or session.

5d. Detach and terminate the old instances

oci compute-management instance-pool-instance detach \
  --instance-pool-id $POOL --instance-id $OLD \
  --is-decrement-size true --is-auto-terminate true \
  --wait-for-state SUCCEEDED

With --is-decrement-size true, the pool size drops back by one, so the surge is undone one instance at a time. The pool removes the instance from its attached backend sets.

Repeat 5a to 5d until no IDs from old-instances.txt remain in list-instances.

Detach options Result Use for
decrement true, terminate true Old instance deleted, pool shrinks Surge-and-detach (this runbook)
decrement false, terminate true Old instance deleted, pool launches a replacement No surge capacity; accept a temporary capacity dip
decrement true, terminate false Instance kept outside the pool Forensics or debugging a bad node

Step 6: Re-enable autoscaling and clean up

oci autoscaling configuration update --autoscaling-configuration-id $ASC --is-enabled true

Keep the old configuration until the new build has run cleanly through a full business cycle, then delete it. Terminate the reference instance.

Rollback

During the rollout:

  1. Point the pool back at $OLDCFG with instance-pool update --instance-configuration-id.
  2. Detach the new instances (--is-decrement-size false --is-auto-terminate true) so the pool replaces them with old-image instances, or surge and detach as above.

Because old instances are only deleted after new ones pass health checks, a failed canary costs you one instance, not the service.

Failure modes

Symptom Cause Fix
Surge stuck, pool SCALING Out of host capacity or limit in one AD Check work requests; use a capacity reservation or another shape; spread placement across ADs
New backends stay CRITICAL Health check path, NSG for LB subnet, app not started Check backend-health, cloud-init log, NSG rules for the health check port
Pool size changes on its own Autoscaling still enabled Step 2
Old instance comes back Detached with decrement false and the old config still on the pool Confirm Step 3 before detaching
Uneven AD or fault domain spread Surge placed new instances where capacity existed Check list-instances; detach and replace to rebalance

Prevent it next time

  • Bake images in a pipeline and create instance configurations from code (Terraform oci_core_instance_configuration), so every rollout is "new config OCID + this runbook".
  • Script the batch loop with the health gate and keep the batch size as a parameter.
  • Hold a capacity reservation sized to one batch for pools on scarce shapes.
  • Alert on pool size not equal to the target when no rollout is running.

Related fixes