OCI instance pool autoscaling not scaling: metrics, cooldown, limits and pool state

#Symptom

One of these:

• Load is high, CPU on the instances is above the scale-out threshold, and the pool size does not change.
• The pool scaled out once and then stopped reacting.
• Load dropped hours ago and the pool never scaled back in.
• A schedule-based policy did not fire at the expected time.

Work through the checks below in order. Most cases end at step 2 (no metrics) or step 4 (limits or capacity).

Set a few variables first:

C=ocid1.compartment.oc1..xxxx        # compartment of the pool and the autoscaling config
POOL=ocid1.instancepool.oc1..xxxx

#1. Is there an enabled configuration for this pool?

Each instance pool can have only one autoscaling configuration. Find it and confirm it is enabled and points at the right pool:

oci autoscaling configuration list -c $C --all \
  --query "data[].{name:\"display-name\", enabled:\"is-enabled\", pool:resource.id, id:id}" \
  --output table

Then read the full configuration, including the policy and its capacity:

ASC=ocid1.autoscalingconfiguration.oc1..xxxx
oci autoscaling configuration get --auto-scaling-configuration-id $ASC

Check in the output:

• is-enabled is true on the configuration and on the policy.
• cool-down-in-seconds (minimum and default 300).
• The policy capacity: initial, min, max. If the pool is already at max, nothing will happen however hot it runs. If it is already at min, it will not scale in.
• The rule thresholds. A scale-out at > 90 on a workload that saturates at 80% will never fire.

A configuration holds either one metric-based (threshold) policy or up to 50 schedule-based policies, not both.

Re-enable a configuration someone switched off during a change:

oci autoscaling configuration update --auto-scaling-configuration-id $ASC --is-enabled true

#2. Is the autoscaler getting metrics?

Metric-based autoscaling reads CpuUtilization or MemoryUtilization from the oci_computeagent namespace, which the Compute Instance Monitoring plugin of Oracle Cloud Agent emits. If the plugin is stopped or the instance can't reach the Monitoring service, autoscaling has nothing to act on.

Query what the autoscaler sees, the pool-wide average per minute:

oci monitoring metric-data summarize-metrics-data -c $C \
  --namespace oci_computeagent \
  --query-text "CpuUtilization[1m]{instancePoolId = \"$POOL\"}.mean()" \
  --start-time $(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ)

An empty aggregated-datapoints list means no data. Break it down per instance to find the ones not reporting:

oci monitoring metric-data summarize-metrics-data -c $C \
  --namespace oci_computeagent \
  --query-text "CpuUtilization[1m]{instancePoolId = \"$POOL\"}.groupBy(resourceId).mean()" \
  --start-time $(date -u -d '-30 min' +%Y-%m-%dT%H:%M:%SZ) \
  --query "data[].{instance:dimensions.resourceId, points:length(\"aggregated-datapoints\")}" \
  --output table

For any instance missing from the list, check the plugin:

oci instance-agent plugin list -c $C --instanceagent-id <instance-ocid> \
  --query "data[].{name:name, status:status}" --output table

Tip

A hand-built golden image that never had Oracle Cloud Agent installed gives a pool that launches fine and never autoscales. Check the agent on the image before you build an instance configuration from it.

#3. Has the threshold been met long enough, and is cooldown blocking?

The service aggregates metrics into one-minute periods, averages them across the pool and triggers an event when three consecutive one-minute values meet the threshold. A spiky graph that crosses the line for one minute at a time never triggers.

After a scaling event, the cooldown starts when the pool goes from SCALING back to RUNNING, not when the event fired. If new instances take eight minutes to provision and boot, the next decision can be 13 minutes or more after the first.

If the workload ramps faster than that, scale by a larger step (add 3 instances, not 1) rather than chasing a shorter cooldown. 300 seconds is already the minimum.

Cooldown does not apply to schedule-based policies.

#4. What state is the pool in, and did the last scale fail?

oci compute-management instance-pool get --instance-pool-id $POOL \
  --query "data.{state:\"lifecycle-state\", size:size}"

oci compute-management instance-pool list-instances -c $C --instance-pool-id $POOL \
  --query "data[].{id:id, state:state, ad:\"availability-domain\", fd:\"fault-domain\"}" \
  --output table

A pool stuck in SCALING, or one that is STOPPED, will not take new metric-based actions. Look at the recent work requests on the pool and read the errors from any that failed:

oci work-requests work-request list -c $C --resource-id $POOL \
  --query "data[].{op:\"operation-type\", status:status, started:\"time-started\", id:id}" \
  --output table

oci work-requests work-request-error list --work-request-id <work-request-ocid>

Check headroom for the shape before raising max. Limit names vary by shape family; find the one for your shape with oci limits definition list:

oci limits resource-availability get --compartment-id <tenancy-ocid> --service-name compute \
  --limit-name standard-e5-core-count --availability-domain "Uocm:AP-MUMBAI-1-AD-1"

The OCPU count needed is max multiplied by the OCPUs per instance in the instance configuration.

Warning

Scale-in terminates instances. It balances across ADs, then fault domains, then removes the oldest instance in a fault domain. Anything written to the boot volume of a pool instance is lost. Keep pool instances stateless before you tighten scale-in thresholds.

#5. Schedule-based policies that did not fire

• Schedules are Quartz cron with seconds first: <second> <minute> <hour> <day of month> <month> <day of week> <year>. Through the CLI, API or SDKs the seconds field must be 0.
• All times are UTC. A policy written for 08:00 local time in India fires at 13:30 local if you forget.
• Scale-out and scale-in need separate policies. A manual resize stays in place until the next scheduled policy runs.
• If two policies fire at the same time, lifecycle-state policies run first, and for size the policy with the highest instance count wins.

Read the schedules in the configuration:

oci autoscaling configuration get --auto-scaling-configuration-id $ASC \
  --query "data.policies[].{name:\"display-name\", type:\"policy-type\", enabled:\"is-enabled\", schedule:\"execution-schedule\".expression, capacity:capacity}"

#Buy time while you fix it

If the pool is overloaded now, resize it by hand. The autoscaler adjusts from the new size on its next decision:

oci compute-management instance-pool update --instance-pool-id $POOL --size 8

Raise the policy limits if max was the problem. Policy updates need the policy type:

oci autoscaling policy update --auto-scaling-configuration-id $ASC \
  --auto-scaling-policy-id <policy-id> --policy-type threshold \
  --capacity '{"initial": 3, "min": 3, "max": 12}'

Generate the full JSON for other fields with --generate-param-json-input capacity rather than guessing key names.

#Prevent it next time

• Alarm on the same metric the autoscaler uses, plus an absence alarm: CpuUtilization[1m]{instancePoolId = "<pool>"}.absent() catches the "no metrics" case before a traffic spike does.
• Alarm when the pool sits at max for a long period. That is the point where autoscaling has stopped helping.
• Keep tenancy limits for the shape above max × OCPUs per instance, in every AD the pool uses.
• Bake Oracle Cloud Agent with the monitoring plugin enabled into the image, and allow the service gateway route in the pool subnet's route table.
• Test autoscaling after every new instance configuration. Run a CPU load generator on the pool in staging and watch it scale out and back in.
• IAM for the operators who manage it: Allow group AutoscalingAdmins to manage auto-scaling-configurations in compartment <name> and Allow group AutoscalingAdmins to manage instance-pools in compartment <name>.