Home / AWS

AWSRunbookAdvanced

ECS tasks stuck in PROVISIONING with an Auto Scaling group capacity provider: managed scaling, termination protection and draining runbook

ECS on EC2 tasks sit in PROVISIONING and the Auto Scaling group never grows, or instances drain and never scale in. Read CapacityProviderReservation, find the blocker and fix managed scaling.

Deepak Bhardwaj6 min read

Applies toAmazon ECS on EC2 with Auto Scaling group capacity providers (managed scaling, managed termination protection, managed instance draining), ECS-optimized AMIs

Symptom

  • aws ecs describe-tasks shows tasks with lastStatus: PROVISIONING for many minutes.
  • Service events say service X was unable to place a task... or nothing at all, and deployments stall.
  • The ASG's desired capacity doesn't change. Or it scales out, the tasks still don't place, and later it can't scale in because instances stay protected or DRAINING.

With a managed-scaling capacity provider, tasks that don't fit on existing instances enter PROVISIONING instead of failing immediately. ECS publishes a metric, and a target-tracking policy on the ASG reacts to it. When any part of that loop breaks, tasks wait indefinitely.

How the loop works

ECS calculates CapacityProviderReservation = M / N × 100, where M is the number of instances needed to run all running plus provisioning tasks and N is the number of instances currently in the ASG. A target-tracking policy (named ECSManagedAutoScalingPolicy-...) drives this value towards your targetCapacity.

Reservation vs target Meaning
Above target Needs more instances: ASG scales out
Equal to target Steady state
Below target Spare instances: ASG scales in (unprotected, empty instances only)
200 with N = 0 Scale from zero: ECS needs at least one instance

Step 1: confirm the state

CLUSTER=prod-ecs
aws ecs list-tasks --cluster $CLUSTER --desired-status RUNNING --query taskArns --output text \
 | xargs -n 100 aws ecs describe-tasks --cluster $CLUSTER --query \
   'tasks[?lastStatus==`PROVISIONING`].[taskArn,capacityProviderName,cpu,memory,createdAt]' --output table --tasks

aws ecs describe-capacity-providers --capacity-providers prod-ec2-cp \
  --query 'capacityProviders[].{status:status,asg:autoScalingGroupProvider.autoScalingGroupArn,scaling:autoScalingGroupProvider.managedScaling,termProtect:autoScalingGroupProvider.managedTerminationProtection,draining:autoScalingGroupProvider.managedDraining,update:updateStatus}'

aws ecs describe-clusters --clusters $CLUSTER --include ATTACHMENTS \
  --query 'clusters[].{cps:capacityProviders,default:defaultCapacityProviderStrategy,attachments:attachments[].status}'

Check that the capacity provider is ACTIVE, associated with the cluster, and actually used by the service's or task's capacity provider strategy. A task launched with --launch-type EC2 doesn't use the capacity provider at all.

Step 2: read the metric and the ASG's activity

aws cloudwatch get-metric-statistics --namespace AWS/ECS/ManagedScaling \
  --metric-name CapacityProviderReservation \
  --dimensions Name=ClusterName,Value=$CLUSTER Name=CapacityProviderName,Value=prod-ec2-cp \
  --start-time "$(date -u -d '-1 hour' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 60 --statistics Maximum --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Maximum]' --output text

ASG=prod-ecs-asg
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG \
  --query 'AutoScalingGroups[].{min:MinSize,max:MaxSize,desired:DesiredCapacity,protect:NewInstancesProtectedFromScaleIn,inst:Instances[].[InstanceId,LifecycleState,ProtectedFromScaleIn]}'

aws autoscaling describe-scaling-activities --auto-scaling-group-name $ASG --max-items 10 \
  --query 'Activities[].[StartTime,StatusCode,StatusMessage]' --output text

aws autoscaling describe-policies --auto-scaling-group-name $ASG \
  --query 'ScalingPolicies[].[PolicyName,PolicyType,TargetTrackingConfiguration.TargetValue]' --output text

Step 3: match the pattern to the cause

What you see Cause Fix
Reservation > target, desired = max ASG at MaxSize Raise MaxSize; review why the fleet needs more
Reservation > target, scaling activity Failed Launch failure: capacity, quota, launch template, AMI, subnet IPs Fix the launch error in StatusMessage; add instance types/AZs
ASG scaled out, instances InService, reservation still high Instances not registering with the cluster Check ECS_CLUSTER in /etc/ecs/ecs.config, the instance role, and the network path to ECS endpoints; list-container-instances
Instances registered, task still won't place Task can't fit on any instance type in the ASG (CPU/memory/GPU/ENI) or placement constraints exclude them Use a bigger instance type or a smaller task; review constraints
awsvpc tasks, lots of free CPU/memory ENI limit per instance reached Enable awsvpcTrunking, use instance types that support trunking
No ECSManagedAutoScalingPolicy on the ASG Policy deleted or managed scaling disabled Re-enable managed scaling with update-capacity-provider
Reservation stuck at 100 with tasks provisioning Instances needed match instances present, so placement is the problem, not capacity Look for daemon tasks, port conflicts (bridge/host mode static ports) or placement constraints

Check registration and remaining resources:

aws ecs list-container-instances --cluster $CLUSTER --status ACTIVE --query containerInstanceArns --output text \
 | xargs aws ecs describe-container-instances --cluster $CLUSTER --query \
   'containerInstances[].[ec2InstanceId,agentConnected,status,remainingResources[?name==`CPU`].integerValue|[0],remainingResources[?name==`MEMORY`].integerValue|[0]]' \
   --output table --container-instances

agentConnected: false means the agent lost contact with ECS. Restart the agent through SSM (sudo systemctl restart ecs) and check its log at /var/log/ecs/ecs-agent.log.

Tip

Task memory has to fit in the instance's registered memory, which is less than the instance type's total memory (the OS and the agent reserve some). A task asking for 16 GiB won't fit on a 16 GiB instance. The MEMORY row in registeredResources is the real ceiling.

Step 4: apply the fix

Raise the ceiling or tune managed scaling:

aws autoscaling update-auto-scaling-group --auto-scaling-group-name $ASG --max-size 40

aws ecs update-capacity-provider --name prod-ec2-cp --auto-scaling-group-provider \
 'managedScaling={status=ENABLED,targetCapacity=90,minimumScalingStepSize=1,maximumScalingStepSize=10,instanceWarmupPeriod=180},managedTerminationProtection=ENABLED,managedDraining=ENABLED'
  • targetCapacity < 100 keeps spare instances warm so new tasks place immediately. 100 packs instances tightly, but every new task waits for an instance launch.
  • instanceWarmupPeriod should be about how long an instance takes to boot and register. If it's too short, ECS scales out again before the first instances are counted.
  • maximumScalingStepSize limits how many instances one scale-out adds. Raise it for bursty deployments.

For ENI-bound awsvpc workloads:

aws ecs put-account-setting-default --name awsvpcTrunking --value enabled
# Applies to instances registered after the change; replace existing instances (instance refresh)

Scale-in problems: protected or draining instances that never go away

Managed termination protection requires NewInstancesProtectedFromScaleIn=true on the ASG. ECS then removes protection only from instances that are running no non-daemon tasks. If an instance never becomes empty (a long-running task, or a service whose placement strategy is spread across instances), it stays protected and the ASG can't scale in.

aws autoscaling update-auto-scaling-group --auto-scaling-group-name $ASG --new-instances-protected-from-scale-in

Managed draining moves an instance to DRAINING before ASG scale-in, Spot interruption or instance refresh, so ECS can reschedule its service tasks. Standalone tasks (run-task) aren't rescheduled. They run until they stop, which can hold an instance in DRAINING for a long time.

aws ecs list-container-instances --cluster $CLUSTER --status DRAINING
# Find what keeps it busy
aws ecs list-tasks --cluster $CLUSTER --container-instance <arn> --desired-status RUNNING

Caution

aws ecs deregister-container-instance --force and turning off scale-in protection on an instance with running tasks kill those tasks without draining. Use them only after you've confirmed the remaining tasks are disposable or have been rescheduled elsewhere.

Prevent it next time

  • Alarm on the number of PROVISIONING tasks (Container Insights PendingTaskCount per service) and on CapacityProviderReservation staying above target for 15+ minutes.
  • Alarm when ASG desired capacity reaches MaxSize.
  • Include instance types in the ASG that can fit the largest task definition, and validate this in CI against registered memory, not nominal memory.
  • Use a mixed-instances ASG across several AZs so a single capacity shortage doesn't stall scale-out.
  • Keep the ECS agent current through an AMI pipeline. Old agents miss newer features and have placement bugs.
  • Don't create the capacity provider, ASG and cluster associations by hand. Define them in infrastructure as code so the protection flag, policy and strategy stay consistent.

Related fixes