ECS deployment stuck IN_PROGRESS: find the failing tasks and turn on circuit breaker rollback

#Symptom

You've deployed a new task definition, and the service sits in IN_PROGRESS. Tasks start, run for a while and stop. Your pipeline eventually times out, or the deployment hangs for hours while old tasks keep serving traffic.

Without a circuit breaker, ECS keeps trying to reach the desired state indefinitely, with throttled retries. There may be no failure to alert on.

#Step 1: see the deployments

CLUSTER=prod
SERVICE=web

aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
  --query 'services[0].deployments[].{status:status, taskDef:taskDefinition, desired:desiredCount, running:runningCount, pending:pendingCount, failed:failedTasks, rollout:rolloutState, reason:rolloutStateReason}' \
  --output table

See what the scheduler has been doing:

aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
  --query 'services[0].events[:15].[createdAt,message]' --output text

#Step 2: find out why tasks stopped

Stopped tasks stay visible only for a short time, usually about an hour, so look soon:

TASKS=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name "$SERVICE" \
  --desired-status STOPPED --query 'taskArns[:10]' --output text)

aws ecs describe-tasks --cluster "$CLUSTER" --tasks $TASKS \
  --query 'tasks[].{def:taskDefinitionArn, stopCode:stopCode, stopped:stoppedReason, containers:containers[].{name:name, exit:exitCode, reason:reason}}'

Map the result to a cause:

Read the logs for a failed task:

aws logs tail /ecs/web --since 30m --filter-pattern "ERROR"

#Step 3: fix target group failures

If tasks are killed by the load balancer, the container is probably fine. The health check doesn't agree.

TG_ARN=$(aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
  --query 'services[0].loadBalancers[0].targetGroupArn' --output text)

aws elbv2 describe-target-groups --target-group-arns "$TG_ARN" \
  --query 'TargetGroups[0].{path:HealthCheckPath, port:HealthCheckPort, interval:HealthCheckIntervalSeconds, timeout:HealthCheckTimeoutSeconds, unhealthy:UnhealthyThresholdCount, matcher:Matcher.HttpCode}'

aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
  --query 'TargetHealthDescriptions[].{ip:Target.Id, state:TargetHealth.State, reason:TargetHealth.Reason, desc:TargetHealth.Description}'

If the app just starts slowly, give it time before health checks can stop it:

aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
  --health-check-grace-period-seconds 120

Pick a figure a little above your real startup time. Too long, and truly broken tasks stay in service longer than they should.

#Step 4: get out of the current deployment

If production is degraded, go back to the last good revision. First, find it:

aws ecs list-task-definitions --family-prefix web --sort DESC --max-items 5

Then deploy it:

aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
  --task-definition web:41

This begins a new deployment, using the old revision. Use a specific revision number, not just the family name, so you know exactly what you're getting.

#Step 5: turn on the circuit breaker

With the circuit breaker, a deployment that can't reach a steady state fails on its own, and rolls back to the last completed one:

aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
  --deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true},maximumPercent=200,minimumHealthyPercent=100"

How it works, in short:

• It watches for tasks that fail to reach RUNNING, or that fail health checks.
• The failure threshold scales with desired count, using 0.5 times desired count, with a floor of 3 and a cap of 200 in current documentation.
• With low desired counts, it can take several failures before it trips.

This feature applies to the rolling update (ECS) controller. It doesn't apply to CodeDeploy blue/green.

Test it on non-production with a deliberately broken image. Confirm that rolloutState becomes FAILED, and that the service returns to the old revision.

Warning

A circuit breaker only catches tasks that fail to start or stay healthy. A release that passes health checks but returns errors to users won't trip it. Add alarm-based rollback for that.

#Step 6: add alarm-based rollback

Roll back when your own metrics go bad, such as ALB 5xx errors:

aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
  --deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true},alarms={alarmNames=[web-5xx-high],enable=true,rollback=true},maximumPercent=200,minimumHealthyPercent=100"

Choose an alarm that reacts faster than your deployment completes. Otherwise, a bad release could finish before the alarm fires.

#Step 7: alert on failed deployments

ECS sends deployment events to EventBridge. Alert on failures, using this event pattern:

{
  "source": ["aws.ecs"],
  "detail-type": ["ECS Deployment State Change"],
  "detail": {
    "eventName": ["SERVICE_DEPLOYMENT_FAILED"]
  }
}

Your pipeline can also wait for an answer, instead of guessing:

aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE"

aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
  --query 'services[0].deployments[?status==`PRIMARY`].rolloutState' --output text

The waiter has its own timeout, and a rollback can also end in a stable state. So check that the running task definition is the one you deployed.

#Prevent it next time

• Turn on the circuit breaker for every service, in your infrastructure code.
• Make health endpoints quick, and check only what the task needs to serve traffic.
• Deploy immutable image tags or digests, not latest.
• Store the logs of failed tasks, since stopped task details disappear quickly.
• Related: tasks keep stopping and ALB 502, 503 and 504 errors.