Sev1: ECS service tasks keep stopping — CannotPullContainerError, failing health checks and essential container exits

#Symptoms

• runningCount stays below desiredCount; service events show tasks starting and stopping.
• ALB targets flap; users see 502/503.
• Often right after a deployment.

#1. Read why tasks stopped

aws ecs describe-services --cluster prod --services orders \
  --query "services[0].{desired:desiredCount, running:runningCount, events:events[:5].message, deployments:deployments[].{id:id,status:status,rollout:rolloutState,td:taskDefinition}}"

TASKS=$(aws ecs list-tasks --cluster prod --service-name orders --desired-status STOPPED --query "taskArns[:5]" --output text)
aws ecs describe-tasks --cluster prod --tasks $TASKS \
  --query "tasks[].{reason:stoppedReason, code:stopCode, containers:containers[].{n:name, exit:exitCode, r:reason}}"

#2. Image pull failures

aws ecr describe-images --repository-name orders --image-ids imageTag=1.42.0

• Tag doesn't exist (lifecycle policy deleted it, or CI never pushed) → push it or roll back.
• Execution role lacks ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer → attach AmazonECSTaskExecutionRolePolicy.
• Private subnets without NAT need VPC endpoints: ecr.api, ecr.dkr, S3 gateway endpoint (image layers), and logs.
• Docker Hub images hit rate limits; mirror them to ECR.

#3. Secrets and initialization

Tasks fetch secrets with the execution role, not the task role.

aws ecs describe-task-definition --task-definition orders \
  --query "taskDefinition.{exec:executionRoleArn, secrets:containerDefinitions[].secrets}"

Check the execution role can secretsmanager:GetSecretValue / ssm:GetParameters on those ARNs and kms:Decrypt on a customer-managed key. A deleted or renamed secret fails every task.

#4. Health check failures

aws elbv2 describe-target-health --target-group-arn <tg-arn> \
  --query "TargetHealthDescriptions[].{ip:Target.Id, state:TargetHealth.State, reason:TargetHealth.Reason}"

• App takes longer to start than the grace period → raise it:

aws ecs update-service --cluster prod --service orders --health-check-grace-period-seconds 120

• Health path changed or now returns 301/401 → fix the path or the matcher.
• Security group of the task doesn't allow the ALB on the container port.

#5. App crashes and OOM

aws logs tail /ecs/orders --since 30m | tail -50

Exit code 137 usually means killed (OOM or stop timeout); 1 is an app error (often bad config or missing env var). Raise memory in the task definition if OOM.

#6. Roll back now, debug after

If the circuit breaker is enabled with rollback, ECS rolls back automatically. Otherwise point the service at the last good revision:

aws ecs update-service --cluster prod --service orders --task-definition orders:41 --force-new-deployment
aws ecs wait services-stable --cluster prod --services orders

Tip

Enable the deployment circuit breaker so a bad release can't take the service to zero: aws ecs update-service --cluster prod --service orders --deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true},minimumHealthyPercent=100,maximumPercent=200"

#Prevent it next time

• Circuit breaker with rollback on every production service.
• minimumHealthyPercent=100 so old tasks stay until new ones are healthy.
• Immutable image tags and ECR lifecycle rules that never delete deployed tags.
• Container Insights alarms on running task count below desired.