Home / AWS

AWSRunbookIntermediate

CloudFormation stack stuck in UPDATE_ROLLBACK_FAILED or DELETE_FAILED: a recovery runbook

Get a CloudFormation stack out of UPDATE_ROLLBACK_FAILED, DELETE_FAILED or a long IN_PROGRESS state without deleting production resources.

Deepak Bhardwaj6 min read

Applies toAWS CloudFormation (including nested stacks, AWS CDK and SAM deployments), AWS CLI v2

Symptoms

  • A deployment pipeline fails with Stack is in UPDATE_ROLLBACK_FAILED state and can not be updated.
  • cdk deploy or sam deploy hangs on one resource for an hour, then the stack rolls back and fails.
  • delete-stack ends in DELETE_FAILED and the stack can't be removed.

All three come from the same thing: CloudFormation could not create, update or delete a resource, and it then failed to undo that change. Until you fix the stack, nobody can deploy to it.

Caution

Don't delete a production stack to "start clean". Deleting the stack also deletes every resource it manages that has no DeletionPolicy: Retain, including databases, buckets and queues. Fix the rollback instead.

1. Find the resource that failed

The first *_FAILED event in time order is usually the real cause. Events after it are side effects.

STACK=prod-app

aws cloudformation describe-stacks --stack-name $STACK \
  --query "Stacks[0].{status:StackStatus,reason:StackStatusReason}" --output table

aws cloudformation describe-stack-events --stack-name $STACK \
  --query "StackEvents[?contains(ResourceStatus,'FAILED')].[Timestamp,LogicalResourceId,ResourceType,ResourceStatus,ResourceStatusReason]" \
  --output table

For nested stacks, the parent only says Embedded stack ... was not successfully updated. Find the child stack and read its events too:

aws cloudformation list-stack-resources --stack-name $STACK \
  --query "StackResourceSummaries[?ResourceType=='AWS::CloudFormation::Stack'].[LogicalResourceId,PhysicalResourceId,ResourceStatus]" \
  --output table

2. Match the reason to a fix

Status reason (abridged) Cause Fix
... is not authorized to perform ... The deploy role or the stack's service role lost a permission Restore the permission, then continue the rollback
Resource ... not found / does not exist Someone deleted the resource outside CloudFormation Skip it in continue-update-rollback
The bucket you tried to delete is not empty S3 buckets must be empty before deletion Empty the bucket, or retain it
... has a dependent object / DependencyViolation A security group, ENI or subnet is still used by something outside the stack Remove the dependency (often a leftover Lambda ENI or an attached instance)
Failed to receive N resource signal(s) cfn-signal never arrived from EC2 or the ASG Fix the bootstrap script or network path, then retry
CloudFormation did not receive a response from your Custom Resource Custom resource Lambda crashed or never called the response URL Fix the function; see step 5
Rate exceeded / Throttling API throttling during a large update Retry; split large stacks

3. Recover from UPDATE_ROLLBACK_FAILED

First fix the cause where you can (restore the IAM permission, recreate the dependency, raise the quota). Then retry the rollback:

aws cloudformation continue-update-rollback --stack-name $STACK
aws cloudformation wait stack-rollback-complete --stack-name $STACK

If you can't fix a resource, for example because it was deleted by hand, tell CloudFormation to skip it. CloudFormation then marks it as rolled back without touching it:

aws cloudformation continue-update-rollback --stack-name $STACK \
  --resources-to-skip AppSecurityGroup MyNestedStack.WorkerFunction

Use NestedStackLogicalId.ResourceLogicalId for resources inside nested stacks, and run the command against the root stack.

Warning

A skipped resource's real state and the template no longer match. Run drift detection afterwards (step 6) and fix the difference on the next deploy, or the next update may fail on the same resource.

You can't skip a resource that other resources still depend on unless you skip the dependants too. Only skip resources whose status is UPDATE_FAILED.

4. Recover from DELETE_FAILED

Find the resources that blocked the delete, then pick one approach:

aws cloudformation list-stack-resources --stack-name $STACK \
  --query "StackResourceSummaries[?ResourceStatus=='DELETE_FAILED'].[LogicalResourceId,PhysicalResourceId,ResourceType]" \
  --output table

Fix and retry. Empty the bucket, detach the ENI, then delete again:

aws s3 rm s3://my-app-logs-bucket --recursive
aws cloudformation delete-stack --stack-name $STACK
aws cloudformation wait stack-delete-complete --stack-name $STACK

On a versioned bucket, aws s3 rm leaves old versions and delete markers behind. Use a lifecycle rule or the console's Empty action instead.

Retain specific resources. The stack is deleted and the named resources stay in your account, no longer managed by CloudFormation:

aws cloudformation delete-stack --stack-name $STACK \
  --retain-resources AppLogsBucket LegacyQueue

Force delete. This skips every resource that failed to delete. They show as DELETE_SKIPPED and stay in your account:

aws cloudformation delete-stack --stack-name $STACK --deletion-mode FORCE_DELETE_STACK

--retain-resources and FORCE_DELETE_STACK only work when the stack is already in DELETE_FAILED. Retained or skipped resources keep costing money, so record their IDs and clean them up or import them into another stack. If the stack has termination protection turned on, you have to turn it off before any delete will work.

5. Stack stuck IN_PROGRESS

UPDATE_IN_PROGRESS on one resource for a long time usually means CloudFormation is waiting on something:

  • Custom resource. If the Lambda function never sends a response to the pre-signed URL, CloudFormation waits until the timeout. That's one hour by default; you can lower it with the ServiceTimeout property on newer templates. Check the function's logs:

    aws logs tail /aws/lambda/my-custom-resource-fn --since 2h

    The fastest way out is to send a SUCCESS or FAILED response to the ResponseURL from the logged event, using the documented custom resource response format.

  • ECS service. The service never stabilises because tasks keep failing health checks. Fix the task, or turn on the deployment circuit breaker so the deployment fails fast.

  • Creation policy or wait condition. Instances never sent cfn-signal.

You can cancel an update that is still running. The stack then rolls back:

aws cloudformation cancel-update-stack --stack-name $STACK

6. Check drift after recovery

DRIFT_ID=$(aws cloudformation detect-stack-drift --stack-name $STACK --query StackDriftDetectionId --output text)
aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id $DRIFT_ID

aws cloudformation describe-stack-resource-drifts --stack-name $STACK \
  --stack-resource-drift-status-filters MODIFIED DELETED \
  --query "StackResourceDrifts[].[LogicalResourceId,StackResourceDriftStatus]" --output table

Fix each drifted resource in the template, or put it back to the template's values, before the next deploy.

Prevent it next time

  • Put DeletionPolicy: Retain and UpdateReplacePolicy: Retain on stateful resources (databases, buckets, KMS keys, file systems), and turn on termination protection for production stacks.
  • Use change sets (create-change-set, then review, then execute-change-set) so you see replacements before they happen.
  • During development, deploy with --disable-rollback. A failed stack stays in UPDATE_FAILED so you can fix the resource and retry, or call aws cloudformation rollback-stack.
  • Never change stack-managed resources by hand. Give engineers read-only console access in production, let changes go through the pipeline's role, and run drift detection on a schedule (the AWS Config managed rule cloudformation-stack-drift-detection-check can do this).
  • Make custom resource functions always send a response, including in except/finally blocks, and set a short ServiceTimeout.

Related fixes