Recover deleted EBS snapshots, AMIs and volumes with Recycle Bin retention rules

#Goal

Someone ran a clean-up script, deregistered the wrong AMI or deleted a snapshot that turned out to be the only good backup. Without protection in place, a deleted EBS snapshot is gone for good.

Recycle Bin changes that. If a retention rule matches a resource when it is deleted, the resource goes into the Recycle Bin and stays there for the retention period you set. You can restore it at any time during that period. After that it is deleted permanently.

This post shows how to:

1. Check whether you already have retention rules.
2. Create a Region-level rule and a tag-level rule.
3. Lock a rule so an attacker or a bad script can't remove it.
4. Find and restore a deleted snapshot, AMI or volume.

Recycle Bin is a Regional service. Rules only apply in the Region where you create them, so repeat the steps in every Region you use.

#Step 1: check what you have

for t in EBS_SNAPSHOT EC2_IMAGE EBS_VOLUME; do
  echo "== $t"
  aws rbin list-rules --resource-type "$t" \
    --query 'Rules[].{id:Identifier,days:RetentionPeriod.RetentionPeriodValue,lock:LockState,desc:Description}' \
    --output table
done

If EBS_VOLUME returns a validation error, update the AWS CLI. Volume support was added later than snapshots and AMIs.

To see the details of a rule, including the tags it matches:

aws rbin get-rule --identifier <rule-id>

#Step 2: create retention rules

There are two kinds of rule:

If more than one rule matches a resource, the longest retention period applies.

#A Region-level safety net for snapshots

aws rbin create-rule \
  --resource-type EBS_SNAPSHOT \
  --retention-period RetentionPeriodValue=7,RetentionPeriodUnit=DAYS \
  --description "Region safety net - all EBS snapshots 7 days"

Leave out --resource-tags to make a rule Region-level. To exclude short-lived snapshots, such as CI or scratch copies, add --exclude-resource-tags ResourceTagKey=lifecycle,ResourceTagValue=ephemeral. Note that a rule with exclusion tags can't be locked (see Step 3).

#A Region-level rule for AMIs

aws rbin create-rule \
  --resource-type EC2_IMAGE \
  --retention-period RetentionPeriodValue=14,RetentionPeriodUnit=DAYS \
  --description "Region safety net - deregistered AMIs 14 days"

#A tag-level rule for production volumes

aws rbin create-rule \
  --resource-type EBS_VOLUME \
  --retention-period RetentionPeriodValue=30,RetentionPeriodUnit=DAYS \
  --resource-tags ResourceTagKey=env,ResourceTagValue=prod \
  --description "Prod volumes 30 days"

The tag must be on the resource when it is deleted. Tagging a resource after deletion doesn't help.

Warning

Resources in the Recycle Bin are billed at their normal storage rate until they are permanently deleted. A Region-level rule with a long retention period on an account that creates and deletes many snapshots each day can add real cost. Start with a short safety net (7 days) and use tag-level rules for longer retention. Check current pricing for your Region.

#Step 3: lock the rule

An attacker with admin rights, or a clean-up script, can delete a retention rule before deleting your snapshots. A locked rule can't be deleted or changed until it is unlocked, and an unlock only takes effect after an unlock delay that you choose.

Only Region-level rules without exclusion tags can be locked. You can lock a rule when you create it:

aws rbin create-rule \
  --resource-type EBS_SNAPSHOT \
  --retention-period RetentionPeriodValue=7,RetentionPeriodUnit=DAYS \
  --description "Locked safety net - EBS snapshots" \
  --lock-configuration 'UnlockDelay={UnlockDelayUnit=DAYS,UnlockDelayValue=7}'

Or lock an existing rule:

aws rbin lock-rule \
  --identifier <rule-id> \
  --lock-configuration 'UnlockDelay={UnlockDelayUnit=DAYS,UnlockDelayValue=7}'

aws rbin unlock-rule --identifier <rule-id> starts the unlock delay. The rule stays locked until the delay ends. Alert on that call: a CloudTrail UnlockRule event should page someone, because it often comes before an attempt to delete backups.

Tip

Add a service control policy that denies rbin:DeleteRule, rbin:UpdateRule and rbin:UnlockRule to everyone except a break-glass role. The lock protects against deletion, and the SCP limits who can even start an unlock.

#Step 4: find and restore deleted resources

#Snapshots

aws ec2 list-snapshots-in-recycle-bin \
  --query 'Snapshots[].{id:SnapshotId,vol:VolumeId,desc:Description,deleted:RecycleBinEnterTime,purge:RecycleBinExitTime}' \
  --output table

aws ec2 restore-snapshot-from-recycle-bin --snapshot-id snap-0abc1234def567890

RecycleBinExitTime is when the snapshot will be deleted permanently. Restore anything you need before then.

#AMIs

An AMI has two parts: the image registration and its backing snapshots. Restoring the AMI doesn't bring back snapshots that were also deleted. Restore both, or instances launched from the AMI will fail.

aws ec2 list-images-in-recycle-bin \
  --query 'Images[].{id:ImageId,name:Name,deleted:RecycleBinEnterTime,purge:RecycleBinExitTime}' \
  --output table

aws ec2 restore-image-from-recycle-bin --image-id ami-0abc1234def567890

# find the snapshots the AMI used, by description or by the snapshot IDs in your records
aws ec2 list-snapshots-in-recycle-bin \
  --query "Snapshots[?contains(Description, 'ami-0abc1234def567890')].SnapshotId"

Then check the AMI is usable:

aws ec2 describe-images --image-ids ami-0abc1234def567890 \
  --query 'Images[0].{state:State,devices:BlockDeviceMappings[].Ebs.SnapshotId}'

#Volumes

aws ec2 list-volumes-in-recycle-bin --output table
aws ec2 restore-volume-from-recycle-bin --volume-id vol-0abc1234def567890

A restored volume comes back unattached. Attach it to an instance in the same Availability Zone and mount it as usual.

#Permissions you need

The restore commands are EC2 API actions, not rbin actions. The person doing the restore needs, for example, ec2:ListSnapshotsInRecycleBin and ec2:RestoreSnapshotFromRecycleBin, with matching actions for images and volumes. Managing rules needs rbin:* actions such as rbin:CreateRule and rbin:ListRules. Add them to your operator role before an incident, not during one.

#Common problems

#Prevent it next time

• Create a locked Region-level rule for EBS_SNAPSHOT and EC2_IMAGE in every Region you use, and include it in your account baseline (CloudFormation AWS::Rbin::Rule or StackSets).
• Add tag-level rules with longer retention for production volumes and golden AMIs.
• Alert on CloudTrail UnlockRule, DeleteRule and bursts of DeleteSnapshot.
• Recycle Bin is a safety net, not a backup. Keep AWS Backup with vault lock for copies you must keep, and test a restore from both.