AWS Backup jobs FAILED, EXPIRED or ABORTED: find the reason and fix the usual causes

#Symptom

The backup report or a daily check shows jobs in FAILED, EXPIRED, ABORTED or PARTIAL. Some resources have no recovery point from last night, and you need to know why before tonight's run fails again.

This runbook starts with the evidence, maps the status message to a cause, and finishes with a test backup and an alert so the next failure is seen the same day.

#Step 1: list the failed jobs

In one account and Region:

aws backup list-backup-jobs \
  --by-state FAILED \
  --by-created-after "$(date -u -d '-2 days' +%Y-%m-%dT%H:%M:%SZ)" \
  --query "BackupJobs[].{Id:BackupJobId,Type:ResourceType,Resource:ResourceArn,Msg:StatusMessage}" \
  --output table

Repeat with --by-state EXPIRED, ABORTED and PARTIAL. Copy jobs fail separately, so also check them:

aws backup list-copy-jobs --by-state FAILED \
  --query "CopyJobs[].{Id:CopyJobId,Source:SourceRecoveryPointArn,Dest:DestinationBackupVaultArn,Msg:StatusMessage}" \
  --output table

#Across the organisation

From the management account or the Backup delegated administrator, with cross-account monitoring turned on, get counts per account and state:

aws backup list-backup-job-summaries \
  --account-id ANY \
  --state FAILED \
  --aggregation-period SEVEN_DAYS \
  --query "BackupJobSummaries[].{Account:AccountId,Region:Region,Type:ResourceType,Count:Count,Category:MessageCategory}" \
  --output table

Then list the jobs for one account with aws backup list-backup-jobs --by-account-id 111122223333 --by-state FAILED from the same administrator account.

#Read the full job

aws backup describe-backup-job --backup-job-id <job-id> \
  --query "{State:State,Msg:StatusMessage,Role:IamRoleArn,Vault:BackupVaultName,Plan:CreatedBy.BackupPlanId,Rule:CreatedBy.BackupRuleId,Start:StartBy,Created:CreationDate}"

StatusMessage is the main clue. IamRoleArn tells you which role AWS Backup used, which matters for the IAM and KMS causes below.

#Step 2: map the message to a cause

Exact wording varies by resource type and changes over time; match on the key phrase.

#Step 3: fix by cause

#IAM role

Find the role from the selection, not from memory:

aws backup list-backup-selections --backup-plan-id <plan-id> \
  --query "BackupSelectionsList[].{Id:SelectionId,Name:SelectionName,Role:IamRoleArn}"

aws iam list-attached-role-policies --role-name AWSBackupDefaultServiceRole

The role needs AWSBackupServiceRolePolicyForBackup (and AWSBackupServiceRolePolicyForRestores for restores). S3 is separate: attach AWSBackupServiceRolePolicyForS3Backup and AWSBackupServiceRolePolicyForS3Restore. The trust policy must allow backup.amazonaws.com.

aws iam attach-role-policy \
  --role-name AWSBackupDefaultServiceRole \
  --policy-arn arn:aws:iam::aws:policy/AWSBackupServiceRolePolicyForS3Backup

If SCPs or permission boundaries apply in that account, check them too; a managed policy can't override an explicit deny.

#KMS keys

For EBS, RDS and other resources encrypted with a customer managed key, the Backup role must be allowed to use the key. Check the key state and policy:

aws kms describe-key --key-id <key-arn> --query "KeyMetadata.{State:KeyState,Manager:KeyManager}"
aws kms get-key-policy --key-id <key-arn> --policy-name default --output text

Add a statement for the Backup role ARN with kms:Decrypt, kms:DescribeKey, kms:GenerateDataKey*, kms:ReEncrypt* and kms:CreateGrant (with a kms:GrantIsForAWSResource condition). For cross-account copies, the destination account's key and vault access policy must also allow the source account.

Warning

A key in PendingDeletion or Disabled state breaks backups and also makes existing data unreadable. Cancel the deletion with aws kms cancel-key-deletion before doing anything else.

#Resource opt-in and selection

If a resource type has no jobs at all, check the Region settings:

aws backup describe-region-settings
aws backup update-region-settings --resource-type-opt-in-preference DynamoDB=true,EFS=true

Then confirm the selection actually matches the resource:

aws backup get-backup-selection --backup-plan-id <plan-id> --selection-id <selection-id> \
  --query "BackupSelection.{Resources:Resources,Tags:ListOfTags,Conditions:Conditions,NotResources:NotResources}"

aws backup list-protected-resources \
  --query "Results[?ResourceArn=='<resource-arn>']"

Tag keys and values are case-sensitive. backup=Daily does not match Backup=daily.

#EXPIRED jobs and windows

aws backup get-backup-plan --backup-plan-id <plan-id> \
  --query "BackupPlan.Rules[].{Rule:RuleName,Cron:ScheduleExpression,StartMin:StartWindowMinutes,CompleteMin:CompletionWindowMinutes}"

A short start window with many resources in one plan often expires jobs that queue behind others. Widen StartWindowMinutes, increase CompletionWindowMinutes for large volumes, or split the plan. For RDS and Aurora, keep the AWS Backup schedule away from the database's automated backup and maintenance windows.

#Windows VSS failures

Application-consistent Windows backups need SSM working on the instance and the VSS components installed:

aws ssm send-command \
  --document-name AWS-ConfigureAWSPackage \
  --targets "Key=InstanceIds,Values=i-0123456789abcdef0" \
  --parameters "action=Install,name=AwsVssComponents"

The instance profile also needs the VSS snapshot permissions listed in the AWS Backup VSS documentation. On the instance, check the writers:

vssadmin list writers | Select-String -Pattern "Writer name|State|Last error"

Any writer not Stable with No error will fail the job. Restarting the owning service (for example SQL Server VSS Writer) usually clears it.

#S3 versioning

aws s3api get-bucket-versioning --bucket my-bucket
aws s3api put-bucket-versioning --bucket my-bucket --versioning-configuration Status=Enabled

Caution

Enabling versioning keeps every overwritten and deleted object version, which increases storage cost. Add a lifecycle rule for noncurrent versions before turning it on for large, busy buckets.

#Step 4: prove the fix with an on-demand job

Don't wait for tonight's schedule:

aws backup start-backup-job \
  --backup-vault-name Default \
  --resource-arn <resource-arn> \
  --iam-role-arn arn:aws:iam::111122223333:role/service-role/AWSBackupDefaultServiceRole \
  --start-window-minutes 60

aws backup describe-backup-job --backup-job-id <new-job-id> --query "{State:State,Pct:PercentDone,Msg:StatusMessage}"

Use the same role and vault as the plan, or you are testing a different path.

#Prevent it next time

• Alert on state changes. Create an EventBridge rule and send it to SNS or your incident tool:

{
  "source": ["aws.backup"],
  "detail-type": ["Backup Job State Change", "Copy Job State Change"],
  "detail": { "state": ["FAILED", "EXPIRED", "ABORTED", "PARTIAL"] }
}

• Report daily. Use AWS Backup Audit Manager or a scheduled list-backup-job-summaries call from the delegated administrator so someone sees failures from every account.
• Pin roles and keys. Use one Backup role per account, deployed by IaC, and include it in every customer managed key policy template.
• Check coverage, not just failures. A resource that no selection matches never fails; it is simply never backed up. Compare list-protected-resources with your inventory each month.
• Test restores. A completed job is not a working restore. Schedule restore tests for critical resources.