Restoring archived S3 objects from Glacier Flexible Retrieval and Deep Archive in bulk

#The goal

An application, auditor or data team needs files that a lifecycle rule moved to S3 Glacier Flexible Retrieval or Glacier Deep Archive months ago. A plain GET fails with:

An error occurred (InvalidObjectState) when calling the GetObject operation:
The operation is not valid for the object's storage class

Archived objects must be restored first. A restore creates a temporary readable copy for a number of days; the archived original stays where it is. For one object you use restore-object. For thousands, use S3 Batch Operations so you get retries and a completion report.

Tip

Objects in S3 Glacier Instant Retrieval don't need a restore. They read like any other object.

#1. Agree the tier and the deadline first

Restore speed and cost depend on the tier you request. Typical times (check the current S3 documentation for your case):

Bulk is the cheapest per GB and is usually right for large restores with a deadline of a day or two. Check current prices on the S3 pricing page; retrieval cost is per GB plus a per-request charge, so millions of small objects cost more than their size suggests.

You also pay for the temporary restored copy for every day it exists, on top of the archive storage. Request only as many days as the consumer needs.

#2. Size the job

Count the archived objects and bytes under the prefix:

aws s3api list-objects-v2 --bucket finance-archive --prefix invoices/2024/ \
  --query "Contents[?StorageClass=='GLACIER' || StorageClass=='DEEP_ARCHIVE'].[Key,Size]" \
  --output text | awk '{n++; s+=$2} END {printf "%d objects, %.1f GiB\n", n, s/1024/1024/1024}'

The storage class values are GLACIER (Flexible Retrieval), DEEP_ARCHIVE and, for Intelligent-Tiering, INTELLIGENT_TIERING. For Intelligent-Tiering, head-object shows ArchiveStatus to tell you whether a given object is in an archive tier.

For buckets with millions of objects, use an S3 Inventory report instead of listing; it includes storage class and can serve directly as the Batch Operations manifest.

#3. Restore one object (smoke test)

Test with one object before launching thousands:

aws s3api restore-object --bucket finance-archive --key invoices/2024/01/inv-0001.pdf \
  --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Bulk"}}'

For a versioned bucket, add --version-id to restore a specific version. For objects in Intelligent-Tiering archive tiers, leave out Days; the object moves back to the Frequent Access tier instead of getting a temporary copy:

aws s3api restore-object --bucket finance-archive --key invoices/2024/01/inv-0002.pdf \
  --restore-request '{"GlacierJobParameters":{"Tier":"Standard"}}'

Check status:

aws s3api head-object --bucket finance-archive --key invoices/2024/01/inv-0001.pdf \
  --query "{Class:StorageClass,Restore:Restore}"

• ongoing-request="true": restore in progress.
• ongoing-request="false", expiry-date="...": restored and readable until that date.
• No Restore field: no restore has been requested.

A second request for an object already restoring returns RestoreAlreadyInProgress. Re-issuing a restore on an already restored object with a new Days value changes its expiry.

#4. Build the manifest

Batch Operations reads a CSV of bucket,key (or bucket,key,versionId). Keys must be URL-encoded. A short Python script handles that and filters to archived objects only:

import boto3, csv, urllib.parse

s3 = boto3.client("s3")
bucket, prefix = "finance-archive", "invoices/2024/"
with open("manifest.csv", "w", newline="") as f:
    w = csv.writer(f)
    for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix=prefix):
        for o in page.get("Contents", []):
            if o["StorageClass"] in ("GLACIER", "DEEP_ARCHIVE"):
                w.writerow([bucket, urllib.parse.quote(o["Key"], safe="/")])

Upload it and capture its ETag, which the job requires:

aws s3 cp manifest.csv s3://ops-batch-jobs/restore/invoices-2024.csv
ETAG=$(aws s3api head-object --bucket ops-batch-jobs --key restore/invoices-2024.csv \
  --query ETag --output text | tr -d '"')

#5. Create the Batch Operations role

The role must trust batchoperations.s3.amazonaws.com and be allowed to restore objects, read the manifest and write the report:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:RestoreObject"],
      "Resource": "arn:aws:s3:::finance-archive/*" },
    { "Effect": "Allow", "Action": ["s3:GetObject", "s3:GetObjectVersion"],
      "Resource": "arn:aws:s3:::ops-batch-jobs/restore/*" },
    { "Effect": "Allow", "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::ops-batch-jobs/restore-reports/*" }
  ]
}

If the bucket uses SSE-KMS with a customer managed key, the role doesn't need KMS for the restore itself, but whoever reads the restored objects does.

#6. Launch the job

aws s3control create-job --account-id 111122223333 \
  --operation '{"S3InitiateRestoreObject":{"ExpirationInDays":7,"GlacierJobTier":"BULK"}}' \
  --manifest "{\"Spec\":{\"Format\":\"S3BatchOperations_CSV_20180820\",\"Fields\":[\"Bucket\",\"Key\"]},\"Location\":{\"ObjectArn\":\"arn:aws:s3:::ops-batch-jobs/restore/invoices-2024.csv\",\"ETag\":\"$ETAG\"}}" \
  --report '{"Bucket":"arn:aws:s3:::ops-batch-jobs","Prefix":"restore-reports","Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"FailedTasksOnly"}' \
  --priority 10 \
  --role-arn arn:aws:iam::111122223333:role/s3-batch-restore \
  --confirmation-required \
  --description "Restore invoices 2024 for audit request"

S3InitiateRestoreObject accepts BULK or STANDARD; Expedited isn't offered through Batch Operations. For Intelligent-Tiering archive tiers, leave out ExpirationInDays.

With --confirmation-required, the job waits in Suspended after S3 reads the manifest. Check the object count matches step 2, then start it:

aws s3control describe-job --account-id 111122223333 --job-id <job-id> \
  --query "Job.{Status:Status,Total:ProgressSummary.TotalNumberOfTasks,Done:ProgressSummary.NumberOfTasksSucceeded,Failed:ProgressSummary.NumberOfTasksFailed}"

aws s3control update-job-status --account-id 111122223333 --job-id <job-id> \
  --requested-job-status Ready

Warning

A completed Batch Operations job means every restore was requested, not that the data is readable. The retrieval itself still takes the hours in the tier table. Don't tell the requester the data is ready when the job finishes.

#7. Know when the data is ready

For large restores, don't poll head-object per key. Configure event notifications for s3:ObjectRestore:Completed on the bucket to an SQS queue or EventBridge and count the completions, or sample:

aws s3api list-objects-v2 --bucket finance-archive --prefix invoices/2024/ \
  --optional-object-attributes RestoreStatus \
  --query "Contents[?RestoreStatus.IsRestoreInProgress==\`false\`] | length(@)"

Read the failure report in s3://ops-batch-jobs/restore-reports/ for keys that failed (wrong encoding, deleted objects, permission errors) and rerun only those.

#8. Copy back to a hot class (only if needed)

If the data is needed beyond the restore window, copy it while the temporary copy exists. Copying over itself changes the storage class:

aws s3 cp s3://finance-archive/invoices/2024/ s3://finance-archive/invoices/2024/ \
  --recursive --storage-class STANDARD --force-glacier-transfer

For very large sets, use a second Batch Operations job with the Copy operation instead.

Caution

Moving objects out of Flexible Retrieval or Deep Archive before their minimum storage duration (90 and 180 days) triggers early deletion charges. Also check the bucket's lifecycle rules: a rule that transitions this prefix will archive the copies again. Exclude the prefix or tag the objects first.

#Checks and failure map

#Prevent it next time

• Record in the lifecycle rule's description or a tag which data is archived and the expected retrieval time, so requesters know restores take hours, not minutes.
• Keep S3 Inventory enabled on archive buckets so a manifest is always ready.
• Keep a tested restore role and a script in your runbook repository, and run a small restore drill each quarter.
• Use Glacier Instant Retrieval for data that is rarely read but must be readable immediately.