S3 Batch Operations cheat sheet: manifests, copy, tagging, Lambda, Batch Replication and reading failure reports

#When to use it

Use Batch Operations when a change has to touch millions of objects. Examples: re-encrypting with a new KMS key, adding tags for lifecycle, copying to a new account, backfilling replication, or running custom logic per object. A aws s3 cp --recursive loop would take days at that scale and would leave no audit trail. Batch Operations gives you per-object retries, a completion report and a single job you can cancel.

All commands below use the S3 Control API, which takes --account-id and is regional (the job runs in the region of the bucket in the manifest).

ACCOUNT=111122223333
REGION=eu-west-1
ROLE=arn:aws:iam::$ACCOUNT:role/s3-batch-ops

#The IAM role

Trust policy principal: batchoperations.s3.amazonaws.com. Permissions: read the manifest, write the report, plus whatever the operation needs on the objects (s3:GetObject/s3:PutObject for copy, s3:PutObjectTagging, lambda:InvokeFunction, KMS Decrypt/GenerateDataKey for encrypted objects, and so on).

aws iam create-role --role-name s3-batch-ops --assume-role-policy-document '{
  "Version":"2012-10-17","Statement":[{"Effect":"Allow",
  "Principal":{"Service":"batchoperations.s3.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

#Manifests: three ways to say which objects

Tip

Keys in a CSV manifest must be URL-encoded. Keys with spaces, + or non-ASCII characters that aren't encoded are the most common cause of a 404 NoSuchKey failure rate.

Build a precise manifest from Inventory with Athena:

-- objects still on the old KMS key, written as bucket,key
SELECT bucket, url_encode(key) AS key
FROM s3_inventory.prod_data
WHERE dt = '2026-11-08-01-00'
  AND encryption_status = 'SSE-KMS'
  AND bucket_key_status = 'DISABLED'
  AND is_latest = true;

Get the manifest ETag (required for CSV and Inventory manifests):

ETAG=$(aws s3api head-object --bucket ops-manifests --key reencrypt/manifest.csv --query ETag --output text | tr -d '"')

#Job 1: copy in place with a new KMS key and Bucket Keys

aws s3control create-job --region $REGION --account-id $ACCOUNT \
  --operation '{"S3PutObjectCopy":{"TargetResource":"arn:aws:s3:::prod-data",
     "SSEAwsKmsKeyId":"arn:aws:kms:eu-west-1:111122223333:key/NEW-KEY-ID",
     "BucketKeyEnabled":true,"MetadataDirective":"COPY"}}' \
  --manifest "{\"Spec\":{\"Format\":\"S3BatchOperations_CSV_20180820\",\"Fields\":[\"Bucket\",\"Key\"]},
     \"Location\":{\"ObjectArn\":\"arn:aws:s3:::ops-manifests/reencrypt/manifest.csv\",\"ETag\":\"$ETAG\"}}" \
  --report '{"Bucket":"arn:aws:s3:::ops-manifests","Prefix":"reports","Format":"Report_CSV_20180820",
     "Enabled":true,"ReportScope":"FailedTasksOnly"}' \
  --priority 10 --role-arn $ROLE --client-request-token "$(uuidgen)" \
  --description "re-encrypt prod-data to new key" --confirmation-required

Warning

S3PutObjectCopy uses CopyObject, so it handles objects up to 5 GB only. Larger objects fail and need a multipart copy, for example a Lambda-based job. An in-place copy also creates a new version on versioned buckets (storage cost doubles until lifecycle expires the noncurrent versions), and it can trigger replication and event notifications for every object.

Copy keeps tags by default, but not ACLs or Object Lock settings unless you specify them. Check the S3PutObjectCopy fields (AccessControlGrants, ObjectLockMode, NewObjectTagging, StorageClass) against what the destination needs.

#Job 2: tag objects (replace the tag set)

aws s3control create-job --region $REGION --account-id $ACCOUNT \
  --operation '{"S3PutObjectTagging":{"TagSet":[{"Key":"retention","Value":"7y"},{"Key":"owner","Value":"finance"}]}}' \
  --manifest "{\"Spec\":{\"Format\":\"S3BatchOperations_CSV_20180820\",\"Fields\":[\"Bucket\",\"Key\"]},
     \"Location\":{\"ObjectArn\":\"arn:aws:s3:::ops-manifests/tagging/manifest.csv\",\"ETag\":\"$ETAG\"}}" \
  --report '{"Bucket":"arn:aws:s3:::ops-manifests","Prefix":"reports","Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"AllTasks"}' \
  --priority 20 --role-arn $ROLE --client-request-token "$(uuidgen)" --no-confirmation-required

S3PutObjectTagging replaces the object's whole tag set. To add one tag and keep the others, use a Lambda job that reads the existing tags and merges them.

#Job 3: invoke Lambda per object

aws s3control create-job --region $REGION --account-id $ACCOUNT \
  --operation '{"LambdaInvoke":{"FunctionArn":"arn:aws:lambda:eu-west-1:111122223333:function:fix-object:live",
     "InvocationSchemaVersion":"2.0","UserArguments":{"dryRun":"false"}}}' \
  --manifest-generator '{"S3JobManifestGenerator":{"SourceBucket":"arn:aws:s3:::prod-data","EnableManifestOutput":false,
     "Filter":{"KeyNameConstraint":{"MatchAnyPrefix":["exports/2025/"]},"CreatedBefore":"2026-01-01T00:00:00Z"}}}' \
  --report '{"Bucket":"arn:aws:s3:::ops-manifests","Prefix":"reports","Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"FailedTasksOnly"}' \
  --priority 10 --role-arn $ROLE --client-request-token "$(uuidgen)" --confirmation-required

The function must return a result per task:

def handler(event, context):
    task = event["tasks"][0]
    bucket, key = task["s3Bucket"], task["s3Key"]        # schema 2.0 field names
    try:
        process(bucket, key)
        code, msg = "Succeeded", "ok"
    except ThrottlingError as e:
        code, msg = "TemporaryFailure", str(e)            # Batch Operations retries it
    except Exception as e:
        code, msg = "PermanentFailure", str(e)            # recorded in the report
    return {"invocationSchemaVersion": "2.0", "treatMissingKeysAs": "PermanentFailure",
            "invocationId": event["invocationId"],
            "results": [{"taskId": task["taskId"], "resultCode": code, "resultString": msg[:1000]}]}

Invoke a published alias, not $LATEST. Give the function reserved concurrency so the job can't starve other functions in the account. A Batch Operations job can push Lambda concurrency very high.

#Job 4: Batch Replication of existing objects

Replication rules only apply to new writes. To backfill existing objects (or retry FAILED ones), use S3ReplicateObject with a generated manifest:

aws s3control create-job --region $REGION --account-id $ACCOUNT \
  --operation '{"S3ReplicateObject":{}}' \
  --manifest-generator '{"S3JobManifestGenerator":{"SourceBucket":"arn:aws:s3:::prod-data","EnableManifestOutput":true,
     "ManifestOutputLocation":{"Bucket":"arn:aws:s3:::ops-manifests","ManifestPrefix":"replication","ManifestFormat":"S3InventoryReport_CSV_20211130"},
     "Filter":{"EligibleForReplication":true,"ObjectReplicationStatuses":["NONE","FAILED"]}}}' \
  --report '{"Bucket":"arn:aws:s3:::ops-manifests","Prefix":"reports","Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"AllTasks"}' \
  --priority 10 --role-arn $ROLE --client-request-token "$(uuidgen)" --no-confirmation-required

The bucket must already have a replication configuration, and the Batch Operations role needs replication permissions in addition to the replication role's own.

#Running, watching and stopping jobs

aws s3control list-jobs --account-id $ACCOUNT --region $REGION --job-statuses Active Suspended Preparing \
  --query 'Jobs[].[JobId,Status,Operation,ProgressSummary.NumberOfTasksSucceeded,ProgressSummary.NumberOfTasksFailed]' --output table

aws s3control describe-job --account-id $ACCOUNT --region $REGION --job-id $JOB \
  --query 'Job.{status:Status,progress:ProgressSummary,failures:FailureReasons,reason:StatusUpdateReason}'

# Confirm a job created with --confirmation-required
aws s3control update-job-status --account-id $ACCOUNT --region $REGION --job-id $JOB --requested-job-status Ready

# Stop it
aws s3control update-job-status --account-id $ACCOUNT --region $REGION --job-id $JOB \
  --requested-job-status Cancelled --status-update-reason "wrong key in manifest"

# Reprioritise (higher number runs first)
aws s3control update-job-priority --account-id $ACCOUNT --region $REGION --job-id $JOB --priority 50

A job fails automatically if more than 50% of tasks fail once it has attempted at least 1,000 tasks. That protects you from a bad manifest. Small jobs never reach that threshold, so always read the report.

#Reading the completion report

The report has a manifest.json that lists one or more CSVs with columns: bucket, key, version ID, task status, error code, HTTP status, result message.

aws s3 cp s3://ops-manifests/reports/job-$JOB/ ./report --recursive
cut -d, -f4,5,6 report/results/*.csv | sort | uniq -c | sort -rn | head

Build a retry manifest from the failed rows (for FailedTasksOnly reports the CSV is already almost a manifest):

cut -d, -f1,2 report/results/*.csv > retry.csv
aws s3 cp retry.csv s3://ops-manifests/retry/manifest.csv

#Before you press Ready

• Run the job against a 100-object manifest first and read that report.
• Use --confirmation-required for anything that rewrites objects.
• Check versioning, replication and event notifications on the target bucket. A copy job touches every object once and triggers each of them.
• Set the report bucket's lifecycle so old reports don't accumulate.