Amazon Bedrock batch inference jobs fail, end PartiallyCompleted or expire: input records, IAM roles and error records

#The symptom

A nightly Bedrock batch inference job (a "model invocation job") did not produce what you expected. Usually it looks like one of these:

• status is Failed minutes after submission and nothing was written to the output bucket.
• status is Completed or PartiallyCompleted, but downstream counts are short.
• The job sat in Submitted, Scheduled or InProgress for hours and then went to Expired.

Batch jobs give you no live request log, so you have to work from three sources: the job record, the manifest.json.out file and the per-record .jsonl.out output files.

#Step 1: read the job record

# Recent failures and partial runs
aws bedrock list-model-invocation-jobs --status-equals Failed \
  --query 'invocationJobSummaries[].{name:jobName,status:status,submitted:submitTime,msg:message}' --output table

aws bedrock list-model-invocation-jobs --status-equals PartiallyCompleted \
  --query 'invocationJobSummaries[].[jobName,jobArn]' --output text

# One job in detail
aws bedrock get-model-invocation-job --job-identifier <job-arn-or-id> \
  --query '{status:status,message:message,model:modelId,role:roleArn,in:inputDataConfig,out:outputDataConfig,timeoutHours:timeoutDurationInHours,expires:jobExpirationTime}'

The message field is the first thing to read. For a Failed job it nearly always names the problem: S3 access, malformed input or an unsupported model.

Job statuses you will see: Submitted, Validating, Scheduled, InProgress, Completed, PartiallyCompleted, Failed, Stopping, Stopped and Expired.

#Step 2: map the status to a cause

Tip

Batch jobs share a Region's batch capacity with other customers. Build your schedule on the idea that a job may take most of its timeout window. If a downstream report needs the results by 08:00, submitting at 02:00 is a bet.

#Step 3: check the service role

Bedrock assumes the role passed in --role-arn. That role needs a trust policy for bedrock.amazonaws.com, plus S3 permissions on both prefixes:

ROLE=BedrockBatchRole
aws iam get-role --role-name $ROLE --query 'Role.AssumeRolePolicyDocument'
aws iam list-attached-role-policies --role-name $ROLE
aws iam list-role-policies --role-name $ROLE

The minimum S3 permissions are s3:GetObject and s3:ListBucket on the input bucket and prefix, plus s3:PutObject on the output prefix. If the buckets use SSE-KMS, the role also needs kms:Decrypt (input) and kms:GenerateDataKey (output), and the key policy has to allow it. A common miss is an output bucket in a different account, or a bucket policy that denies any principal outside a specific VPC endpoint.

You can test S3 access by assuming the role from the CLI, if your own identity is allowed to assume it:

aws sts assume-role --role-arn arn:aws:iam::123456789012:role/BedrockBatchRole \
  --role-session-name batch-test --query Credentials
# export the returned keys, then:
aws s3 ls s3://my-batch-input/jobs/2026-11-09/

#Step 4: validate the input file before submitting

Each line in the JSONL file must be one object containing a recordId and a modelInput, and modelInput must be the body that model's InvokeModel call would accept. For Anthropic models on Bedrock, that means the Messages format with anthropic_version, max_tokens and messages.

import json, sys

ids, bad = set(), 0
with open(sys.argv[1], encoding="utf-8") as f:
    for n, line in enumerate(f, 1):
        try:
            rec = json.loads(line)
        except json.JSONDecodeError as e:
            print(f"line {n}: invalid JSON: {e}"); bad += 1; continue
        if "recordId" not in rec or "modelInput" not in rec:
            print(f"line {n}: missing recordId or modelInput"); bad += 1; continue
        if rec["recordId"] in ids:
            print(f"line {n}: duplicate recordId {rec['recordId']}"); bad += 1
        ids.add(rec["recordId"])
print(f"{len(ids)} records, {bad} problems")

Before a large run, send one modelInput through a synchronous call. That catches a wrong body format in seconds instead of waiting for a whole job to fail:

head -1 input.jsonl | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['modelInput']))" > body.json
aws bedrock-runtime invoke-model --model-id <model-or-profile-id> \
  --body fileb://body.json --content-type application/json out.json && cat out.json

Warning

Make recordId unique and meaningful, for example the source ticket ID. Output files aren't guaranteed to keep input order, and recordId is the only reliable way to join results back to source rows and to find which records still need a re-run.

#Step 5: read the manifest and the error records

When a job finishes, Bedrock writes a folder named after the job ID under your output prefix. It contains one <input-file>.jsonl.out per input file and a manifest.json.out with the counts:

OUT=s3://my-batch-output/jobs/2026-11-09/
JOB_ID=<last part of the job ARN>
aws s3 ls ${OUT}${JOB_ID}/
aws s3 cp ${OUT}${JOB_ID}/manifest.json.out - | python -m json.tool

The manifest includes totalRecordCount, processedRecordCount, successRecordCount, errorRecordCount, inputTokenCount and outputTokenCount. If processedRecordCount is lower than totalRecordCount, records were never attempted. If errorRecordCount is above zero, individual records failed.

Records that failed have an error object (with errorCode and errorMessage) in place of modelOutput. Summarise them:

aws s3 cp ${OUT}${JOB_ID}/ ./out --recursive --exclude "*" --include "*.jsonl.out"
python - <<'EOF'
import json, glob, collections
errs = collections.Counter()
for p in glob.glob("out/*.jsonl.out"):
    for line in open(p, encoding="utf-8"):
        r = json.loads(line)
        if "error" in r:
            e = r["error"]
            errs[(e.get("errorCode"), e.get("errorMessage", "")[:120])] += 1
for k, v in errs.most_common(20):
    print(v, k)
EOF

400 errors here are almost always input problems: a prompt longer than the model's context window, a malformed message list, or an image the model can't read. Fix the records themselves. Re-running them unchanged gives the same errors.

#Step 6: re-run only what's missing

Build a new input file from the records that have no successful output:

import json, glob
done = set()
for p in glob.glob("out/*.jsonl.out"):
    for line in open(p, encoding="utf-8"):
        r = json.loads(line)
        if "modelOutput" in r:
            done.add(r["recordId"])
with open("input.jsonl", encoding="utf-8") as src, open("retry.jsonl", "w", encoding="utf-8") as dst:
    for line in src:
        if json.loads(line)["recordId"] not in done:
            dst.write(line)

If the retry file is below the per-job minimum record count, either combine it with the next scheduled batch or send those records through the synchronous API.

aws s3 cp retry.jsonl s3://my-batch-input/jobs/2026-11-09-retry/retry.jsonl
aws bedrock create-model-invocation-job --job-name tickets-20261109-retry \
  --role-arn arn:aws:iam::123456789012:role/BedrockBatchRole \
  --model-id <model-or-profile-id> \
  --input-data-config 's3InputDataConfig={s3Uri=s3://my-batch-input/jobs/2026-11-09-retry/}' \
  --output-data-config 's3OutputDataConfig={s3Uri=s3://my-batch-output/jobs/2026-11-09-retry/}' \
  --timeout-duration-in-hours 48

A job that is stuck and no longer needed can be stopped with aws bedrock stop-model-invocation-job --job-identifier <job-arn>.

Caution

Stopping a job or re-running a whole input file can double-process records that already succeeded. If downstream steps write tickets or send messages, deduplicate by recordId before loading results. Don't rely on "this job only ran once".

#Prevent it next time

• Run the validation script and one synchronous test call as the first step of the pipeline, before create-model-invocation-job.
• Check the current batch quotas (minimum and maximum records per job, file size, concurrent jobs) in Service Quotas for your model, and split input to fit them.
• Give each run its own input and output prefix, named by date, so retries never overwrite results.
• After each job, alert on status other than Completed, or on errorRecordCount > 0 in the manifest. An EventBridge rule on Bedrock job state changes, or a scheduled get-model-invocation-job poll, both work.
• Keep a synchronous fallback path for small retry sets and urgent records.