Transformation: moving bulk LLM jobs from synchronous calls to batch inference

#Which jobs to move

Benefits: lower per-token price on most providers (check current pricing), separate quota so bulk jobs don't cause 429s for production, no client-side retry loops.

#1. Build the request file

OpenAI / Azure OpenAI JSONL, one request per line:

{"custom_id": "ticket-100231", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1-mini", "messages": [{"role": "system", "content": "Summarise the ticket in two lines."}, {"role": "user", "content": "..."}], "max_tokens": 120}}

• custom_id is how you join results back. Use your record key.
• Azure OpenAI: model is the batch deployment name, and the URL form follows the Azure batch docs for your API version.
• Respect per-file limits on request count and file size (check provider docs); split large jobs into several files.

#2. Submit and track

f = client.files.create(file=open("batch_0001.jsonl", "rb"), purpose="batch")
job = client.batches.create(input_file_id=f.id, endpoint="/v1/chat/completions", completion_window="24h",
                            metadata={"job": "ticket-summaries", "run": "2026-10-25"})

job = client.batches.retrieve(job.id)
print(job.status, job.request_counts)   # validating → in_progress → finalizing → completed / failed / expired

Bedrock equivalent: upload JSONL records (recordId, modelInput in the model's native format) to S3, then:

aws bedrock create-model-invocation-job --job-name tickets-20261025 \
  --role-arn arn:aws:iam::123456789012:role/BedrockBatchRole \
  --model-id <model-id> \
  --input-data-config '{"s3InputDataConfig":{"s3Uri":"s3://ai-batch/in/"}}' \
  --output-data-config '{"s3OutputDataConfig":{"s3Uri":"s3://ai-batch/out/"}}'
aws bedrock get-model-invocation-job --job-identifier <job-arn>

Bedrock batch has minimum record counts and per-model support. Check the quotas page.

#3. Collect and merge results

out = client.files.content(job.output_file_id).text
for line in out.splitlines():
    r = json.loads(line)
    if r.get("error") or r["response"]["status_code"] != 200:
        failed.append(r["custom_id"]); continue
    text = r["response"]["body"]["choices"][0]["message"]["content"]
    upsert_summary(r["custom_id"], text)

if job.error_file_id:
    errors = client.files.content(job.error_file_id).text   # malformed lines, etc.

Results are not guaranteed in input order. Always join on custom_id.

#4. Handle failures and expiry

• Collect failed and missing custom_ids; resubmit them as a new, smaller batch.
• If a batch expires (not finished in the window), completed requests are still returned; resubmit only the rest.
• Make the merge idempotent (upsert by key) so a rerun can't create duplicates.

#5. Orchestrate

Scheduler (ADF / Step Functions / Airflow / Databricks job)
  → build JSONL files → upload → create batch
  → poll every 10–15 min (or event-driven where supported)
  → on complete: merge results, resubmit failures, alert on failure rate > threshold

#6. Measure

Tip

Validate a 50-line file first. Most failed batches are format errors (wrong url, model/deployment name, malformed JSON), and you only discover them after the validation step finishes.

#Checklist

• Bulk jobs identified that can wait for the completion window
• JSONL builder with stable custom_id
• Submit, poll, merge, resubmit automated
• Idempotent writes; alerts on failure rate
• Cost and 429 impact compared with the old loop