Azure OpenAI batch jobs fail, stay in validating or never finish: input file errors, enqueued token quota and stuck jobs

#Symptom

A scheduled bulk job (summaries, re-classification, evaluation runs) uses the Azure OpenAI Batch API and one of these happens:

• The batch goes to failed a few minutes after submission and the output file is empty.
• The batch stays in validating for a long time, or fails with token_limit_exceeded.
• The batch reaches completed, but many lines in the output have errors or some custom_ids are missing.
• The batch is still in_progress the next morning and the downstream job is waiting.

Each of these has a different cause. Start by reading the job object.

#Quick checks

#1. Read the batch status and its errors

import os
from openai import OpenAI

client = OpenAI(
    base_url=f"https://{os.environ['AOAI_RESOURCE']}.openai.azure.com/openai/v1/",
    api_key=os.environ["AOAI_KEY"],
)

job = client.batches.retrieve("batch_abc123")
print(job.status, job.request_counts)
print(job.errors)          # validation errors when status is failed
print(job.output_file_id, job.error_file_id)

The same thing over REST:

curl -s "https://$AOAI_RESOURCE.openai.azure.com/openai/v1/batches/batch_abc123" \
  -H "api-key: $AOAI_KEY" | jq '{status, request_counts, errors}'

#2. Know what each status means

#3. Confirm the deployment type

Batch requests only work against a batch deployment. A Standard or Global Standard deployment of the same model will not accept them.

az cognitiveservices account deployment list \
  -g rg-ai-prod -n aoai-prod \
  --query "[].{name:name, model:properties.model.name, version:properties.model.version, sku:sku.name}" -o table

Look for a SKU of GlobalBatch or DataZoneBatch (the portal shows "Global Batch" and "Data Zone Batch").

#Input file errors (status failed)

Validation errors show up in job.errors.data[].code. The common ones:

Validate the file before you upload it. Most of the codes above can be caught locally:

import json, sys
from collections import Counter

ids, models, urls, bad = Counter(), set(), set(), []
with open(sys.argv[1], encoding="utf-8") as f:
    for n, line in enumerate(f, 1):
        try:
            r = json.loads(line)
        except json.JSONDecodeError as e:
            bad.append((n, str(e))); continue
        ids[r.get("custom_id")] += 1
        models.add(r["body"].get("model"))
        urls.add(r.get("url"))

dupes = [k for k, v in ids.items() if v > 1]
print(f"lines={sum(ids.values())} bad_json={len(bad)} dupes={len(dupes)} models={models} urls={urls}")
if bad or dupes or len(models) != 1 or len(urls) != 1:
    sys.exit(1)

Run it as a gate in the pipeline, before client.files.create().

Tip

Submit a 20–50 line file to a new deployment first. A format mistake in a 100,000-line file costs you the whole validation wait before you find out.

#token_limit_exceeded and long validating times

Batch quota is measured in enqueued tokens: the tokens in every batch file that has not yet reached a terminal state, for that deployment. A new file is rejected if it would push the total over the limit, even if the file itself is small.

Checks:

1. List what is still queued or running on the deployment:

for b in client.batches.list(limit=100):
    if b.status in ("validating", "in_progress", "finalizing"):
        print(b.id, b.status, b.created_at, b.request_counts)

2. Check the deployment's batch quota in the Foundry portal (Quota page) or with the usage list:

az cognitiveservices usage list --location swedencentral -o table

Usage names vary by model and deployment type; look for the batch entry for your model.

Fixes:

• Stagger submissions. Submit the next file only when the previous one reaches completed, or when the enqueued total leaves room for it.
• Retry with backoff when submission fails with token_limit_exceeded, instead of failing the pipeline:

import time

def create_with_backoff(file_id, retries=8, delay=60):
    for _ in range(retries):
        job = client.batches.create(input_file_id=file_id,
                                    endpoint="/chat/completions",
                                    completion_window="24h")
        while job.status == "validating":
            time.sleep(30)
            job = client.batches.retrieve(job.id)
        codes = [e.code for e in (job.errors.data if job.errors else [])]
        if job.status == "failed" and "token_limit_exceeded" in codes:
            time.sleep(delay); delay *= 2; continue
        return job
    raise RuntimeError("enqueued token quota still exhausted")

• Enable dynamic quota on batch deployments where available; Microsoft recommends it to reduce quota failures.
• Request more quota if the steady-state workload needs it. Don't guess numbers; check the current batch quota for your subscription and model.
• Trim requests. Set a sensible max_tokens per line and remove repeated boilerplate from prompts. Enqueued tokens include prompt tokens, so long prompts fill the queue quickly.

#Completed, but results are missing or failed

completed means the job finished, not that every request succeeded.

import json

ok, failed = {}, []
for line in client.files.content(job.output_file_id).text.splitlines():
    r = json.loads(line)
    if r.get("error") or r["response"]["status_code"] != 200:
        failed.append(r["custom_id"]); continue
    ok[r["custom_id"]] = r["response"]["body"]["choices"][0]["message"]["content"]

if job.error_file_id:
    for line in client.files.content(job.error_file_id).text.splitlines():
        failed.append(json.loads(line)["custom_id"])

missing = set(all_input_ids) - set(ok) - set(failed)
print(len(ok), len(failed), len(missing))

Resubmit failed and missing IDs as a new, small batch. Keep writes idempotent (upsert by custom_id) so reruns never duplicate data.

#Job runs past 24 hours

completion_window must be 24h, but that is a target. Large jobs at busy times can take longer. Decide what the pipeline does when the deadline passes:

• Downstream jobs should check for completed instead of assuming the batch is done by a fixed time.
• If the result is needed now, cancel and take what is finished:

curl -s -X POST "https://$AOAI_RESOURCE.openai.azure.com/openai/v1/batches/batch_abc123/cancel" \
  -H "api-key: $AOAI_KEY"

The job stays in cancelling for a few minutes. Then the output file holds the partial results. Merge them, and send the remaining IDs to a synchronous fallback or the next night's batch.

Warning

Cancelling throws away the queue position of every unfinished request. Do it only when a late result costs more than a rerun.

#Prevent it next time

• Validate every JSONL file in the pipeline (JSON, unique IDs, one model, one URL) before upload.
• Point model at the batch deployment name, held in config, not hard-coded per script.
• Track enqueued work per deployment and submit files in order, with backoff on token_limit_exceeded.
• Alert on: batch failed, validating longer than your normal baseline, failed-line ratio over a threshold, and no completed by the business deadline.
• Store each job's ID, input file ID and counts with the pipeline run, so a rerun can pick up only what's missing.
• Keep a synchronous fallback path, rate-limited, for small urgent remainders.