vLLM structured outputs: requests that are slow, rejected with 400, or still return broken JSON

#Symptoms

You moved an extraction or classification workload onto self-hosted vLLM with JSON schema enforcement and one of these happens:

• The first request with a new schema takes several seconds longer than the rest.
• Requests fail with 400 Bad Request mentioning the schema, a regex, or an unsupported feature.
• The response passes through the constraint but json.loads still fails.
• The model emits pages of whitespace or newlines until max_tokens.
• Throughput drops when structured requests are mixed with normal chat traffic.

Constrained decoding works by compiling your schema into a grammar and masking every token that would break it. Each symptom above maps to one part of that pipeline.

#Check 1: which request format are you sending?

vLLM changed how structured output is requested. Mixing formats is a common cause of "the constraint was silently ignored" or a 400.

Sending both guided_* and structured_outputs in one request raises an error on versions that support both.

The portable form:

from openai import OpenAI
client = OpenAI(base_url="http://vllm:8000/v1", api_key="token")

schema = {
    "type": "object",
    "properties": {
        "severity": {"type": "string", "enum": ["sev1", "sev2", "sev3"]},
        "service":  {"type": "string"},
        "summary":  {"type": "string", "maxLength": 300},
    },
    "required": ["severity", "service", "summary"],
    "additionalProperties": False,
}

r = client.chat.completions.create(
    model="llama-8b",
    messages=[{"role": "user", "content": "Classify: checkout API 500s in eu-west"}],
    response_format={"type": "json_schema",
                     "json_schema": {"name": "triage", "schema": schema, "strict": True}},
    max_tokens=400,
)
print(r.choices[0].finish_reason, r.choices[0].message.content)

Always log finish_reason. Most "invalid JSON from a constrained model" reports are really length.

#Check 2: which backend ran?

vLLM supports several grammar backends: xgrammar, guidance, outlines and lm-format-enforcer, plus auto (the default). With auto, vLLM picks a backend per request based on what the schema needs and what each library supports, and that choice can change between releases.

Find the server setting and any fallback messages:

kubectl -n llm logs deploy/llama-8b | grep -iE "structured|guided|xgrammar|guidance|outlines|fallback" | tail -30
vllm serve --help | grep -iA2 structured

On recent versions the backend is set through the structured outputs config group, for example --structured-outputs-config.backend xgrammar; older versions used --guided-decoding-backend. Names vary by version; check your build's help output.

Tip

In production, pin the backend instead of using auto, and set disable_fallback to true in staging. Then an unsupported schema fails loudly in tests rather than quietly switching to a slower backend in production.

#Symptom → cause → fix

#Why the content gets worse under constraints

A grammar mask removes tokens; it does not make the model smarter. When the schema forces a field order or vocabulary the model would not naturally produce, quality can drop. Three practices help:

1. Describe the schema in the prompt as well as enforcing it. The model then wants to produce the shape the mask allows.
2. Put a short reasoning or evidence string field before the decision fields if you need justification. Fields are generated in order.
3. Prefer enum over free strings for categorical outputs. It is cheaper to enforce and easier to evaluate.

#Measure the overhead

Compare constrained and unconstrained latency for the same prompt set before rollout:

import time, statistics

def timed(fmt):
    t = []
    for p in prompts:
        s = time.perf_counter()
        client.chat.completions.create(model="llama-8b", messages=[{"role": "user", "content": p}],
                                       max_tokens=400, **fmt)
        t.append(time.perf_counter() - s)
    return statistics.median(t), sorted(t)[int(len(t) * 0.95)]

print("plain", timed({}))
print("json ", timed({"response_format": {"type": "json_schema",
                      "json_schema": {"name": "triage", "schema": schema}}}))

Run it twice: the first pass includes compilation, the second shows steady state. Also watch vllm:time_to_first_token_seconds and vllm:num_requests_waiting in Prometheus during the test; a growing queue with flat GPU utilisation points at CPU-side grammar work.

Warning

Do not upgrade vLLM in place for a structured-output workload without rerunning your schema tests. Backend defaults, supported schema features and request field names have all changed between minor releases.

#Fallback when the constraint fails

Even with enforcement, keep a parser-side safety net:

import json, jsonschema

def parse_or_none(resp):
    c = resp.choices[0]
    if c.finish_reason != "stop":
        return None
    try:
        obj = json.loads(c.message.content)
        jsonschema.validate(obj, schema)
        return obj
    except (json.JSONDecodeError, jsonschema.ValidationError):
        return None

On None, retry once with a larger max_tokens, then route to a fallback model or a human queue. Count each path in your metrics.

#Prevent it next time

• Keep a schema test suite: every production schema, run against the pinned backend with fallback disabled.
• Pin vLLM version and backend together; treat either change as a release.
• Warm up schemas in a readiness or post-start hook.
• Bound every string and array in the schema.
• Alert on the rate of finish_reason != "stop" for structured requests.