Sev1: the LLM step in a production workflow starts returning invalid JSON

#Symptoms

• json.decoder.JSONDecodeError: Unterminated string starting at ...
• Schema validation errors: missing fields, wrong enum values, extra text like "Here is the JSON:".
• Downstream queue filling with failed messages.

#Step 1: categorise failures

Log raw responses for failed items (with PII controls) and check:

resp = client.chat.completions.create(...)
print(resp.choices[0].finish_reason, resp.usage)

#Step 2: enforce the schema at the API

Structured outputs constrain generation to your JSON schema (supported models and API versions only; check yours):

schema = {
  "type": "object",
  "properties": {
    "category": {"type": "string", "enum": ["incident", "request", "question"]},
    "priority": {"type": "integer", "minimum": 1, "maximum": 4},
    "summary":  {"type": "string"}
  },
  "required": ["category", "priority", "summary"],
  "additionalProperties": False
}

resp = client.chat.completions.create(
    model=DEPLOYMENT,
    messages=messages,
    response_format={"type": "json_schema",
                     "json_schema": {"name": "ticket_triage", "schema": schema, "strict": True}},
    max_tokens=800,
    temperature=0,
)

• Bedrock: define a tool whose input schema is your output schema and force the model to use it (toolChoice), then read the tool input.
• vLLM: guided decoding with a JSON schema (guided_json / structured outputs options, depending on the version).

json_object mode (just "valid JSON") is weaker than schema mode. Use schema mode where available.

#Step 3: validate anyway

from pydantic import BaseModel, ValidationError
from typing import Literal

class Triage(BaseModel):
    category: Literal["incident", "request", "question"]
    priority: int
    summary: str

def parse(raw: str) -> Triage | None:
    try:
        return Triage.model_validate_json(raw)
    except ValidationError:
        return None

Even with enforcement, validate: refusals, content filter results and truncation can still produce unusable output.

#Step 4: handle failures without stopping the pipeline

1. finish_reason == "length" → retry once with higher max_tokens or shorter input.
2. Validation error → retry once with the error message added ("Your previous output failed validation: ..."), at temperature 0.
3. Still failing → dead-letter the item with input, output and error, and continue. A human or a later job handles it.
4. Refusal / content filter → separate path; don't retry blindly.

Track the dead-letter rate. If it passes a threshold, alert.

#Step 5: stabilise the inputs

• Truncate or summarise very long inputs before the extraction step.
• Pin the model version for this workflow (see the model retirement post); upgrade only with tests.
• Keep prompts in version control; log the prompt version with each result.

Tip

Keep a fixed test set of 50–100 real inputs (including the weird ones that broke production) and run it on every prompt or model change. Schema-valid rate and field accuracy are the two numbers to watch.

#Prevent it next time

• Schema-enforced output for every machine-read LLM response.
• Validation + retry + dead-letter in the pipeline.
• Alerts on parse-failure and length finish rates.
• Regression test set run in CI for prompt/model changes.