Same prompt, different answer at temperature 0: where LLM non-determinism comes from and how to live with it

#Symptoms

• An automated test asserts an exact LLM output; it passes and fails at random.
• Two users asking the same question get different wording, or different conclusions, at temperature=0.
• A regression appears "after deployment" but you can't reproduce it reliably.

temperature=0 means greedy decoding: always pick the most likely next token. That is deterministic only if the probabilities are computed identically every time. In real serving systems, they often aren't.

#Where the variation comes from

Once one token differs, everything after it can differ, so a small numeric effect produces a visibly different answer.

#Step 1: rule out your own inputs

Log the exact request: messages, tools, retrieved chunks (with IDs and order), parameters. Many "non-deterministic" bugs are actually:

• Retrieval returning chunks in a different order (ties in scores, approximate search).
• A date/time or user name injected into the system prompt.
• Different model deployment or version behind the same alias.

Hash the full request payload and compare hashes for "identical" calls.

#Step 2: use what the provider offers

• Seed parameter: some APIs accept seed. With the same seed, parameters and backend configuration, they aim for (but don't guarantee) repeatable output.
• System fingerprint / model version fields in the response identify the backend configuration. If the fingerprint changed, expect different output.

resp = client.chat.completions.create(
    model="gpt-4o-mini", messages=msgs, temperature=0, seed=1234)
print(resp.system_fingerprint)

Check your provider's docs for which parameters exist and what they promise; support differs between APIs and models.

#Step 3: self-hosted serving

On vLLM/SGLang you control more:

• Set a fixed seed for sampling.
• Serving with batch size 1 removes most batch-induced variation, but costs throughput.
• Some engines have added batch-invariant or deterministic modes that use kernels whose results don't depend on batch composition, at a performance cost. Check your engine version's docs for such options before relying on them.
• Pin the exact model files, engine version, CUDA/driver versions and GPU type for reproducibility tests.

#Step 4: design tests that tolerate variation

Exact string comparison is the wrong test for LLM output.

def passes(case, runs=5, threshold=0.8):
    ok = sum(check(case, call_llm(case.prompt)) for _ in range(runs))
    return ok / runs >= threshold

Track pass rates over time; a drop from 98% to 85% is a real regression even if single runs flap.

#Step 5: when users need consistency

• Cache answers for identical questions (exact or semantic cache) so the same question returns the same stored answer for a period.
• Constrain the output: structured outputs, enumerated choices and short answers vary less than free text.
• Make decisions in code: let the model extract facts, and apply business rules deterministically.
• Version prompts and models and record them with each answer, so you can explain why an answer changed.

Tip

For classification or routing tasks, read the model's choice from a constrained set (function call with an enum, or logprobs over labels). Near-ties then become visible as low-confidence cases you can route to a fallback, instead of random flips.

#Decision table

#Prevent it next time

• Never write exact-match assertions on free-text LLM output.
• Log request hashes, model versions and fingerprints with every call.
• Use structured outputs for anything a program consumes.
• Communicate to stakeholders that wording varies; decide which outputs must be stable and cache those.