Sev1: self-hosted LLM returns gibberish, never stops or leaks special tokens after a deploy — chat template and tokenizer fixes

#Symptoms

• Health checks pass, but answers:
  • run until max_tokens and repeat themselves,
  • contain raw markers like <|im_end|>, <|eot_id|>, [/INST] or assistant\n,
  • are empty, or answer a different question than asked.
• Started right after changing the model, the server version or the deployment config.

#1. Test the raw endpoint

curl -s http://llm.ai.svc:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "chat-prod",
  "messages": [{"role":"user","content":"Reply with exactly: OK"}],
  "max_tokens": 20, "temperature": 0
}' | jq '.choices[0] | {finish_reason, content: .message.content}'

#2. Roll back first if you can

If the previous image/model version worked, roll back and investigate in staging:

kubectl rollout undo deployment/vllm-chat -n ai
kubectl rollout status deployment/vllm-chat -n ai

#3. Check the chat template

Chat models need the right template to turn messages into the prompt format they were trained on. Servers normally read it from the model's tokenizer_config.json. Problems appear when:

• The model was copied without tokenizer_config.json / chat_template.jinja.
• A fine-tuned or quantized repack changed or dropped the template.
• A --chat-template flag in the deployment points to a template for a different model family.

kubectl exec -n ai deploy/vllm-chat -- sh -c 'ls /models/chat-prod; grep -c chat_template /models/chat-prod/tokenizer_config.json'
kubectl get deploy vllm-chat -n ai -o jsonpath='{.spec.template.spec.containers[0].args}'

Fix by restoring the original tokenizer files from the base model, or passing the correct template file:

vllm serve /models/chat-prod --served-model-name chat-prod \
  --chat-template /models/chat-prod/chat_template.jinja

#4. Check stop tokens and generation config

The model must stop at its end-of-turn token. vLLM reads generation_config.json (including eos_token_id) by default in recent versions; a missing file or a server flag that ignores it can remove the stop.

kubectl exec -n ai deploy/vllm-chat -- cat /models/chat-prod/generation_config.json

As a quick mitigation, pass stop strings from the client or gateway:

{"stop": ["<|im_end|>", "<|eot_id|>"]}

Use only tokens valid for your model family.

#5. Weights, tokenizer and dtype

• Tokenizer from one model with weights from another produces garbage. Check config.json model_type and vocabulary size match the tokenizer.
• A quantized build needs the matching --quantization setting (or auto-detection) and a supported dtype on your GPU.
• Compare checksums of model files against the source to rule out a partial download.

Tip

Keep an "output smoke test" in the deployment pipeline: 5-10 fixed prompts with temperature 0 and expected short answers. Health checks alone don't catch this failure.

#6. Verify

Re-run the curl test and a few real conversations. Check finish_reason is stop for normal answers and that average output length returns to its usual level in your metrics.

#Prevent it next time

• Pin the model by revision/hash and store the full model directory (weights, tokenizer, templates, generation config) as one artifact.
• Canary new model or server versions on a small share of traffic, watching output length and finish_reason.
• Alert on a jump in average output tokens and in finish_reason=length.
• Don't pass --chat-template unless you must; when you do, version it with the model.