vLLM tool calling not working: fix auto tool choice errors, parsers and tool calls returned as text

#Symptom

Your agent worked against a hosted API, and it breaks when you move it to vLLM. You see one of these:

• 400: "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set.
• message.tool_calls is empty, while content holds something like <tool_call>{"name": ...}</tool_call>.
• The model answers in plain text, and never calls a tool.
• arguments contains broken or incomplete JSON.

Unlike a hosted API, vLLM runs raw model output through a parser that you have to choose. Model, chat template and parser all have to agree.

#Step 1: test with one request

Take your agent framework out of the picture. A plain curl shows what the server is really returning:

curl -s http://vllm:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-model",
    "messages": [{"role": "user", "content": "What is the weather in Pune?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto",
    "max_tokens": 300,
    "temperature": 0
  }' | jq '.choices[0] | {finish: .finish_reason, content: .message.content, tools: .message.tool_calls}'

Compare your result with this table:

#Step 2: set the server flags

vllm serve my-org/my-model \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

Check what the server is actually running with:

ps -ef | grep -- '--tool-call-parser'
# In Kubernetes
kubectl get deploy vllm -o jsonpath='{.spec.template.spec.containers[0].args}'

For your version, list the valid options:

vllm serve --help | grep -A3 -E 'tool-call-parser|enable-auto-tool-choice'

How tool_choice behaves:

If you only need to force one tool, a named tool_choice can be more reliable, as it uses constrained generation instead of parsing.

#Step 3: choose the parser that fits the model

The parser reads the format that the model writes. Each model family has its own. Common parser names include:

This table is a starting point. Newer models get new parsers, and some need a specific template. So read the tool calling section of the vLLM docs and the model card.

To see what the model really writes, look at content from the curl test, or send a request with no parser. Then pick the parser that matches that format.

If nothing built-in fits, vLLM supports custom parsers through a plugin (--tool-parser-plugin). But try another template first.

#Step 4: fix the chat template

The template turns your tools array into the prompt text. If it ignores tools, the model never knows they exist.

Check whether the model's template mentions tools:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("my-org/my-model")

tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
}}]

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "What is the weather in Pune?"}],
    tools=tools,
    tokenize=False,
    add_generation_prompt=True,
)
print(prompt)
print("get_weather" in prompt)

If get_weather isn't in the printed prompt, the template drops your tools. vLLM ships example templates in its repository, and you can pass one explicitly:

vllm serve my-org/my-model \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --chat-template ./tool_chat_template.jinja

A wrong template can also cause garbage output. Then there's the model itself: not all open-weight models are trained for tool use, and small or heavily quantized ones may do it badly.

Make sure tool descriptions are clear, too. A vague description means fewer calls. Check that the system prompt doesn't discourage tool use.

#Step 5: fix broken or truncated arguments

import json

call = response.choices[0].message.tool_calls[0]
print(response.choices[0].finish_reason)
print(repr(call.function.arguments))

try:
    args = json.loads(call.function.arguments)
except json.JSONDecodeError as e:
    print("Bad JSON:", e)

Warning

Parsing can succeed and the arguments can still be wrong. The model might invent an ID or a file path. Validate every argument against your schema and your access rules before you act on it, particularly for anything that changes state. See tool call failures.

#Step 6: fix the agent loop

If the raw server response is correct, the fault is in how your code uses it.

import json
from openai import OpenAI

client = OpenAI(base_url="http://vllm:8000/v1", api_key="not-used")
messages = [{"role": "user", "content": "What is the weather in Pune?"}]

first = client.chat.completions.create(
    model="my-model", messages=messages, tools=tools, tool_choice="auto", max_tokens=300,
)
msg = first.choices[0].message

if msg.tool_calls:
    messages.append(msg)   # must include the assistant's tool call
    for call in msg.tool_calls:
        args = json.loads(call.function.arguments)
        result = {"city": args["city"], "temp_c": 31}   # your real tool here
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(result),
        })

    final = client.chat.completions.create(model="my-model", messages=messages, tools=tools, max_tokens=300)
    print(final.choices[0].message.content)

Common mistakes:

Streaming returns tool calls in pieces, and you have to assemble them by index. Test without streaming first. Retries can also repeat side effects, so see tool calls executed twice.

#Build a small regression test

Parser behavior can change between vLLM releases, and between models. Keep a test that you run before each upgrade:

CASES = [
    ("What is the weather in Pune?", "get_weather"),
    ("Hello, how are you?", None),
]

for prompt, expected in CASES:
    r = client.chat.completions.create(
        model="my-model",
        messages=[{"role": "user", "content": prompt}],
        tools=tools, tool_choice="auto", temperature=0, max_tokens=300,
    )
    names = [c.function.name for c in (r.choices[0].message.tool_calls or [])]
    ok = (expected in names) if expected else not names
    print("PASS" if ok else "FAIL", prompt, names)

Run each case several times. Model output isn't fully repeatable, so one pass doesn't prove much.

#Prevent it next time

• Treat the model, parser and template as a set, and pin all three with the vLLM version.
• Keep these flags in your deployment config, not in someone's shell history.
• Validate tool arguments, and limit which tools an agent can use.
• Test tool calling after each upgrade. See vLLM operations and human-in-the-loop design.