vLLM Tool Calling Not Working? 1 in 12 Calls Fail by Default. Here's the Fix

If your vLLM tool calling is not working with Kimi or DeepSeek, and your agent keeps emitting broken JSON, inventing arguments or looping on the same call, check your server before you blame the model. Ubicloud measured Kimi K2.6 on vLLM 0.28 and got schema-valid tool calls only 92% of the time. That is 2 failures in every 25 calls, roughly 1 in 12.

With constrained decoding turned on, Ubicloud got to 100%. vLLM doesn't turn it on for you, even though Moonshot, who make the model, tell vendors to use it (they call it guided encoding). On vLLM 0.31+ the fix is one server flag, --tool-strict-level parameter, and the rest of this post is about why and how to check it worked.

A lot of the "open models are bad at agents" talk I've seen comes down to a server default.

What broken vLLM tool calls look like

On the client it rarely says "tool call failed". Instead you get:

• arguments that won't parse as JSON, or parse but miss a required field
• a parameter the tool never declared, or an enum value that isn't in the enum
• the agent reading back its own validation error, retrying with the same mistake and burning your token budget in a loop
• a framework that quietly drops the call and lets the model answer in prose

Each of these looks like a model problem, so people swap models or add three paragraphs of "ALWAYS RETURN VALID JSON" to the system prompt. Neither fixes it.

92% sounds fine until you chain calls

Per-call accuracy compounds. At 92%, a 10-step agent run gets through clean 0.92^10 ≈ 43% of the time. At 20 steps it's about 19%. In the Hacker News thread, one commenter worked out that even 95% per call gives you about an 8% chance of finishing a 50-call sequential task. Someone else in the same thread said they run around 300 tool calls per prompt.

92% wasn't a fluke of Ubicloud's setup, either. Moonshot's own K2 Vendor Verifier results (test date 2025-11-15) show the vLLM reference deployment at 87.22% schema accuracy on K2-thinking and 76.00% on K2-0905, while Moonshot's API and providers like Fireworks hit 100% serving the same weights.

Why vLLM leaves tool calls unconstrained by default

Without constraints, the model samples tool-call tokens like any other text. Kimi K2.6's format looks like this:

<|tool_calls_section_begin|><|tool_call_begin|>functions.webfetch:0<|tool_call_argument_begin|>{"url": "https://ubicloud.com", "format": "markdown"}<|tool_call_end|><|tool_calls_section_end|>

Nothing stops the model from writing an unknown function name, a non-integer index or JSON that ignores your schema. Constrained decoding uses XGrammar structural tags to mask the vocabulary at each step, so once the model opens a tool call it can only produce a declared tool name, then an integer index, then JSON that matches that tool's schema, then the end tag.

vLLM supports this, just not on every request. PR #45600, merged in June 2026, changed tool_choice="auto" so structural tags apply only when at least one tool sets "strict": true. A missing strict counts as not strict. A reviewer on that PR objected that Claude Code's protocol doesn't include a strict field at all, which left Messages API clients with no obvious opt-in; the PR ended up adding strict to vLLM's Anthropic tool model. Plenty of OpenAI-style agent code doesn't set it either.

So if you're on tool_choice="auto" (most agents are) without strict on your tools, you're sampling free-form. Ubicloud originally added constrained decoding in their own vLLM fork. After vLLM shipped the feature in v0.25 they switched to upstream, and their score dropped back to 91% because of this default change.

The fix: --tool-strict-level in vLLM 0.31+

vLLM 0.31.0 added a server-side floor, --tool-strict-level (PR #56268). From the tool calling docs:

Value	What gets constrained
auto	Default for most parsers. Structural tags only for required/named tool choice, or auto when a tool sets strict: true
function	Tool-call envelope (markup, function name) on every request with tools. Arguments stay free unless the tool is strict
parameter	Everything in function, plus argument schemas for every tool, as if all were strict

If you don't set it, the parser's default applies: function for glm47, auto for everything else, Kimi included.

For a Kimi deployment I'd go straight to parameter. Invented args and missing required fields are argument problems, and function doesn't touch those:

vllm serve <your-kimi-k2.6-checkpoint> \
    --enable-auto-tool-choice \
    --tool-call-parser kimi_k2 \
    --tool-strict-level parameter

Two things to know before you deploy it:

• It's a floor. It never loosens what a request already asked for, and it doesn't force a tool call. The grammar engages only after the model opens the tool-call wrapper itself.
• VLLM_ENFORCE_STRICT_TOOL_CALLING=false turns structural tags off entirely and overrides this flag. If someone set it while chasing a bug months ago, the flag does nothing.

Stuck below 0.31? Set "strict": true on every function definition you send. Under tool_choice="auto" that turns on the same structural tags without a server flag.

Where constrained decoding still bites

This has costs, and the HN thread argues about them at length.

A grammar guarantees valid syntax but can't make the arguments correct: a well-formed call to the wrong tool with a wrong URL still passes. One commenter argued that grammar masking only gives you token-level retries rather than retrying the whole sequence, so it can skew output away from what the model would have picked, and suggested just retrying failed calls. They later allowed that retries might not win once you count cost. Ubicloud's post calls constrained decoding "a per-model task", and one of its authors said in the thread that Qwen 3.8 doesn't come with the clear recommendation Moonshot gives for Kimi, and advised checking model quality with it on and off. Measure both.

It also has bugs you should know about:

• #58930: JSON schemas nested inside structural tags skip XGrammar's unsupported-feature check, so a constraint like multipleOf is silently ignored. Still open at the time of writing, with a fix proposed in PR #58940.
• #44006: strict tool calling plus MTP speculative decoding threw "Failed to advance FSM" errors and 500s. Closed by PR #44297 in July 2026, though a later comment on v0.28.0 still saw the FSM error, just without the 500.
• #55152: the guidance backend rejects the new structural tag format with KeyError: 'triggers'. A vLLM contributor replied that the tool-call structural tags were built for xgrammar and expects all of them to fail on guidance.

Ubicloud also notes that gpt-oss-20b, Qwen 3.6 and GLM 5.1 users reported issues back when constrained decoding was on by default. PR #45600 says it wanted to stop structural tags from over-constraining generation when the caller hadn't asked for strict. I think that was the wrong trade: a few loud regressions on some models in exchange for Kimi and DeepSeek deployments silently failing a chunk of their tool calls.

Measure your own deployment with K2VV

Don't take my numbers or Ubicloud's on faith; test your own deployment. Moonshot's K2 Vendor Verifier works against any OpenAI-compatible endpoint. The repo ships a pyproject.toml and uv.lock, and the tool-call sample set (half of the 4,000-request test set, which matches the 2,000 requests Ubicloud ran) lives at https://statics.moonshot.cn/k2vv/tool-calls.tar.gz.

The README's example for Kimi K2.5 served by vLLM or SGLang, with thinking turned off (Instant mode):

python tool_calls_eval.py samples.jsonl \
    --model kimi-k2.5 \
    --base-url YOUR_API_BASE_URL \
    --api-key YOUR_API_KEY \
    --concurrency 5 \
    --extra-body '{"chat_template_kwargs": {"thinking": false}, "temperature": 0.6}'

Swap --model for whatever name your server reports. You get two numbers in summary.json:

• schema_accuracy: successful tool calls divided by responses that finished with tool_calls. Anything under 100% means your server is shipping broken calls.
• tool_call_f1: how closely your deployment matches Moonshot's API on when to call a tool. Moonshot's acceptable bar is above 73% for K2-thinking and 80% for K2-0905.

Run it once with your current flags and once with --tool-strict-level parameter. Use --incremental to rerun only the failed requests. If the schema number doesn't move, your problem is somewhere else, but at least you've ruled out the cheapest fix.

The checklist:

1. Upgrade to vLLM 0.31 or newer.
2. Add --tool-strict-level parameter.
3. Confirm VLLM_ENFORCE_STRICT_TOOL_CALLING isn't set to false.
4. Use the xgrammar backend, not guidance.
5. If you run MTP or other speculative decoding, retest after enabling strictness.
6. Run K2VV before and after, and keep the summary.json files.

So what's your number?

My guess is most teams have never measured their tool-call failure rate. They have a vague feeling the agent is "flaky" and a retry loop that hides it. What's yours in production? Have you actually measured it, and did constrained decoding help or hurt your model? If you think retries beat grammars, tell me why in the comments.

Sources

• Ubicloud: Fixing Tool-Call Accuracy in vLLM
• Hacker News discussion
• MoonshotAI K2 Vendor Verifier
• vLLM tool calling docs
• vLLM v0.31.0 release notes
• vLLM PR #45600, PR #56268, issues #58930, #44006, #55152