This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
What I Built
My AI agent kept calling tools it didn't need.
I gave gpt-oss-120b 161 tasks it can reliably answer on its own, no tools. It still reached for a tool on 69 of them. A small 4B model running locally did the same on 36 of 68. Every one of those is an extra round trip, more tokens and more waiting.
This week's theme is making AI do less. So I tried to make my agents lazier, in the good way.
Whetherman (as in weather*man*, it forecasts whether you need a tool) is a small proxy that sits between your agent and its model. Before every model call that offers tools, a fine-tuned FunctionGemma-270M makes one of four calls:
| Decision | What the proxy does |
|---|---|
| ACT | Pass the request through untouched. This is the default. |
| ANSWER | Strip the tools and tell the model to answer from what it already has. |
| ASK | Strip the tools and ask the user one clarifying question. |
| DECLINE | Strip the tools and explain what it can't do. |
It only steps in when it's confident enough for that specific agent. If the router errors out, returns something invalid or takes longer than 3 seconds, the request goes through unchanged. A confident but wrong decision is different: the tools still get stripped, and that's what cost the big agent tasks in the results below.
To be upfront: it didn't make any of my three agents better. It hurt the big one and was basically neutral on the other two. What I did learn is that whether a tool call is "necessary" depends on which agent is asking.
A few results that stood out:
- Give the small agent the big agent's settings and it fails 32 more tasks out of 150 than with no gate at all. The router and the requests didn't change, only the profile did.
- Removing tools isn't free. gpt-oss wrote about 60% more output tokens when I took its tools away, and it tried to call the removed tools 20 times anyway.
- Where the agent is actually capable (math, code, primes), the gate cut calls from 18 to 6 with zero lost tasks. Everywhere else, a wrong skip cost way more than the cheap call it saved.
Demo
The video shows the four decisions, the same math question routed differently for two agents, one call saved, and one false skip. It's all replayed from the recorded held-out runs, nothing is re-run.
The demo page in the repo (demo/index.html) replays every recorded decision. Red rows are episodes where a skip most likely caused the failure.
Code
vinmay
/
whetherman
A 270M model that forecasts whether your AI agent actually needs to call a tool.
Whetherman
A 270M model that forecasts whether your AI agent actually needs to call a tool.
Agents call tools they don't need. On When2Tool training tasks that a large hosted model (gpt-oss-120b) reliably answers without tools, it still called a tool on 69 of 161; a small local model (Qwen3.5-4B) did so on 36 of 68 Each unneeded call costs a model round trip, tokens and wall time.
Whetherman is a small OpenAI-compatible proxy that sits between an agent and its model. Before each model call that offers tools, a fine-tuned FunctionGemma-270M forecasts one of four decisions:
| Decision | What the proxy does |
|---|---|
| ACT | Forward the request unchanged. This is the default. |
| ANSWER | Remove the tools and ask the model to answer from the conversation. |
| ASK | Remove the tools and ask the model for one clarifying question. |
| DECLINE | Remove the tools and ask the model to explain what it cannot do. |
…
The trained adapter and all three agent profiles are on Hugging Face: vinmaynair/whetherman-functiongemma-270m. It's a research artifact. Read the results before you put it in front of anything real.
git clone https://github.com/vinmay/whetherman && cd whetherman
uv sync --extra ml
# needs access to google/functiongemma-270m-it on Hugging Face (HF_TOKEN in .env)
uv run hf download vinmaynair/whetherman-functiongemma-270m --local-dir checkpoints/shared-v3
uv run whetherman calibrate --agent-endpoint http://localhost:11434/v1 --model qwen3.5:4b --name my-agent
uv run whetherman serve --profile my-agent --upstream http://localhost:11434/v1 --port 8088
# point your agent's base_url at http://127.0.0.1:8088/v1
How I Built It
"Necessary" depends on the agent
Compute exactly and return only number: (372*79)-115+13
gpt-oss-120b gets that right in its head. Qwen3.5-4B doesn't. So for the big model a calculator call is waste, and for the small one it's the difference between right and wrong. One global "does this need a tool?" classifier can't serve both.
Here's that same task in my labeled When2Tool data, measured on each agent:
flowchart TD
Q["(372*79)-115+13<br/>calculator available"]
Q --> L1["Agent L: gpt-oss-120b<br/>without tools: 3/3 right"]
Q --> S1["Agent S: Qwen3.5-4B<br/>without tools: 0/3 right<br/>with the calculator: 3/3"]
L1 --> L2["L's profile: ANSWER (0.94)<br/>skip the calculator"]
S1 --> S2["S's profile: ACT (0.73)<br/>keep the calculator"]Same question, opposite decisions, and both are correct for that agent.
So Whetherman has two parts:
flowchart TD
A[Agent request<br/>messages + tools] --> P{Proxy}
P -->|no tools, streaming,<br/>forced tool_choice| U[Upstream model]
P --> B[FunctionGemma-270M<br/>shared backbone + LoRA]
B --> R[Per-agent profile<br/>logistic layer + Ï„]
R -->|ACT or not confident| U
R -->|ANSWER / ASK / DECLINE| S[Strip tools,<br/>add instruction] --> UThe shared backbone is FunctionGemma-270M with LoRA and a 4-class head. It's trained on NVIDIA's When2Call, multi-step decision points I generated, and When2Tool tasks labeled on L and S. Where the two agreed, that's the label. Where they disagreed or a label was ambiguous (70 tasks), the target is the average of how often each one failed without tools. It never sees which agent it's routing for. Training took about 50 minutes on my GTX 1660 Ti, in FP32, because that card has no BF16.
The per-agent profile is fitted to one agent. whetherman calibrate runs a probe pack against your agent: every task 3 times with tools and 3 times without, 5 if the results are mixed. The agent's own results decide the label. If it always succeeds without tools, the label is ANSWER. If it fails without tools but succeeds with them, ACT. Anything in between is ambiguous and gets dropped. Then a small regularized logistic layer and a threshold Ï„ are fitted on that, picking the smallest Ï„ that keeps false skips at or under 5% on a separate split.
The probe pack ships with real tools (a fixture repo to read and test, a calculator) and simulators for anything with side effects. If an agent says "email sent" without calling the tool, that's a fail.
Rules I set before the held-out runs
freeze-v3).
Why Does Open Innovation Matter?
This project doesn't work at all without open models, on both sides of the proxy.
The gate runs before every model call. If it costs money per call or adds a network hop, it can't save more than it spends. It has to be small, local and free, and FunctionGemma at 270M runs on a GPU from 2019.
Calibration only works because I can measure the agent. The whole method is running the same model with and without tools until I know what it can do alone. With open models I pinned the exact weights, quantization and serving stack. With a closed API the model can change under you between your probe run and next Tuesday, and your profile silently goes stale.
And I needed to get inside the router. I trained a LoRA, bolted on a classification head, read its hidden state for the per-agent layer and exported the whole thing to Hugging Face. None of that is possible through a chat endpoint.
A closed frontier model would probably over-call less to begin with, which would leave a gate like this less to do. But you'd still have no way to calibrate anything against it, and you'd be paying per request to find out.
The Lazy Payoff
End to end: no gate vs Whetherman
80 held-out tasks (half of them multi-step) × 2 repeats per agent. Success excludes the 10 guessable episodes. Brackets are Wilson 95% CIs.
| Agent | No gate | Whetherman | Tool calls / episode | Tokens / episode | Cost (whole suite) |
|---|---|---|---|---|---|
| L: gpt-oss-120b (DigitalOcean) | 120/150 [0.73, 0.86] | 105/150 [0.62, 0.77] | 2.36 → 2.15 | 1906 → 1897 | $0.0522 → $0.0651 |
| S: Qwen3.5-4B (local Ollama) | 130/150 [0.80, 0.91] | 128/150 [0.79, 0.90] | 1.92 → 1.91 | 2297 → 2156 | local |
| U: gpt-oss-20b (never seen in training) | 115/150 [0.69, 0.83] | 114/150 [0.69, 0.82] | 2.09 → 1.93 | 2062 → 1824 | $0.0461 → $0.0438 |
So: worse on the big agent, neutral on the small one, and on the unseen agent roughly neutral on success with 8% fewer calls, 12% fewer tokens and 5% lower cost. That last row is the closest thing to a win, and it's within noise on success.
I also ran the other gates through the same suite:
| Gate (successes of 150) | On L (120 without a gate) | On S (130 without a gate) |
|---|---|---|
| Prompted Qwen3.5-4B router | 108 | 96 |
| Rules-only control | 90 | 60 |
| Shared backbone, no profile | 99 | 84 |
| The other agent's profile | 113 | 98 |
| Whetherman, own profile | 105 | 128 |
The big agent copes with aggressive gating because it can answer most things itself. The small agent does badly under the same gates because it actually needs its tools. Whetherman's profiles picked that up: on S it barely intervenes (34 skips out of 454 steps) and it's safe.
On L, the prompted Qwen router was actually a better trade than Whetherman. It cut cost 40% for 12 lost tasks. Whetherman lost 15 and cost more. It was also slower: a 4.3 s median per episode against 3.7 s with no gate.
Offline calibration results
False skips on held-out When2Tool tasks, labeled by measuring each agent:
| Shared, no profile | Own profile | Other agent's profile | |
|---|---|---|---|
| S | 65/70 | 4/70 | 42/70 |
| L | 13/17 | 1/17 | 3/17 |
For S, without a profile the router withheld a needed tool 65 times out of 70. With its own profile it was 4, using the exact same backbone weights.
What went wrong on the big agent
It stopped one lookup too early. 11 of the 15 lost tasks have replay evidence of a false skip, and 7 of those are multi-hop retrieval: look something up, then look up the next thing. The gate answered after the first lookup. That makes sense in hindsight: every task in my probe pack is a first-step decision, so L's profile never once saw "one lookup done, another one needed".
Fewer calls also didn't mean lower cost. With its tools removed, gpt-oss wrote longer answers: completion tokens went from 36k to 58k over the suite, and output costs 7× input. It also called tools that weren't there anymore 20 times. Each time, the proxy had to re-send the original request, so those steps saved nothing and cost an extra call.
And the calls I was skipping were cheap to begin with. A calculator or a fixture file read costs almost nothing. One wrong skip can fail a whole task.
This isn't specific to my router. Roughly:
payoff ≈ (cost of a call × calls avoided) − (cost of a failed task × false skips) − (what removing tools does to the model)
With cheap tools, that comes out negative. A gate like this only makes sense when the calls it skips are slow, expensive or risky: paid APIs, rate-limited search, anything with side effects. My benchmark had none of those. That's the next experiment.
It did work on calculator, code, matrix and prime tasks on L: 18 calls down to 6, with no lost tasks. When the agent really can do it alone, telling it to just answer works.
What this shows
There are two results here, and they point in different directions.
As a product, Whetherman didn't work. It made the big agent worse and the other two no better, and on L a prompted 4B router was the better trade.
The research result held up better. A single "does this need a tool?" classifier that ignores which agent it's serving was badly mismatched to agents with different abilities. The same backbone with no profile withheld a needed tool from S 65 times out of 70. With S's own measured profile it was 4. Most of that drop comes from S's own threshold: the backbone with only S's Ï„ got 3/70. On L the learned profile layer added more, 86 correct skips of 129 against 53 for the threshold alone. Giving S the big agent's profile cost it 32 tasks end to end. Tool necessity behaved like a property of the agent and the task together, not the task alone.
That's three agents on one family of benchmark tasks, so it's something to test on more agents and more expensive tools, not a general law. But if you're building a tool router, I'd measure your own agent before trusting a universal label.
More detail
I also ran the backbone (no profile, agent-agnostic) through official evaluation code. Self-reported, not leaderboard entries. *Over the three classes When2Call actually labels, it's 0.629. The router guessed "direct answer" 6 times, and that class never appears in the gold labels, so it adds a zero to the official average. BFCL lines up with everything else: Whetherman is good at "don't call anything here" and too willing to skip when a call really is needed. And a warning about my own numbers: validation macro-F1 was 0.925, but on a balanced 900-item sample of the When2Call test (same three classes) it's 0.642, so validation overstated it.Benchmarks, self-run
Benchmark
Whetherman
Prompted Qwen3.5-4B
When2Call official MCQ test (3,652 items)
accuracy 0.628, macro-F1 0.472*, tool hallucination rate 0.070
BFCL v4 irrelevance (no function fits)
187/240 (0.78)
95/240 (0.40)
BFCL v4 live irrelevance
764/884 (0.86)
533/884 (0.60)
BFCL v4 live relevance (should call)
6/16 (0.38)
10/16 (0.62)
I also tested Laya, an encoder model built for typed decisions, as a second router. Same data, same splits, same calibration, offline only. FunctionGemma won on almost everything: macro-F1 on the same 900 When2Call test items was 0.642 vs 0.497, and on L, Laya couldn't find any safe threshold at all, so it never skipped. Laya was faster on short inputs on the GPU (36 ms vs 55 ms) but twice as slow on long ones (510 ms vs 249 ms), and it has to truncate anything over 512 tokens.Laya vs FunctionGemma
On a GPU it's cheap. On CPU, with 8 tools, it adds about a second to every model call, which eats most of what it could save. Calibration: $0.24 and 83 minutes of agent time for L, $0.35 for U, 19 minutes locally for S, about 120 usable labels each. At U's measured savings it pays for itself after roughly 24,000 episodes. L's never does.What running it costs
GTX 1660 Ti p50
CPU p50
Short request, 2 tools
55 ms
188 ms
8 tools
122 ms
1075 ms
Long history (1.5k tokens)
249 ms
2152 ms
Prize Categories
- Gemma: FunctionGemma-270M is the router, so every decision Whetherman makes comes from it. I fine-tuned it on my 6 GB GTX 1660 Ti: LoRA rank 16 on every attention and MLP projection, plus a 4-class head on the last token's hidden state, trained in FP32 because Turing cards have no BF16. Training took about 50 minutes. The per-agent profiles sit on that same hidden state as a small logistic layer each, so one 270M model serves every agent and only the profile changes between them. On a balanced 900-item sample of the When2Call test it goes from macro-F1 0.519 untuned to 0.642, ahead of a prompted Qwen3.5-4B router (0.281) and Laya (0.497). It runs at 55 ms per decision in about 1.2 GB of VRAM on the same card. The adapter, the head and all three profiles are on Hugging Face with a model card.
-
DigitalOcean: DigitalOcean's serverless inference ran both hosted agents, so the agent side of every measurement happened there. Agent L is gpt-oss-120b and the unseen agent U is gpt-oss-20b, both called through the OpenAI-compatible endpoint with six parallel workers. That covered labeling L (every probe task 3 to 5 times with tools and 3 to 5 times without), calibrating U from scratch as a brand-new agent, and the dev and frozen held-out A/B runs: 20 logged jobs covering 952 tasks. The client retries rate limits and 5xx errors with jittered backoff, prices every response from its token usage at DigitalOcean's published rates and appends it to
eval/results/spend.json, and a hard spend cap stops any job before it goes over. Every cost number in this post comes from that log.
Limitations
- Three agents, measured on one serving stack each, on one day, at temperature 0. A different quantization or provider is a different agent. Recalibrate.
- Small denominators. About 120 usable probe labels per agent, and only tens of truly-needed calls in some held-out sets. The intervals are wide and they're in every table.
- The proxy handles non-streaming OpenAI-compatible chat requests with
tool_choiceauto. Everything else is passed through untouched. - It's not a security guardrail. It decides whether a call is useful, not whether it's allowed, and it fails open on purpose.
What I'd do next
- Multi-step probes, so the profile learns later-step decisions, especially "one lookup isn't enough".
- Test with tools that are actually expensive: paid APIs and slow web search. That's where the payoff math could flip.
- Factor the output-token shift into the skip decision, not just call counts.
- Detect per model when it ignores removed tools, and back off.
If you've tried something like this, or you think it's the wrong approach, I'd like to hear about it in the comments.
Thanks to NVIDIA for When2Call, the When2Tool authors, Google for FunctionGemma, convaiinnovations for Laya, and the Berkeley Gorilla team for BFCL. Code is Apache-2.0; the adapter follows the Gemma Terms of Use.
Top comments (0)