My baseline agent found a bug in my own ground truth

I implemented Amazon's "Tool-Making and Self-Evolving LLM Agents" paper (arXiv 2607.08010) on real Kraken/Binance market data. The tools it generated hit 25/25 accuracy at ~9,600× lower latency than the baseline agent — but the most interesting thing that happened was my baseline agent proving one of my hand-made labels wrong.

---

The problem the paper attacks

If you've run LLM agents in production, you've seen this: the agent solves the same procedural step over and over - and re-derives the solution from scratch every single time. Same instruction re-interpreted, same API schema re-discovered, similar code re-written. Every repetition costs seconds of latency, thousands of tokens, and a fresh chance to get it wrong.

Amazon's paper (from their Fulfillment Technologies & Robotics group) proposes something simple and radical: let the agent compile its own reasoning into permanent tools. Solve the step with an LLM once, capture the working code, freeze it into a validated deterministic function, and never pay the LLM tax for that step again - falling back to the LLM only when a tool is missing or can't answer.

The paper's setting is Amazon's internal robotics-monitoring stack, which nobody outside can reproduce. So I transplanted it to a domain where everything is real and public: cryptocurrency market monitoring - live Kraken/Binance REST endpoints, a 312MB archive of historical trades, and alarms like "is this market stale?" and "is this volume spike real?"

One rule I set myself: no mocks anywhere. Real exchange APIs with their real quirks, real historical data, and human-labeled ground truth. (The quirks earn their keep immediately: ask Kraken for XBTUSD and it answers under the key XXBTZUSD. No SOP would ever tell an agent that. This is exactly why the paper's "trace" concept exists.)

What I built

Four days, five components:

1. SOP decision tree (sop.yaml) - 9 decision nodes across 3 alarm families (spread anomaly, volume spike, feed health), the kind of runbook a human on-call engineer would follow.
2. Main agent - walks the tree per alarm. The key design decision: the node evaluator is injected (evaluate(node, pair, t) → true | false | no_data), so the same walker runs against human labels, an LLM, or compiled tools.
3. Baseline sub-agent - the expensive thing we're trying to beat. For each node, an LLM writes Python that queries the data layer; I execute it in a subprocess, feed back stdout/stderr, and iterate (max 3 attempts) until it commits to a verdict.
4. The pipeline - the paper's contribution: a tool-maker LLM that reads the node text + my labeled cases + the baseline's successful traces and writes a permanent tool; and a reflector that tests every tool against the full labeled set and repairs on failure (max 3 rounds).
5. Runtime - tool-first evaluation with LLM fallback.

Ground truth: 25 cases I labeled by hand from the historical archives - real staleness gaps, real volume spikes, real calm hours — each with a one-line why stating my reasoning.

Results

System	Accuracy	Hold-out	Median latency / node	Tokens / node
Baseline (code-writing agent)	22/25	—	16,717 ms	~7,000
Compiled tools	25/25	5/5	1.74 ms	0

The tool-maker one-shotted the entire SOP: all six tools passed every labeled case at version 1 - the repair loop never fired. The hold-out set was labeled after tool generation, and 4 of its 5 cases use trading pairs (ETH, SOL) the tool-maker never saw. The tools generalized because they encoded my conventions, not my examples.

That last row is the whole paper in one line: the baseline pays ~17 seconds and ~7,000 tokens to answer a question a compiled tool answers in under 2 milliseconds for free.

The three findings I didn't expect

1. My baseline agent found a bug in my ground truth. One of my labeled cases pointed at BTC's only >15-minute trading gap of the entire quarter — I'd placed the case timestamp mid-gap and labeled it "stale = true." The baseline agent returned false, and it was right: at that timestamp, the last trade was only ~10.7 minutes old, inside the 15-minute lookback. My label reasoned about the gap as a whole; the agent reasoned about the moment. Labels are a spec, and specs have bugs — I moved the timestamp late into the gap and thanked the agent for the code review.

2. Ambiguity - not capability - dominated the baseline's errors. The baseline's misses weren't "the LLM is dumb." They were cases where my SOP said things like "significant price move" or "recent typical volume" without defining them - and a fresh LLM guessed conventions different from mine (a 30-day baseline where I meant 24 hours; the previous candle where I meant the current one). Every fresh call re-guesses your conventions. That's the deep argument for compiling tools: they freeze the SME's conventions once, correctly.

3. Four labeled examples pinned an ambiguous convention. My "significant move" labels were extreme: 0.2% moves labeled not significant, +20% moves labeled significant. From just four such cases the tool-maker committed to a concrete |move| ≥ 5% threshold - a sensible interpolation of my intent that I never stated anywhere. (The paper's Appendix B.4 claims ~5 labels get you near ceiling; I can now confirm the vibe.) The boundary between the extremes stays genuinely undefined: a hold-out candidate at −1.6% with 5× volume showed up and I couldn't honestly label it either way — so it's documented as exactly where the ambiguity lives.

What I didn't reproduce, honestly

• Training-set caveat: the 25 cases both taught and tested the tools; the independent evidence is the small post-hoc hold-out. A larger temporal split is future work.
• The paper's LoRA fine-tuning section needs Amazon's production trajectories — can't reproduce outside.
• Tool latency is file-scan-bound on the biggest archive (one tool scans 5.75M rows of BTC trades; 8–22s cold). An index would fix it; irrelevant to the architecture claim.
• The label-free variant (Appendix C - validation by voting instead of human labels) is my Part 2.

Takeaway

The paper's claim survives contact with a completely different domain, real messy data, and a solo implementer: an LLM agent that compiles its own reasoning into validated tools keeps the flexibility of agents and gains the speed, cost, and determinism of code. And the artifacts along the way — the label my agent falsified, the convention four examples pinned — taught me more about specification than about LLMs.

Code, data pipeline, labeled cases, charts, and the full evaluation: github.com/0xTaneja/tool-making-llm-agents

I'm implementing more agent papers on real market data - if you're working on agent systems or know an unpopular paper that deserves a real implementation, my GitHub inbox is open.