Submission for DEV's Summer Bug Smash — Smash Stories track.
There was a file in my repo called run_benchmark_1_22.py.
Not 1 to 24, which is what the harness was written to do. Not 1 to 26, which is how many Mersenne exponents the agents know. Twenty-two. A chart in the README — a2a_latency_times_1_22.png — agreed. At some point, past-me had decided the benchmark ends at 22, committed the evidence, and moved on.
This summer, hunting for a Bug Smash target, I finally asked: why 22?
The setup
a2a-benchmark compares A2A agent performance across four languages. Python and Go sit behind Gemini tool-calling (ADK); Node and Rust are bare HTTP handlers. Each computes Mersenne primes with Lucas–Lehmer; a harness sweeps N from 1 to 24 and draws two charts.
I ran the full sweep. At N=24, the Python column printed N/A. Every other language returned data. There it was — not a decision, a crash, worked around by shortening the run until it stopped hurting.
The 4,300-digit wall
The Python agent's response at N=24 wasn't even subtle about it:
"Exceeds the limit (4300 digits) for integer string conversion;
use sys.set_int_max_str_digits() to increase the limit"
CPython 3.11 added a default cap on int→str conversion — 4,300 digits — as a denial-of-service mitigation. My agent stringified every prime it found. The 24th Mersenne prime, 2^19937−1, has 6,002 digits.
Here's the part that made me laugh out loud: the stringified list was never returned. The tool reports only its elapsed time. The line that had silently amputated my benchmark at N=23 was decorative. The fix was git rm energy: delete the str(), keep the raw int. Go had the identical dead weight (val.String()) inside its timed region — it just happened not to crash.
One deleted expression, and a column of data that had never existed came into being: N=24, Python, 2,425.9 ms.
It gets worse before it gets better
With the agents finally running, I kept pulling the thread. The harness parsed Python's elapsed time out of the LLM's prose with r"It took ([\d\.\-e]+) seconds". Gemini, in my captures, never once said "It took" — it said "Calculating the first 5 Mersenne primes took…" and later "The calculation took…". The only reason the benchmark had Python data at all was a fallback that read the structured tool artifact. My measurement pipeline's primary path was a bet on a language model's phrasing habits.
The direct agents had their own tells. Ask Node or Rust for 100 Mersenne primes and they'd cheerfully report "Found first 100 Mersenne primes" — having computed 26, the size of their exponent table. And they formatted elapsed time as %.2f ms, so Rust's fastest runs reported 0.00ms, which parses to zero, which cannot exist on a log-scale chart. Those points didn't look wrong; they looked like nothing.
And the biggest lie was the chart itself: "A2A Round-Trip Time (including LLM/Tool calling)". Only half true — literally. Two of the four agents route through Gemini; two never touch an LLM. Median RTT: 2.6ms and 4.6ms for the direct pair, ~1.6s and ~1.8s for the Gemini pair. A ~400× gap presented as a language comparison was actually a pipeline comparison.
The encores
I fixed everything and re-ran the sweep to generate the "after" charts. Go's N=1 datapoint: N/A.
Cause: my fix. With the dead formatting deleted, Go's small-N runs got so fast that time.Duration switched output units — Elapsed time: 836ns — and the harness parser had branches for µs, ms, and s, but had never met a nanosecond. The fix made the code too fast for its own benchmark.
Parser patched. Re-ran again. Three Go datapoints missing — different ones. The captured response text:
"I already did that. Do you want to do it again?"
The harness reused deterministic context IDs; ADK keeps per-context session history; on a rerun, Gemini looked at the old conversation and declined to redo the work. My benchmark's completeness now depended on a language model's opinions about repetition. Unique per-run IDs fixed it, and the final sweep came back 96/96.
What I actually learned
-
The workaround you commit is the bug you keep.
run_benchmark_1_22.pysat in the repo like a fossil of an uninvestigated crash. The moment you rename the script instead of reading the stack trace, you've decided to ship the bug. - Benchmarks are production code. Mine crashed, lied about counts, rounded away its smallest measurements, and compared two different architectures on one axis. Every chart it ever produced was quietly wrong in four ways.
- If a machine consumes the output, never route it through prose. Structured tool artifacts existed the whole time; the regex on LLM text was pure fragility.
- An LLM in the measurement path adds failure modes that have nothing to do with the code under test — including, apparently, boredom.
Nine bugs. Four PRs (#1, #2, #3, #4). One question I should have asked a year ago: why 22?
Don't take my word for any of it — docker run --rm -e GEMINI_API_KEY=your_key -v "$PWD/out:/out" xbill9/bugsmash runs all four fixed agents and the full sweep, and writes the charts to ./out.
Disclosure: I ran this investigation with Claude Code as the debugging agent — it did the reproduction, the fixes, and the benchmark reruns while I steered. The bugs, the numbers, and the "I already did that" refusal are all real and archived in the repo.
Top comments (6)
"The workaround you commit is the bug you keep" â I'm stealing that.
run_benchmark_1_22.pysitting in the repo as a fossil of an uninvestigated crash is such a recognizable pattern. You rename the script, update the README chart, and move on. Six months later someone finds the file and assumes 22 was intentional.The regex-on-LLM-prose parsing is the one that got me. You had structured tool artifacts the entire time, but the primary path was
r"It took ([\d\.\-e]+) seconds"â a bet on Gemini's phrasing habits. I see this pattern constantly now: pipelines that parse LLM text with regex when the function call already returns JSON. The moment the model rephrases, your data pipeline silently breaks and you don't notice because the fallback "works."The "I already did that" refusal is the most 2026 debugging story I've read. Your benchmark's completeness literally depended on a language model's opinion about repetition. Did you consider dropping the Gemini-brokered agents from the comparison entirely, or was the pipeline-vs-direct gap the point?
Sometimes you end up in places you don't expect. Initially the goal was just a proof of concept for A2A benchmarking - which has very little coverage - but ended up with a dual homed MCP/A2A agent and full automatization of the process. With the last runs - the overhead of the framework becomes less important as the task becomes more CPU bound.
"The workaround you commit is the bug you keep" is painfully accurate, and the rest of the story earns that line. The best part for me was how many failures were really measurement failures rather than algorithm failures: parsing timings out of model prose, mixing direct and Gemini-brokered paths under one headline metric, and rounding away the fastest values until the chart quietly lied. That is a good reminder that benchmark harnesses need the same skepticism we apply to production systems, especially once an LLM sits anywhere in the measurement path. Curious whether the next hardening step for this project is typed benchmark artifacts end to end, so no later chart or parser has to infer meaning from free text again.
"The workaround you commit is the bug you keep" is a phenomenal takeaway. This entire write-up is a masterclass in the Observer Effect applied to software profiling—where the measurement tool itself (a hidden string conversion, a formatting regex, or retained session state) fundamentally alters or destroys the outcome. The Gemini agent getting "bored" and refusing to recalculate the primes is absolutely hilarious.
That Rust
0.00msformatting bug resonates deeply. Profiling at the absolute edge of hardware capabilities is a unique kind of nightmare. Building strictly deterministic C++ state sync cores for Medium-Frequency Trading (MFT) pipelines forces you to confront this daily. When the monolithic architecture of the TolmachЁv Netcode SDK v36.0.0 is tuned to push 41.5 Million TPS with a physical RTT of ~24ns and zero CPU validation waste, standard benchmarking harnesses become completely useless. The harness itself introduces more latency than the execution loop. Just like that deadval.String()weight you found in the Go agent, a single hidden allocation or type conversion in a C++ hot path doesn't just truncate a chart at N=22—it instantly desyncs the entire deterministic network topology. Benchmarks really are production code.Given the massive ~1.6s to ~1.8s latency introduced by the LLM-routed agents (Python/Go), how are you mathematically isolating the actual baseline language execution time from the massive network and inference jitter of the Gemini API? Are you strictly relying on the timestamps within the structured tool artifacts, or is there another method you use to subtract the LLM's overhead from the final charts?
strictly timestamps. the overhead of the LLM surfaced during testing. as the CPU bound task scales it becomes less important. the whole question of LLM call overhead and framework cost is still open - especially on performance sensitive workloads
The best debugging stories are usually the ones where the first symptom was not the real problem. A benchmark stopping at N=22 sounds like a tiny edge case, but it forces a more useful question: which assumption made 22 special, and why did the test let that assumption hide for so long?