TL;DR — After 1,800+ evaluation trials across 9 system generations, we withdrew from the Gemma 4 competition and are publishing our measurements instead.
- Our best custom system scored 15/114 (13.2%) — one task behind the organizer's simplest zero-think baseline (16/114, 14.0%), and never ahead of it.
- Longer thinking actively hurt the 4B agent: the think=4096 baseline scored 10/114 vs 16/114 for zero-think (−37.5% relative).
- On the unseen holdout, our custom systems (5–6/44) consistently beat the zero-think baseline (4/44) — the machinery helped there, just not enough overall.
Hi everyone — we're Japan Open Data Lab. After 1,800+ evaluation trials across 9 system generations, we've decided to withdraw from this competition, and we'd rather publish our measurements than let them sit in a drawer. Everything below is aggregate results and findings only — no task-level data, no patches, no submission code.
Setup. Gemma 4 E4B (W4A16 quantized, instruction-tuned) as a single autonomous SWE agent in a fully offline, time-limited container (vLLM + Google ADK, 300–355s per task). Our internal sound evaluation set: 114 tasks (a 70-task development set from FastAPI/requests/httpx, plus a 44-task unseen holdout from Textualize/rich, kept fully isolated from tuning and training data).
Score progression (solved out of 114)
- Organizer reference, E2B zero-think: 5 (4.4%)
- Organizer baseline, E4B think=4096: 10 (8.8%)
- Organizer baseline, E4B zero-think: 16 (14.0%)
- Sprint 2B, single LoRA, think=1024: 12 (10.5%)
- Sprint 2B, multi-LoRA serial (Solver+Critic): 13 (11.4%)
- Sprint 3, multi-LoRA serial + guards, 10-run mean: 12.90 (11.3%), range 11–15, SD 1.52
- Sprint 4, pure base (no LoRA), serial + AST checks: 14 (12.3%), holdout 6/44 (13.6%, best on holdout)
- Sprint 6, custom engine, zero-think single-stage: 12 (10.5%); zero-think serial: 13 (11.4%)
- Sprint 7, balanced think 768/1024, serial, pure base: 15 (13.2%) — our best custom system
- Sprint 8, calibrated (think 640/1024, output cap 2560): 14 (12.3%), zero timeouts, fewest empty patches (6), fastest (89.0s avg)
- Sprint 9, 3-stage Arbiter: 13 (11.4%)
Six findings
1. Longer thinking actively hurts a 4B agent. The organizer's think=4096 baseline scored 10/114 vs 16/114 for zero-think (−37.5% relative). A mid-range budget (768–1024) was the sweet spot for our custom systems. But fully disabling thought output (include_thoughts: false) exploded tool-argument JSON errors from 3 to 132 — a small model needs a minimal planning space to emit structured tool calls. And cutting max_output_tokens 4096→2560 doubled file-edit errors (53→121) because edit JSON got truncated mid-argument. Rule we take home: keep thinking budgets small (0–1024), never shrink the output channel.
2. LoRA distillation from a 31B teacher lost to the raw base model. We verified the LoRA/quantization math to near-perfection (logit cosine similarity 0.999986 after dequantization; tensor-parallel sharding exact). It still hurt: standard SFT dropped our 20-task dev subset 4/20 → 3/20 with 16/20 timeouts; adding DPO collapsed it to 1/20. Head-to-head on the same harness, pure base beat multi-LoRA 6.0 vs 4.5 (dev subset) and 14–15 vs 12.9 (full 114). Small trajectory sets (tens of examples) import the teacher's long, cautious exploration style, which a 4B model can't afford within the time budget, and narrow its generalization to unseen codebases.
3. temperature=0.0 is not deterministic for agents. We ran one fixed configuration 10 times over all 114 tasks (1,140 trials, 13.16 hours): solved counts were 14, 14, 15, 11, 13, 14, 11, 12, 11, 14 — mean 12.90, SD 1.52, 95% CI [11.81, 13.99]. Continuous batching changes floating-point summation order, and tiny timing differences flip single tokens, which reroutes the whole multi-turn search tree. Meanwhile Pass@5 reached 26/114 (22.8%), and the union of solved tasks across all our generations reached 24/70 (34.3%) on the dev set: the capability is there, single-shot selection is the bottleneck. Never judge a ±1–2 task delta from a single run.
4. The funnel collapses at verification — and existing tests are a double trap. In ~60% of tasks (67–70/114) the agent reached the correct file, yet only ~13% ended solved. Two structural reasons: (a) 32.9% of dev tasks require simultaneous multi-file edits, which a 4B agent almost never completes; (b) existing tests mislead in both directions. In ~65.7% of dev tasks the pre-existing test suite already passes on the unmodified code, so incomplete patches look "verified" (72–84% of our Critic's READY_TO_SUBMIT: YES verdicts were false positives). In ~34.3%, the task is a spec change: the old tests assert the buggy behavior, so a 100%-correct patch makes the visible suite fail — and any "prefer the patch that passes existing tests" logic actively destroys correct patches. Forcing the agent to write its own failing reproduction first also backfired (6.0 → 3.5 on the dev subset): a 4B model gets stuck writing the repro itself.
5. Textbook multi-agent patterns break on small models in specific ways. (a) Wrapping standard tools in a custom nested tool multiplied argument-schema errors 1 → 29; keeping the model's familiar flat tool surface and putting guards (snapshots, AST syntax checks) invisibly behind the tool layer halved timeouts and cut empty patches ~40%. (b) Prompt-level budget instructions ("finish within 9 tool calls") were ignored in 61.4% of tasks (mean 17.7 calls, max 54); role budgets must be enforced in code at the tool layer. (c) Context compaction plus output discipline got us to zero context-overflow errors and zero timeouts in Sprint 8 — stability is achievable, it just doesn't add solves by itself.
6. The honest summary. On the unseen holdout, our custom systems (5–6/44) consistently beat the organizer zero-think baseline (4/44), and Sprint 8 beat it on speed and stability too. But overall our best (15/114) finished one task behind the simplest organizer baseline (16/114) — within noise, yet never ahead. Every layer of custom machinery (second-stage critics, arbiters, verification gates) sometimes repaired failures and sometimes broke tasks the first stage had already solved, via the test traps in finding 4. For a 4B model, added harness complexity taxes the exact tool-calling distribution the base model is good at.
What a 4B agent can and cannot solve (from the 10-run data)
- Reliably solvable (8–10/10 runs): single-file edits of ~1–12 lines where the issue names the function/exception or includes a repro snippet; typically 5–15 tool calls, 90–160s.
- Borderline (1–4/10): correct file reached, but multiple plausible implementations the visible tests can't distinguish, or exploration that runs into the 250–330s range.
- Effectively unsolvable for this setup (0/10): multi-file coordinated changes; spec-change fixes where old tests assert the buggy behavior (the agent either submits a false-positive early or loops rewriting a correct patch); broad design issues that send the agent wandering through grep results.
Seven principles we take to other projects
- Route by task shape: single-file, small, well-specified maintenance tasks to the small model; multi-file spec changes to bigger models or humans.
- Minimal thinking budget (0 for routine, 512–1024 for state-tracking tasks); never cap output below what tool-call JSON needs.
- Prefer the raw base model + a good harness over small-data LoRA SFT/DPO.
- Keep the tool surface exactly as the model learned it; put protection behind the tools, not in front of them.
- Enforce agent role/call budgets in code, never in prompts.
- Never treat "existing tests pass" as correct or "existing tests fail" as wrong; observe the before/after test delta.
- Evaluate with repeated runs and statistics, not single-run deltas.
Infrastructure notes (outline only)
Three reusable lessons from the plumbing: the E4B's non-uniform attention layout (7 of 42 layers with a larger global head dimension) needed a small compatibility shim for our vLLM/transformers versions; the 262k vocabulary makes naive cross-entropy spike VRAM (~17.5 GiB), which chunked CE loss plus exact W4A16 dequantization avoided; and a dynamic sandbox that resolves per-repo test dependencies from each project's own metadata kept 9 heterogeneous OSS repos reproducible offline.
Thanks to the organizers for a genuinely interesting problem — the zero-think baseline is embarrassingly strong, and that's a finding in itself. Good luck to everyone still in; we'll be watching the leaderboard.
Originally published on Kaggle as a discussion post in the Gemma 4 competition forum: https://www.kaggle.com/competitions/gemma-4-developer-agent/discussion/760014
— Japan Open Data Lab (https://jpopendata.jp)
Top comments (5)
Publishing a withdrawal with 1,800 trials behind it is worth more than most leaderboard entries — the zero-think baseline beating every custom system at 16/114 is the kind of result people usually bury. The temperature=0 finding is the one I'd steal: a 10-run SD of 1.52 means single-run deltas of one or two tasks are noise. Did the holdout edge (5–6/44 vs 4/44) survive the same repeated-run treatment, or is it inside the noise too?
Short answer: it is inside the noise too, and we should probably have said that more plainly. The 10-run treatment (mean 12.90, SD 1.52) was one fixed configuration over the full 114 tasks — we did not break out a repeated-run distribution for the 44-task holdout on its own. The 5–6/44 vs 4/44 edge is only 1–2 tasks, seen across several system generations, which is suggestive but, by our own rule in finding 3, not enough to call it real. Thanks for holding us to the standard we asked readers to apply.
That's the honest answer, and it's the reason the withdrawal reads as credible rather than defensive — holding yourself to the standard you set in finding 3 is the whole point. The 1–2 task edge across generations is suggestive enough to be worth one clean 10-run holdout, if only to close the question properly. Would you rerun the holdout that way, or is the cost of the runs the reason it stays open?
It stays open by decision, not just cost. We have withdrawn from the competition and wound down that setup, so we will not rerun the holdout. If we ever return to small-model agents, a clean 10-run holdout is the first experiment on the list — the question deserves a proper close, just not on a line we have already closed.
Fair — a closed line deserves a clean close, not a half-rerun. The honest part is that the holdout edge is now labelled as open rather than quietly presented as a win, which is exactly the kind of thing most papers skip. If the 10-run holdout ever does happen, that's the result I'd want to read first.