DEV Community

Cover image for I watched a 501B model from a billion dollar lab fail 3 times on one tool call. And my startup just got a reason to exist.
Sumama-Jameel
Sumama-Jameel

Posted on

I watched a 501B model from a billion dollar lab fail 3 times on one tool call. And my startup just got a reason to exist.

So I was building my first YouTube video, footage done, VO in progress, and I left Beam running in the background.

Beam. You know Beam right? Reflection AI's new monster. 501B total params, 23B active per token, 1M context, Apache 2.0 license, built from scratch for coding and agents. Lab founded by two DeepMind legends, raised $4.7B, Nvidia backed, valued at $25B pre money. On paper it's a beast. 80.9 on SWE-bench Verified.

And it's running in xhigh reasoning mode. Like, the smart mode.

Task was simple: check if ocr tools exist, run tesseract on an image. What it did:

Thinking: let me check for OCR tools...
run_commands() -> Error: Invalid input
Thinking: let me fix...
run_commands() -> Error: Invalid input
Thinking: let me retry properly with commands array...
run_commands(which tesseract...) -> command fails with code 1
Thinking: let me OCR with tesseract...
run_commands() -> Error: Invalid input again

3 times. Same tool, Same error.

I am not roasting Beam. Beam is genuinely brilliant for its efficiency story. The problem is not Beam. The problem is how we train these models. We show them millions of perfect git diffs, first diff: bug, second diff: bug fixed. We never show them the 5 stupid tool call failures that a real dev does before getting it right. So when they fail, they don't learn to recover, they just loop.

This is literally why I am building MegaBit. My factory records EVERYTHING - The failed bash, model's thinking, tool calls, the retries, etc. No "Synthetic" outputs. Because that failure loop IS the data.

If you are training coding agents, you need trajectory data with honest failures, not just clean final code. I have 59 of them now, all with real terminal outputs, actual work done, and recovery from these kind of situations.

Repo: github.com/Sumama-Jameel/megabit-trajectories

Anyone else seen frontier models loop like this?

Top comments (0)