What model we are talking about (for context)
Beam-1.0-501B-A23B by Reflection AI.
It is a 501B total, 23B active per token MoE. Text only, 1M context. It only uses 23B per token so it is cheap to run for its size. That is the pitch.
I have used it for a few days now and for me it is an ass model. Genuinely ASS.
Benchmarks look good on paper
Reflection says it matches GLM-5.2 on reasoning with 3 to 4 times less compute.
Some numbers from their launch table:
Beam vs GLM-5.2 which is also a big model:
- SWE-Bench Pro v1: Beam 65.5 vs GLM-5.2 62.1
- DeepSWE v1.1: Beam 44.4 vs GLM-5.2 44.0
- Terminal Bench v2.1: Beam 80.1 vs GLM-5.2 81.0
- SWE-bench Verified: Beam 80.9
So it looks close to a big competitive model. But against bigger new models it trails:
- Terminal Bench v2.1: Beam 80.1 vs GLM-5.3 88.2 and Kimi K3 88.3
And overall from my experience, this model is benchmark maxxed. On paper it looks like it beats or ties big models. In real use it does not.
In real use cases, it fails at basic tool calls
3 out of 5 tool calls by this model go either in wrong format, or the command itself is wrong. It barely gets a right tool call.
Instruction following is worst. In max reasoning mode it overthinks a lot but still gets things wrong.
It exposed my API keys twice
Context: I was automating my email pipeline. I already told it in prompt that the automation is built and exists in CWD. It should just use it... Right?
Instead it ran this, grep-ing for GOOGLE_EMAIL_KEY, but guess what? That grep-ing was also wrong:
env | sort; echo '---grep---'; grep -R "GOOGLE_EMAIL_KEY" -n /home/sumama/ClientHunt/International /home/sumama/.config 2>/dev/null | head -50
What this command does:
-
env | sort- prints every single env variable on my system, sorted. That includes all my API keys, tokens, secrets. Everything in env gets printed. -
echo '---grep---'- just prints a line to separate output. -
grep -R "GOOGLE_EMAIL_KEY" -n /home/sumama/ClientHunt/International /home/sumama/.config- searches for the textGOOGLE_EMAIL_KEYin every file inside those two folders, recursively.-nshows line numbers. -
2>/dev/null- hides errors so you do not see permission errors. -
| head -50- shows only first 50 matches.
So in one tool call it dumped all env keys and then scanned my project folder and my config folder for keys. I told it not to do this. It did it twice.
The main catch - Eureka Machine
Reflection built this model with their automated research engine called Eureka Machine.
In easy words, it is a system that runs the research loop on its own. It proposes an idea, implements it, tests it and uses what it learns to pick the next experiment. It tries to automate the whole model training process.
They tried to automate everything. But not everything can be scored without human judgement. Tool calling, following instructions, not leaking secrets - you need human eyes for that.
The overall catch is this: researchers and engineers are still required. AI has not replaced them yet. And this model just proved it.
I already posted about this on Oct 25, that this model is worst in tool calling and how my startup fixes it. Here is proof with screenshot:
If you are using Beam for any real work that needs tools or agentic execution, be careful. Check every command it makes before you let it run.
What has been your experience with this ASS model?
Top comments (0)