i spent this week trying to understand where reasoning actually happens in a model.
it started with a post on my timeline. reflection and hugging face were giving $20k to people building with Beam, reflection's new 501B open model. i had never gone this deep into how a model thinks, so i used it as an excuse to learn.
demo: https://huggingface.co/spaces/PengoSword123/beamdial
data: https://huggingface.co/datasets/PengoSword123/beamdial-v0.1
code: https://github.com/CoderRahul01/beamdial
the question
Beam has a setting called reasoning_effort. 5 levels, low to max. higher means the model thinks longer before it answers, and those thinking tokens are billed like any other output.
most developers set it once and forget it. usually on max.
a smart model should think hard on hard problems and answer fast on easy ones. you shouldn't have to guess.
what i measured
-> 410 problems with known answers. school math, competition math, olympiad problems, python coding
-> every problem at all 5 levels. 2,050 runs
-> math checked with hugging face's math-verify, code checked by running the real tests in a sandbox
what i found
on standard math and code every level landed inside the same error bars.
low 96.3%. max 96.6%. low used 55% of the tokens.
the dial does work. on competition math max thinks 2.2x longer than low. it just doesn't buy more right answers on these tasks.
one interesting hint: when low thinks way longer than usual, the problem is usually hard. on the 23 tasks where that happened, high got every one right. but when i picked the threshold properly on held out data, "re-ask at high" only tied "always use high". so it's off by default until i can test it on harder problems.
the bug i found in my own grader
halfway through, accuracy looked like 93.1%. something felt off, so i read the "wrong" answers. most were right. 1/4 written as \frac14, 15 pi with a space.
the math checker can't run inside worker threads, and my grader was quietly counting that as a wrong answer. after the fix: 96.7%.
rule: read your failures before you trust your numbers.
what i built
BeamDial wraps the openai client you already use with Beam.
-> picks the level per prompt with a tiny router (pure python, under 1 MB, no gpu)
-> moves up a level only when your own check fails
-> retries cut-off answers at the same level with more room instead of thinking even longer
-> always sends back the reasoning_content Beam needs on follow-up turns, which otherwise breaks the whole chat
and beamdial calibrate. point it at your own prompts and it tells you if more thinking actually helps for you. my benchmark is a starting point, not the final word.
where this stops
these are standard benchmarks where Beam already scores about 97%, and it has likely seen many of them in training. harder or newer problems might need more thinking. next i'm testing aime 2026, hmmt 2026, apex and freshly generated long-context tasks.
the whole study cost about $4 of credits.
do you set reasoning effort yourself, or leave it on max?
i post the build as it happens on X: x.com/rahulpandey187
Top comments (1)
Two numbers worth putting next to "same error bars". At 96.3% to 96.6% on 410 problems the standard error is about 0.9 points, so each level's 95% interval is roughly +-1.8. What the unpaired comparison can rule out is a gap of around 2.5 points or more; a 1-point gain from max would be invisible. Since every problem ran at all 5 levels, a paired test (McNemar on the problems where low and max disagree) is much sharper, and only the discordant counts matter. Do you have those counts?
The 23-tasks result has the same issue: high getting 23 of 23 has a 95% Wilson lower bound of about 86%, which sits below the 96% baseline, so that cohort can't show high beats low there. It fits your conclusion that the router only tied "always high".
On the grader bug: counting a harness failure as a wrong answer is the sneaky kind because it lowers every level about equally. Logging a separate "grader error" outcome next to correct/wrong would catch it earlier. Did the 93.1% vs 96.7% gap differ by level, or was it flat?