Qwen's reported math accuracy rose after GPT-6 Astra edited a small part of its weights. The repair addressed a concrete failure: repeated reasoning that exhausted the space available for an answer. For developers evaluating weight-space programming, the useful questions concern the failure being repaired, the assistance supplied to the editor and the tests used to judge the result.
TL;DR
- An MIT-affiliated team reports Qwen3.5-0.8B rising from 7.9% to 48.1% on MATH-500 under its fixed protocol. About a third of edited answers still reached the length cap.
- Astra worked across many turns with Python, practice problems with answers and batched model-run feedback. Gradient training was forbidden.
- The researchers say the final evaluation questions stayed unseen by the editor. The benchmark names and evaluation protocol were disclosed in the prompt.
- Today's critique from Matthew Honnibal concerns the impression created by combining experiments with different conditions. The primary's smaller Gemma follow-up failed to establish a gain.
Qwen weight editing repaired a failure to finish
Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab published their study on November 2. The page lists MIT CSAIL affiliations. OpenAI's GPT-6 Astra was the editor used in the experiments. This attribution matters: the research account belongs to the team, and the model provider supplies the editing model.
Their Qwen experiment used the text part of a small language model with about 0.8 billion weights. In thinking mode, the base model often repeated its work until it reached the output limit. On the reported MATH-500 evaluation, 93% of base answers hit that limit. A model can perform useful intermediate work and still receive a failed grade when a final answer never arrives.
The authors describe a repetition-associated direction written by the feed-forward components of layers 15 and 19. Astra subtracted that contribution from selected output columns. The resulting model completed more answers, and the reported accuracy increased. The researchers explain the achievement as allowing the model to finish answers it could already work out.
The repair remains incomplete. Edited Qwen hit the cap in 33.1% of MATH-500 answers. That residual matters when judging a deployment: an intervention that rescues many requests can still leave a sizable queue of requests consuming their whole budget. The title's word “patched” reflects that limitation.
The original Qwen model card already warns about thinking loops. It recommends tuning sampling for the use case and using streaming to detect and interrupt anomalous generation. A weight edit is one demonstrated intervention against a known obstacle. It should be evaluated alongside the controls available in the serving application.
What Python and practice feedback contributed
The actual initial prompt supplies a substantial research environment. Astra receives a persistent CPU Python kernel over the weights, a tokenizer and libraries for numerical analysis. The kernel has no internet or GPU. Model execution happens through a separate forward-pass tool.
Practice files contain problems, answers and, for the GSM8K and MATH training splits, reference solutions. The editor can inspect generated texts, grades, failure counts and internal activations. It can compare original and edited versions on the same problems. Those observations provide evidence for choosing a component and testing an intervention.
The forward-pass budget needs careful reading. A generation request can contain up to 512 answers in total. Paired comparisons can generate up to 256 answers per model. A teacher-forced request can inspect up to 1,024 texts within its token constraints. Fifty budget units therefore describe batches of useful feedback. They are separate from the count of Python runs, conversation turns or training steps.
The pinned logs show that the successful Qwen run was a multi-turn process. The editor inspected failures, tried interventions and compared outcomes. The narration keeps the exact elapsed time out of the story because that wall-clock total includes a provider wait. Calling the process instant would erase the work visible in those logs.
The prompt also provides a plan. It asks Astra to diagnose the largest explainable failure category, localize components causally, test several strengths and confirm gains on problems unused for selection. It explicitly names repetitive-thinking repair as an eligible target. That guidance belongs in an account of the capability: the model produced a useful repair within a purpose-built task and testing environment.
Direct weight edits, gradient training and held-out evaluation
The experiment forbids backward passes, optimizers and iterative fitting of weights to a loss. It also restricts which tensors may be changed. The editor can analyze existing weights and feedback, then compute and save a direct intervention. “No gradient training” describes those operation restrictions precisely. Practice examples and evaluation feedback remain part of the process.
The reported final edit touches 131,072 entries, approximately 0.016% of the model's weights. The pinned experiment README notes that 130,188 differ after conversion to the evaluated bfloat16 format. This rounding detail explains why touched entries and distinct final values can have slightly different counts.
The researchers declare that final evaluation questions were withheld. Practice feedback helps select the edit; final benchmarks measure the submitted model afterwards. Astra knows the benchmark names and protocol from its prompt. A practice confirmation split is another category again: it can help check a candidate during editing while remaining separate from the final benchmark.
The final Qwen comparison keeps the decoding protocol fixed. Thinking is enabled, the answer budget is 32,768 new tokens and MATH-500 averages four samples per problem. The reported decimal accuracy is an average across those generations. The table is useful within that protocol; it does not describe every possible sampling configuration or each user's workload.
| Reported Qwen condition | MATH-500 accuracy | Answers hitting the cap |
|---|---|---|
| Original | 7.9% | 93.0% |
| Edited, 50-pass budget | 48.1% | 33.1% |
The gain also reached instruction following in the authors' IFEval test. Finishing the output can help beyond math. This is a practical reminder to examine completion behavior alongside correctness whenever a benchmark score moves sharply.
Why the tiny game experiments have their own scope
Matthew Honnibal's November 3 critique argues that readers can combine the conditions of several experiments into an impression of one deep technique. Honnibal created spaCy and cofounded Explosion. His article credits the tool-assisted work as performing tasks a human machine-learning engineer could do, while questioning the inference encouraged by the bundled presentation.
The tiny demonstrations have different inputs. In one, Astra receives raw weights from game-playing networks and names the games without tools. In another, it writes game-network weights from zero using the rules and source code, with no practice rollouts. The Qwen repair adds Python and model-run feedback. Each is a useful observation with a distinct boundary.
The game results also depend on the chosen baseline. Astra's generated networks beat PPO trained for 100 million steps in Breakout, Seaquest and Asterix. Against PPO trained for 500 million steps, Breakout remains a win, while Seaquest and Asterix become losses. The actual table preserves both comparison columns.
I would keep those columns visible in any presentation. “Beats trained networks” needs its opponent attached. Otherwise a change of baseline can disappear between the source table and the headline, which is a surprisingly efficient form of compression.
The smaller Gemma follow-up supplies a useful boundary
The researchers' Gemma follow-up asks how the method fares on a smaller model that rarely loops. Its observed failures include wrong arithmetic and invalid algebraic steps, without one removable component behind them. Astra again receives practice feedback and makes targeted edits.
The reported MATH-500 baseline is 6.8%. The best row is 7.7% at the 25-pass budget; the 50-pass row is 7.0%. The authors report no established improvement. A chart should keep that scale honest and distinguish the best reported row from the largest-budget row.
Gemma uses its own model size, sampling settings, output cap and grading procedure. It is a scoped failed follow-up, rather than an equal-controlled ablation proving looping was the only changed variable. Its value is concrete: the approach's successful Qwen repair did not produce an established gain in this second setting.
The released edited Qwen checkpoint card also credits akhaliq with a separate re-run. The edited model scored 51.0% on the full MATH-500 protocol, while the base check used only 25 problems. That supports the released edited model's performance under the stated settings. The asymmetric base sample prevents treating it as a matched full-baseline replication.
Verdict: NEEDS REVIEW
I'd take the targeted loop repair seriously and test the saved weights. I'd keep broader claims gated on improvements beyond that failure mode. For a developer, the first acceptance question is specific: what failure does the intervention repair, and how often does that failure occur in the workload that matters?
A useful evaluation preserves the original model, the candidate weights and the serving settings. It checks final-answer correctness alongside completion failures, and it retains examples that regress after the edit. The researchers' published protocol and counterexample make those questions easier to ask. The reported multiplier can start an investigation; the deployment decision still needs the application's own evidence.
FAQ
Did Astra use only raw Qwen weights?
The Qwen test allowed Python analysis, practice answers and batched model-run feedback. Its final benchmark questions were declared withheld. Tiny raw-number tasks used separate conditions.
Did the edit remove all loops?
About a third of edited MATH-500 answers still hit the length cap in the primary protocol. The repair reduced the failure and improved the reported score.
Does the Gemma result rule out weight editing?
It establishes a failed improvement attempt in that smaller model's setup. Broader conclusions need further experiments with clear inputs, protocols and counterexamples.
Sources
- Takehi and coauthors' study
- Honnibal's attributed critique
- Pinned Qwen protocol and logs
- Actual initial prompt
- Pinned Gemma follow-up
- Original Qwen model card
- Released checkpoint and attributed re-run
- Primary discussion and critique discussion
Cover portrait: Steve Jennings/Getty Images for TechCrunch, Sam Altman TechCrunch SF 2019 crop, CC BY 2.0. Cropped, background removed, resized and flipped; provider identification only, no reaction or endorsement. Research logs/data/networks: Rikiya Takehi and coauthors, CC BY 4.0. Primary footage is a brief credited excerpt for reporting/commentary.
This article expands on an episode of **The Daily Diff, a five-minute daily video on what shipped and what broke in tech.
Watch the episode · Subscribe on YouTube · the written diff lands in your inbox every morning at thedailydiff.dev.
Top comments (0)