A correct intermediate calculation does not guarantee a correct delivered answer. Participant Pururin made that distinction visible in a local Spark-X2.5-1.7B math experiment for HER Hack-Astron #6.
Watch the short case walkthrough:
One laptop, 40 prompts
The participant used an RTX 5060 Laptop GPU with approximately 8GB VRAM, BF16, and no tool-assisted solving. Five mathematical structures, four numerical settings, and two equivalent wordings produced 40 English prompts. Each prompt received one greedy output under each setting.
| Reported setting | Strict correct finals | Missing numeric final | Token cap reached |
|---|---|---|---|
| Thinking off | 10/40 | 0 | 0 |
| Thinking on | 24/40 | 13 | 14 |
These are the participant's original strict scores. The output ceilings were 512 tokens off and 1024 on, so this comparison does not isolate the causal effect of thinking. It is a small, correlated template diagnostic, not a model-wide benchmark estimate.
The fraction was right; the delivered answer was wrong
In average_speed-large-v1, two route segments are each 840 km long, with speeds of 35 and 70 km/h. Total distance is 1680 km and total time is 36 hours, giving 140/3 km/h.
The model derived that fraction, then repeatedly deliberated about whether to output a fraction or a decimal. Its output reached the token ceiling and ended at FINAL: 140, cut partway through its intended fraction. The delivered value fails the scorer even though the preceding explanation derives the correct answer.
Other failures were semantic. Two uncapped responses misinterpreted keeping a fraction of water as removing it. The report notes a wording-and-number interaction, rather than claiming that every use of “keep” fails.
The strict absolute tolerance was 0.0001. Two thinking-off answers used 33.33 after a correct method and failed that tolerance. A separately disclosed, post-hoc check with tolerance 0.005 gave 12/40 off and 24/40 on; it does not replace the primary result.
What makes this entry useful
The case publishes its prompts, raw responses, scoring code, model revision, environment, and limitations. It keeps failed cases in the denominator and distinguishes proposals for further experiments from measured results.
Our video reformats the participant's public evidence as infographics and staged summaries. We did not rerun the model for the video. This is a community case showcase, not an award announcement.
Your turn: HER Hack-Astron #6
Use Spark-X2.5-1.7B or 4B for a real mathematical-reasoning evaluation. You could investigate paraphrases, numerical substitutions, reasoning errors, or comparisons with Python/SymPy tools.
- Pin the model revision and retain prompts, complete outputs, scoring code, and environment details.
- Publish the complete report in the Discussions section of the model you actually used. Include HER Hack-Astron #6 in the title.
- Reply to event Issue #9 with the direct Discussion link.
Deadline: September 13, 2026, 24:00 Beijing time (UTC+8), meaning September 14 at 00:00. Prize: one winner receives USD 100. For team entries, at least 50% of listed contributors must have profiles identifying them as women. Read the full event page for all acceptance and eligibility requirements.
Share a real run, including what failed, and make the next experiment easier to reproduce.
Video music: Courante in A minor, FVB 203 — Anonymous; performed by Jewgeni Smirnow (digital keyboard), IMSLP, CC BY 4.0. Excerpted, looped, normalized, and mixed under narration.
Top comments (0)