Spark answered 2180, while the GSM8K reference said 2280. In Vincebye and SafeAnd’s community case, checking the original question reveals why: 430 + 750 + 300 + 700 = 2180. The reference substitutes 400 for 300 in its final sum. Pinned source.
We independently verified this question and arithmetic. Model outputs and aggregate scores remain participant-reported: we did not rerun the models or download the complete run artifacts, and the reasoning labels are heuristic. This single reference error does not establish a dataset-wide error rate. Read the case.
Join HER Hack-Astron #6: evaluate Spark-X2.5 1.7B or 4B, pin the revision, and retain prompts, complete outputs and scoring code. Publish your results in the tested model’s Hugging Face Discussions, include HER Hack-Astron #6 in the title, then reply to Issue #9 with the link.
Deadline: September 13, 2026, 24:00 Beijing time (UTC+8). Prize: one winner receives USD 100. At least 50% of team contributor profiles must identify as women. Read the full rules for individual eligibility and human review requirements. Share a real experiment, including its failures.
Top comments (0)