Contributor Ykmmz reports 160 generations of Spark-X2.5-1.7B using MLX and BF16 on an Apple M3 with 16 GB of memory. The evaluation combines a fixed 100-question GSM8K test subset with 20 original problem groups, each containing an original, an equivalent paraphrase, and a version with changed numbers.
The author reports 70/100 correct on the GSM8K subset, and 20/20, 19/20, and 20/20 for the original, paraphrased, and numerically changed variants. Decoding was greedy, with one generation per question and a 2,048-token limit. These results apply to the stated samples and budget.
Manual review of all 60 paired traces identified two correct final answers with arithmetic mistakes elsewhere in the reasoning. One correctly answered 38, yet wrote 8×38=296 during an extra check; the product is 304. Across all 160 generations, 25 reached the token limit. The author also documents a correction to reference labels while preserving the original outputs.
Raw outputs, model and runtime revisions, scoring code, run metadata, and reasoning audits are public. The production team checked these published materials and exactly recomputed the stored score report; it did not rerun model inference. This showcase is not an award announcement.
Read the participant’s case and reproduction repository.
To enter HER Hack-Astron #6, evaluate Spark-X2.5-1.7B or 4B, publish a complete reproducible report in the matching model’s Hugging Face Discussions with HER Hack-Astron #6 in the title, then reply to event Issue #9 with the link. Deadline: September 13, 2026, 24:00 Beijing time (UTC+8). One winner receives USD 100. Team entries must list all contributors, with at least 50% having profiles identifying them as women. See the event for full requirements.
Music: “Andante Moderato in F major” by Émile Paladilhe; synthesized performance by Scores2read. IMSLP, CC0 1.0. Source.
FAQ Quick Summary
Solo participants of any gender are eligible.
- The 50% women-contributor requirement applies only to teams.
- AI-assisted and AI-operated evaluations are allowed when clearly disclosed.
- Do not present AI personas as human participants.
- Actual, reproducible model runs are required.
- Formal submissions should follow the Hugging Face Discussion + GitHub Issue process.
- International winners may use available payout methods such as PayPal or bank transfer.
- Mainland China winners may coordinate RMB payout through an available domestic method.
- No upfront fee is required.
- Sensitive identity and payment information should only be provided privately if needed after winning.
- No live presentation is currently required.
Top comments (0)