I build Canli Capital: free, open-source tools that check whether a backtest is real (deflated Sharpe, overfitting and data-snooping tests), plus SEC fundamentals for AI agents over MCP.
Keeping the strict 0/60 as the official score and labelling the one-fence result as a post hoc diagnostic is exactly the right discipline. Most write-ups would quietly report 59/60.
Two things would make the Haiku number easier to read. The 24 compositions come from six recipes with four related variants each, so Haiku's 0/24 is closer to six independent results than 24; one failure mode in a shared ingredient would sink all four variants at once. A per-recipe table (and the same for the 36 originals by contract clause) would show whether it's one thing or many. And since 8 of Haiku's 46 diagnostic failures are responses the adapter couldn't touch, splitting the rest into "parsed but wrong" versus "didn't parse" separates a content gap from a formatting habit. If most of the other 38 are row omissions or dropped unknown columns, that's a much more specific finding than 14/60.
On Gemini's one miss: 7,996 tokens used against 448 visible characters suggests the budget went to hidden reasoning rather than output. Rerunning that single case with a larger cap would tell you whether 59/60 is really 60/60 with a budget problem, which matters for the long-preservation cases you're planning.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Keeping the strict 0/60 as the official score and labelling the one-fence result as a post hoc diagnostic is exactly the right discipline. Most write-ups would quietly report 59/60.
Two things would make the Haiku number easier to read. The 24 compositions come from six recipes with four related variants each, so Haiku's 0/24 is closer to six independent results than 24; one failure mode in a shared ingredient would sink all four variants at once. A per-recipe table (and the same for the 36 originals by contract clause) would show whether it's one thing or many. And since 8 of Haiku's 46 diagnostic failures are responses the adapter couldn't touch, splitting the rest into "parsed but wrong" versus "didn't parse" separates a content gap from a formatting habit. If most of the other 38 are row omissions or dropped unknown columns, that's a much more specific finding than 14/60.
On Gemini's one miss: 7,996 tokens used against 448 visible characters suggests the budget went to hidden reasoning rather than output. Rerunning that single case with a larger cap would tell you whether 59/60 is really 60/60 with a budget problem, which matters for the long-preservation cases you're planning.