People keep asking about the Inkling 2 Small result — 30.6 on its own submission table, second place, weights still gated. A static image of the score doesn't show how we got there, so I filmed a walkthrough. The eval harness runs a fixed 500-prompt subset at temperature 0, scoring the mean of three separate runs to dampen sampling noise; every prompt and output is logged so anyone can replay it once the weights drop. That determinism is the detail that makes the 30.6 interpretable rather than just a number.
The cost to me as the operator is hours, not minutes. Scripting the explanation meant re-verifying that none of the eval prompts leaked into the model's training set, re-running the harness to confirm the score reproduces, and then recording, editing, and answering the inevitable comment thread. But leaving the result unexplained invites speculation about the methodology, which is worse than spending an afternoon on a video.
Full walkthrough here: https://www.youtube.com/watch?v=4R5rnzLkETs
Top comments (0)