<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ward Ed</title>
    <description>The latest articles on DEV Community by Ward Ed (@ward_ed_6b5e6aa8ded94a987).</description>
    <link>https://dev.to/ward_ed_6b5e6aa8ded94a987</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4163315%2F0bd91171-b421-4ec2-9ce1-8301c0077eca.jpg</url>
      <title>DEV Community: Ward Ed</title>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ward_ed_6b5e6aa8ded94a987"/>
    <language>en</language>
    <item>
      <title>IFBench on Hugging Face: 0 of 23 Rows Are Verified, and Ranks 1 to 6 Sit Inside One 300-Prompt Error Band</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Thu, 19 Nov 2026 13:42:47 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/ifbench-on-hugging-face-0-of-23-rows-are-verified-and-ranks-1-to-6-sit-inside-one-300-prompt-oa4</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/ifbench-on-hugging-face-0-of-23-rows-are-verified-and-ranks-1-to-6-sit-inside-one-300-prompt-oa4</guid>
      <description>&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; The official IFBench board on Hugging Face (&lt;code&gt;allenai/IFBench_test&lt;/code&gt;) lists 23 rows for 21 distinct models. On Nov 19, 2026 none of the 23 is marked verified, 16 come from pull requests that are still open, and 15 of the 23 values cannot be a whole number of correct prompts out of the benchmark's 300. Ranks 1 to 6 span 3.3 points, which is about 10 prompts, and a rough 95% half-width at n=300 is about 4.6 points. Treat the top of this board as a cluster, not a podium.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522IFBench%2520score%252C%2520%2525%2520%2528blue%2520%253D%2520open%2520PR%252C%2520orange%2520%253D%2520on%2520main%252C%2520grey%2520%253D%2520lab%2520runs%2520%252F%2520EvalEval%2529%2522%252C%2522data%2522%253A%255B82.8%252C81.3%252C80.61%252C79.8%252C79.6%252C79.5%252C77%252C76.5%252C76.5%252C76.1%252C75.9%252C72%252C71.88%252C71.5%252C70%252C70%252C67.4%252C67.3%252C64.7%252C47%252C40%252C36.67%252C30%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522IFBench%2520board%2520JSON%252C%2520Nov%252019%252C%25202026%253A%252023%2520rows%252C%25200%2520verified%252C%252016%2520from%2520open%2520pull%2520requests%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522prompt-level%2520loose%2520accuracy%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522IFBench%2520score%252C%2520%2525%2520%2528blue%2520%253D%2520open%2520PR%252C%2520orange%2520%253D%2520on%2520main%252C%2520grey%2520%253D%2520lab%2520runs%2520%252F%2520EvalEval%2529%2522%252C%2522data%2522%253A%255B82.8%252C81.3%252C80.61%252C79.8%252C79.6%252C79.5%252C77%252C76.5%252C76.5%252C76.1%252C75.9%252C72%252C71.88%252C71.5%252C70%252C70%252C67.4%252C67.3%252C64.7%252C47%252C40%252C36.67%252C30%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522IFBench%2520board%2520JSON%252C%2520Nov%252019%252C%25202026%253A%252023%2520rows%252C%25200%2520verified%252C%252016%2520from%2520open%2520pull%2520requests%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522prompt-level%2520loose%2520accuracy%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" alt="IFBench board rows by rank" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;IFBench is a good benchmark to audit because its design is clear. This post reads the public board JSON the way a reviewer would read a results table: what is each row, where did it come from, and how much of the ordering can 300 prompts support. All numbers below were pulled from the Hugging Face API and from each row's &lt;code&gt;.eval_results&lt;/code&gt; file on Nov 19, 2026.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does IFBench actually measure?
&lt;/h2&gt;

&lt;p&gt;The dataset's own &lt;code&gt;eval.yaml&lt;/code&gt; describes it: 300 single-turn prompts from held-out WildChat conversations, paired with 58 out-of-domain constraints (counting, formatting, sentence, word and custom rules, one or two per prompt), each checked by a verification function. The headline metric is prompt-level loose accuracy. A prompt counts as correct only if every one of its constraints passes under loose verification, which also accepts the response with markdown asterisks or its first or last line removed. Scores are percentages, higher is better, and the board declares a single task, &lt;code&gt;prompt_level_loose&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Two consequences follow from that design:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The score granularity is 1/300, so one prompt is 0.333 points.&lt;/li&gt;
&lt;li&gt;Every row is supposed to answer the same question on the same 300 prompts. Whether that holds depends on how each row was produced, and the board does not show that.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What does the board JSON look like?
&lt;/h2&gt;

&lt;p&gt;The endpoint is public: &lt;code&gt;https://huggingface.co/api/datasets/allenai/IFBench_test/leaderboard&lt;/code&gt;. Each row carries &lt;code&gt;rank&lt;/code&gt;, &lt;code&gt;value&lt;/code&gt;, &lt;code&gt;verified&lt;/code&gt;, a &lt;code&gt;source&lt;/code&gt; object, the &lt;code&gt;pullRequest&lt;/code&gt; number when the score lives in an unmerged model-repo PR, &lt;code&gt;notes&lt;/code&gt;, and &lt;code&gt;num_parameters&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"User-Agent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Mozilla/5.0"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"utf-8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="no"&gt;DS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"allenai/IFBench_test"&lt;/span&gt;
&lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"https://huggingface.co/api/datasets/{DS}/leaderboard"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"rows,"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"modelId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"distinct models"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"verified:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"verified"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"from a pull request:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pullRequest"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;whole&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.02&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"values that are a whole number of 300 prompts:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;whole&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"top 6 spread:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Nov 19 this printed 23 rows, 21 distinct models, 0 verified, 16 from a pull request, 8 whole-count values, and a top-6 spread of 3.3. Everything else in this post is derived from those fields plus the raw YAML files they point to.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is anything on the IFBench board verified?
&lt;/h2&gt;

&lt;p&gt;No. All 23 rows have &lt;code&gt;verified: false&lt;/code&gt;. That is normal for a young board and says nothing bad about any model. It does change how to read the table: every number is a claim from the model's own card, a paper, or a lab's own run, not a number the benchmark owner reproduced.&lt;/p&gt;

&lt;p&gt;Provenance of the 23 rows:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Source type&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model-card YAML in an open pull request&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model-card YAML already on the repo's main branch&lt;/td&gt;
&lt;td&gt;3 (ranks 5, 11, 19)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lab evaluation runs and an EvalEval datastore entry&lt;/td&gt;
&lt;td&gt;4 (ranks 20 to 23)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Of the top six rows, five are open pull requests and one (rank 5) is on main. An open PR means the repo owner has not merged the score into the card yet. The board still counts it, which is a reasonable design for coverage, but it means a rank can move if a PR is edited or closed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can 300 prompts separate rank 1 from rank 6?
&lt;/h2&gt;

&lt;p&gt;Rank 1 is &lt;code&gt;Qwen/Qwen3.8-2.4T-A95B&lt;/code&gt; at 82.8. Rank 6 is &lt;code&gt;Qwen/Qwen3.8-27B&lt;/code&gt; at 79.5. The gap is 3.3 points, about 10 prompts.&lt;/p&gt;

&lt;p&gt;For a binomial proportion near 0.8 on 300 items the standard error is about 2.3 points, so a naive 95% half-width is about 4.6 points. That is an unpaired approximation. All rows answer the same 300 prompts, and paired differences between two models have a smaller variance than this unpaired figure, so the real uncertainty on a head-to-head gap is smaller. Even so, the test is small, and a 3.3-point spread across six rows is the size of ordinary sampling variation plus ordinary run-to-run variation from sampling temperature.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Qwen3.8-2.4T-A95B%2522%252C%2522Qwen3.8-Flash-Next%2522%252C%2522Agents-A1%2522%252C%2522Inkling%2522%252C%2522Beam-1.0-501B-A23B%2522%252C%2522Qwen3.8-27B%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522listed%2520score%2522%252C%2522data%2522%253A%255B82.8%252C81.3%252C80.61%252C79.8%252C79.6%252C79.5%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522listed%2520score%2520minus%25204.6%2520%2528rough%252095%2525%2520lower%2520bound%252C%2520n%253D300%2529%2522%252C%2522data%2522%253A%255B78.2%252C76.7%252C76.0%252C75.2%252C75.0%252C74.9%255D%252C%2522backgroundColor%2522%253A%2522%2523a0aec0%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Ranks%25201%2520to%25206%2520span%25203.3%2520points%253B%2520one%2520prompt%2520is%25200.33%2520points%2520and%2520a%2520rough%252095%2525%2520half-width%2520is%2520about%25204.6%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A60%252C%2522max%2522%253A90%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Qwen3.8-2.4T-A95B%2522%252C%2522Qwen3.8-Flash-Next%2522%252C%2522Agents-A1%2522%252C%2522Inkling%2522%252C%2522Beam-1.0-501B-A23B%2522%252C%2522Qwen3.8-27B%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522listed%2520score%2522%252C%2522data%2522%253A%255B82.8%252C81.3%252C80.61%252C79.8%252C79.6%252C79.5%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522listed%2520score%2520minus%25204.6%2520%2528rough%252095%2525%2520lower%2520bound%252C%2520n%253D300%2529%2522%252C%2522data%2522%253A%255B78.2%252C76.7%252C76.0%252C75.2%252C75.0%252C74.9%255D%252C%2522backgroundColor%2522%253A%2522%2523a0aec0%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Ranks%25201%2520to%25206%2520span%25203.3%2520points%253B%2520one%2520prompt%2520is%25200.33%2520points%2520and%2520a%2520rough%252095%2525%2520half-width%2520is%2520about%25204.6%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A60%252C%2522max%2522%253A90%257D%257D%255D%257D%257D%257D" alt="Ranks 1 to 6 and a rough lower bound" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The sizes make the point sharper. The 27B model at rank 6 trails the 2.4T-parameter model at rank 1 by 3.3 points, with roughly one eighty-eighth of the parameters (27.78B against 2,446B in the board's &lt;code&gt;num_parameters&lt;/code&gt; field). A board that sorts by score alone cannot show that, and for someone choosing a model to deploy, that ratio matters more than the order of the first six rows. Rank 3, &lt;code&gt;InternScience/Agents-A1&lt;/code&gt; at 80.61 with 35.1B parameters, is also ahead of the 27B Qwen row.&lt;/p&gt;

&lt;p&gt;Ties and near-ties add to this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ranks 8 and 9 both read 76.5, and both carry the date 2026-04-24.&lt;/li&gt;
&lt;li&gt;Ranks 13 and 14 (two Nemotron checkpoints) differ by 0.38 points, about one prompt.&lt;/li&gt;
&lt;li&gt;Ranks 15 and 16 (&lt;code&gt;MiniMax-M2.1&lt;/code&gt; and &lt;code&gt;MiniMax-M2.5&lt;/code&gt;) both read 70, so their order is a tie-break, not a result.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Are the values counts out of 300?
&lt;/h2&gt;

&lt;p&gt;Not for most rows. If a row came from one pass over the 300 prompts, &lt;code&gt;value * 3&lt;/code&gt; would be an integer. For 15 of 23 rows it is not. Rank 1 is 248.4 prompts, rank 2 is 243.9, rank 3 is 241.83 and rank 4 is 239.4. The most likely explanations are averaging over several runs, a different harness, or a different sample size, and the board has no column to say which. The YAML entries for these rows carry no sample-size field either.&lt;/p&gt;

&lt;p&gt;The eight whole-count values are 77, 72, 70, 70, 47, 40, 36.67 and 30. The last four are the Swiss AI rows, whose notes spell out the setup: &lt;code&gt;n=300&lt;/code&gt;, &lt;code&gt;repeats=1&lt;/code&gt;, temperature, &lt;code&gt;max_tokens&lt;/code&gt;, and prompt-level loose accuracy. That is what a row should disclose. The rest of the board mostly does not.&lt;/p&gt;

&lt;p&gt;One more row-level detail is worth knowing. Rank 12, &lt;code&gt;MiniMax-M2&lt;/code&gt; at 72, carries the note "Artificial Analysis methodology." It sits above &lt;code&gt;MiniMax-M2.1&lt;/code&gt; and &lt;code&gt;MiniMax-M2.5&lt;/code&gt; (70 each) from the same lab, all three with the date 2026-10-26. A newer checkpoint ranking below an older one is not impossible, but with a different methodology noted on only one of the three, the order of those three rows is not a like-for-like comparison.&lt;/p&gt;

&lt;h2&gt;
  
  
  What happens when one model has two rows?
&lt;/h2&gt;

&lt;p&gt;The board has 23 rows and 21 distinct models because &lt;code&gt;swiss-ai/Apertus-v1.5-8B&lt;/code&gt; and &lt;code&gt;swiss-ai/Apertus-v1.5-70B&lt;/code&gt; each appear twice. The note on the 47 and 36.67 rows says thinking is enabled with &lt;code&gt;temperature=0.6&lt;/code&gt;, &lt;code&gt;top_p=0.95&lt;/code&gt;, &lt;code&gt;max_tokens=8192&lt;/code&gt;. The 40 and 30 rows say thinking is disabled with &lt;code&gt;temperature=0.0&lt;/code&gt;, &lt;code&gt;max_tokens=2048&lt;/code&gt;. Same checkpoints, same 300 prompts, different settings.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25228B%2520thinking%2520on%2522%252C%25228B%2520thinking%2520off%2522%252C%252270B%2520thinking%2520on%2522%252C%252270B%2520thinking%2520off%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522Apertus%2520v1.5%2520IFBench%2520rows%2522%252C%2522data%2522%253A%255B47%252C40%252C36.67%252C30%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232f855a%2522%252C%2522%2523a0aec0%2522%252C%2522%25232f855a%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Same%2520two%2520checkpoints%252C%2520four%2520rows%253A%2520the%25208B%2520beats%2520the%252070B%2520once%2520settings%2520differ%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A60%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522score%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25228B%2520thinking%2520on%2522%252C%25228B%2520thinking%2520off%2522%252C%252270B%2520thinking%2520on%2522%252C%252270B%2520thinking%2520off%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522Apertus%2520v1.5%2520IFBench%2520rows%2522%252C%2522data%2522%253A%255B47%252C40%252C36.67%252C30%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232f855a%2522%252C%2522%2523a0aec0%2522%252C%2522%25232f855a%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Same%2520two%2520checkpoints%252C%2520four%2520rows%253A%2520the%25208B%2520beats%2520the%252070B%2520once%2520settings%2520differ%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A60%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522score%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" alt="Four Apertus rows" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The effect of settings is larger than the effect of scale in this slice: the 8B model with thinking on (47) outranks the 70B model with thinking off (30) and also the 70B model with thinking on (36.67). Board order is a function of the checkpoint and of how it was run. That is useful information about instruction following under a token budget, and it is also a reminder that rank positions mix two variables.&lt;/p&gt;

&lt;h2&gt;
  
  
  How should I read the IFBench board?
&lt;/h2&gt;

&lt;p&gt;A short checklist that works on this board and on others with the same JSON shape:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Check &lt;code&gt;verified&lt;/code&gt; first.&lt;/strong&gt; Zero here, so read every row as a self-reported claim.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check &lt;code&gt;pullRequest&lt;/code&gt;.&lt;/strong&gt; Sixteen of 23 rows are unmerged, so rows can change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compute &lt;code&gt;value * n&lt;/code&gt;.&lt;/strong&gt; If the product is not an integer, ask what was averaged.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read &lt;code&gt;notes&lt;/code&gt; and the YAML.&lt;/strong&gt; Look for methodology, thinking mode, temperature and token limits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Convert rank gaps into prompts.&lt;/strong&gt; At n=300, a 1-point gap is 3 prompts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look at &lt;code&gt;num_parameters&lt;/code&gt;.&lt;/strong&gt; Rank gaps with an 88x size difference deserve a different conclusion from rank gaps at equal size.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Group by lab.&lt;/strong&gt; Qwen holds 6 of the 23 rows and 6 of the top 10, so lab-level effects are part of the picture.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is IFBench?&lt;/strong&gt; A 300-prompt benchmark of precise, verifiable output constraints, with 58 constraints not seen in training, scored by prompt-level loose accuracy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Which model is first on the IFBench board on Hugging Face?&lt;/strong&gt; On Nov 19, 2026 it is &lt;code&gt;Qwen/Qwen3.8-2.4T-A95B&lt;/code&gt; at 82.8, from a model-card entry in an open pull request.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the IFBench ranking statistically meaningful?&lt;/strong&gt; The gap between rank 1 and rank 6 is 3.3 points on 300 prompts, inside a rough 4.6-point half-width. The coarse ordering (top cluster, middle cluster, lab runs at the bottom) is solid. The exact order within the top six is not.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why are some scores not multiples of 1/300?&lt;/strong&gt; Those values were likely averaged over repeated runs, rounded, or produced with a different harness. The board does not state which.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does an open pull request count on the board?&lt;/strong&gt; Yes. Sixteen of the 23 rows come from open PRs, so they appear on the board before the model repo has merged them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I reproduce this?&lt;/strong&gt; Yes. The snippet above uses only the public API and prints the same counts.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/deepswe-on-hugging-face-rank-1-is-an-agent-layer-21-points-above-its-base-model-and-22-of-34-rows-5geb"&gt;DeepSWE on Hugging Face: Rank 1 Is an Agent Layer 21 Points Above Its Base Model&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/gsm8k-on-hugging-face-4-of-the-top-12-rows-show-a-sibling-checkpoints-score-and-20-of-28-rows-1e50"&gt;GSM8K on Hugging Face: 4 of the Top 12 Rows Show a Sibling Checkpoint's Score&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Sources: &lt;a href="https://huggingface.co/datasets/allenai/IFBench_test" rel="noopener noreferrer"&gt;IFBench dataset and eval.yaml&lt;/a&gt;, &lt;a href="https://huggingface.co/api/datasets/allenai/IFBench_test/leaderboard" rel="noopener noreferrer"&gt;board JSON&lt;/a&gt;. Data pulled Nov 19, 2026; boards change daily.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>DeepSWE on Hugging Face: Rank 1 Is an Agent Layer 21 Points Above Its Base Model, and 22 of 34 Rows Name No Scaffold</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Wed, 18 Nov 2026 13:46:44 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/deepswe-on-hugging-face-rank-1-is-an-agent-layer-21-points-above-its-base-model-and-22-of-34-rows-5geb</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/deepswe-on-hugging-face-rank-1-is-an-agent-layer-21-points-above-its-base-model-and-22-of-34-rows-5geb</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On November 18, 2026 the Hugging Face board for the DeepSWE coding-agent benchmark lists &lt;strong&gt;34 rows from 34 repos (19 organisations)&lt;/strong&gt;, each a single percentage over 113 tasks. 20 rows come from pull requests that are still open, 14 sit on &lt;code&gt;main&lt;/code&gt;, and all 34 carry &lt;code&gt;verified: false&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rank 1 is a different kind of row.&lt;/strong&gt; It is &lt;code&gt;PCL-RSI-lab/Seshat-agent&lt;/code&gt;, 90.3 (102 of 113 tasks), 13.3 points above rank 2. That is the biggest step on the board; the next biggest is 9.87. The repo holds a README and two YAML files and no weights, and its own note reads "Seshat + GLM-5.3".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The base model it wraps is on the same board at rank 12.&lt;/strong&gt; GLM-5.3 shows 66.9 on its model card. The benchmark's own site shows 68.96 pass@1 and 87.6 pass@4 for it. The rank 1 row is 21.3 points above the official pass@1 and 3 tasks above the official best-of-4.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The rest of the board is consistent with the official site.&lt;/strong&gt; Five more rows can be matched to it, and they agree within 2.4 points (mean 1.1). So the board's data pipeline is fine; the problem is that one column holds different kinds of measurement.&lt;/li&gt;
&lt;li&gt;22 of 34 notes name no agent scaffold. 16 of 33 steps between neighbouring rows are 1.0 point or less, and one task is 0.88 point. A single row's 95% interval is about 17 points wide.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rank%25201%253A%2520agent%2520layer%2520on%2520GLM-5.3%2520%2528no%2520scaffold%2520named%2520in%2520notes%2529%2522%252C%2522data%2522%253A%255B90.3%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%255D%252C%2522backgroundColor%2522%253A%2522%2523c53030%2522%257D%252C%257B%2522label%2522%253A%2522notes%2520name%2520a%2520scaffold%2520%252812%2520rows%2529%2522%252C%2522data%2522%253A%255Bnull%252C77%252C75.5%252C74.2%252C73.7%252C72.1%252Cnull%252Cnull%252Cnull%252C67.7%252Cnull%252Cnull%252Cnull%252C63.4%252C63.4%252Cnull%252Cnull%252C59.7%252Cnull%252C58.7%252Cnull%252Cnull%252Cnull%252Cnull%252C55.8%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C30.53%252Cnull%252Cnull%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522notes%2520name%2520no%2520scaffold%2520%252821%2520rows%2529%2522%252C%2522data%2522%253A%255Bnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C71.9%252C71.7%252C67.9%252Cnull%252C67.5%252C66.9%252C64.3%252Cnull%252Cnull%252C62.7%252C60%252Cnull%252C59.3%252Cnull%252C58.4%252C58.4%252C56.6%252C56%252Cnull%252C54.4%252C53.2%252C46.2%252C43.3%252C42.2%252C40.4%252Cnull%252C28%252C22%255D%252C%2522backgroundColor%2522%253A%2522%2523a0aec0%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522DeepSWE%2520board%2520JSON%252C%2520Nov%252018%252C%25202026%253A%2520all%252034%2520rows%2520in%2520board%2520order%252C%2520colored%2520by%2520what%2520the%2520notes%2520field%2520says%2520about%2520the%2520scaffold%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522stacked%2522%253Atrue%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522stacked%2522%253Atrue%252C%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522listed%2520value%252C%2520%2525%2520of%2520113%2520tasks%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rank%25201%253A%2520agent%2520layer%2520on%2520GLM-5.3%2520%2528no%2520scaffold%2520named%2520in%2520notes%2529%2522%252C%2522data%2522%253A%255B90.3%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%255D%252C%2522backgroundColor%2522%253A%2522%2523c53030%2522%257D%252C%257B%2522label%2522%253A%2522notes%2520name%2520a%2520scaffold%2520%252812%2520rows%2529%2522%252C%2522data%2522%253A%255Bnull%252C77%252C75.5%252C74.2%252C73.7%252C72.1%252Cnull%252Cnull%252Cnull%252C67.7%252Cnull%252Cnull%252Cnull%252C63.4%252C63.4%252Cnull%252Cnull%252C59.7%252Cnull%252C58.7%252Cnull%252Cnull%252Cnull%252Cnull%252C55.8%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C30.53%252Cnull%252Cnull%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522notes%2520name%2520no%2520scaffold%2520%252821%2520rows%2529%2522%252C%2522data%2522%253A%255Bnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C71.9%252C71.7%252C67.9%252Cnull%252C67.5%252C66.9%252C64.3%252Cnull%252Cnull%252C62.7%252C60%252Cnull%252C59.3%252Cnull%252C58.4%252C58.4%252C56.6%252C56%252Cnull%252C54.4%252C53.2%252C46.2%252C43.3%252C42.2%252C40.4%252Cnull%252C28%252C22%255D%252C%2522backgroundColor%2522%253A%2522%2523a0aec0%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522DeepSWE%2520board%2520JSON%252C%2520Nov%252018%252C%25202026%253A%2520all%252034%2520rows%2520in%2520board%2520order%252C%2520colored%2520by%2520what%2520the%2520notes%2520field%2520says%2520about%2520the%2520scaffold%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522stacked%2522%253Atrue%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522stacked%2522%253Atrue%252C%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522listed%2520value%252C%2520%2525%2520of%2520113%2520tasks%2522%257D%257D%255D%257D%257D%257D" alt="DeepSWE board rows in board order, colored by what the notes field says about the agent scaffold" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What is the DeepSWE leaderboard on Hugging Face?
&lt;/h2&gt;

&lt;p&gt;DeepSWE is a set of 113 long-horizon engineering tasks drawn from 91 repositories in 5 languages. The benchmark's own site (deepswe.datacurve.ai) runs every model through one open scaffold, &lt;code&gt;mini-swe-agent&lt;/code&gt;, four attempts per task, and publishes two numbers: &lt;strong&gt;pass@1&lt;/strong&gt;, the pass rate over all scored attempts, and &lt;strong&gt;pass@4&lt;/strong&gt;, the share of tasks with at least one passing attempt.&lt;/p&gt;

&lt;p&gt;The Hugging Face board is assembled differently. Model repos carry a file such as &lt;code&gt;.eval_results/deep-swe.yaml&lt;/code&gt; with one &lt;code&gt;value&lt;/code&gt; and a free-text &lt;code&gt;notes&lt;/code&gt; field, and the board at &lt;code&gt;GET /api/datasets/datacurve/deep-swe/leaderboard&lt;/code&gt; sorts those values in descending order. There is no column for scaffold, number of attempts, benchmark version or metric type. Whatever the author knows about those goes into &lt;code&gt;notes&lt;/code&gt;, if anywhere.&lt;/p&gt;

&lt;p&gt;That design is easy to join and easy to extend, and it works well when every row is the same kind of number. The question for this audit is how many of the 34 rows are.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is rank 1 on the DeepSWE board a different kind of row?
&lt;/h2&gt;

&lt;p&gt;Four facts from public files, no inference needed:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The board row is &lt;code&gt;PCL-RSI-lab/Seshat-agent&lt;/code&gt;. Its file list is &lt;code&gt;README.md&lt;/code&gt;, &lt;code&gt;.gitattributes&lt;/code&gt; and two eval YAML files. There are no model weights, and the repo shows 0 downloads.&lt;/li&gt;
&lt;li&gt;The notes field says "Seshat + GLM-5.3; DeepSWE-v1.1 (113), resolve rate 102/113=90.3%". The README describes an agent layer that adds a reusable knowledge base around a base model, and it quotes the uplift on GLM-5.3 as 59.3% to 90.3%.&lt;/li&gt;
&lt;li&gt;The linked evidence dataset says &lt;code&gt;agent: mini-swe-agent&lt;/code&gt;, &lt;code&gt;model: openai/GLM-5.3&lt;/code&gt;, &lt;code&gt;reasoning_effort: max&lt;/code&gt;, 102 resolved, 11 unresolved, 4 empty patches, generated 2026-11-02. It contains patches and pass/fail flags only, with no trajectories or prompts.&lt;/li&gt;
&lt;li&gt;Every other row is a model checkpoint with a score from its own card or from the official site.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So rank 1 is "an agent system built on GLM-5.3", ranked in the same column as the checkpoints it sits above. Nothing in the board marks the difference.&lt;/p&gt;

&lt;p&gt;I am not claiming that 90.3 is wrong. Outputs-only evidence is a common way to publish, and a different layer around a model can legitimately move a score. The point is that a reader cannot tell from the board what the number measures, and the checkable comparison is awkward:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Row&lt;/th&gt;
&lt;th&gt;Board&lt;/th&gt;
&lt;th&gt;Official pass@1&lt;/th&gt;
&lt;th&gt;Official pass@4&lt;/th&gt;
&lt;th&gt;Board minus pass@1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;GLM-5.3 (rank 12)&lt;/td&gt;
&lt;td&gt;66.9&lt;/td&gt;
&lt;td&gt;68.96&lt;/td&gt;
&lt;td&gt;87.6&lt;/td&gt;
&lt;td&gt;-2.06&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kimi-K3 (rank 11)&lt;/td&gt;
&lt;td&gt;67.5&lt;/td&gt;
&lt;td&gt;68.51&lt;/td&gt;
&lt;td&gt;89.4&lt;/td&gt;
&lt;td&gt;-1.01&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-5.3-Flash (rank 14)&lt;/td&gt;
&lt;td&gt;63.4&lt;/td&gt;
&lt;td&gt;63.39&lt;/td&gt;
&lt;td&gt;85.0&lt;/td&gt;
&lt;td&gt;+0.01&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-5.2 (rank 28)&lt;/td&gt;
&lt;td&gt;46.2&lt;/td&gt;
&lt;td&gt;43.78&lt;/td&gt;
&lt;td&gt;77.0&lt;/td&gt;
&lt;td&gt;+2.42&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kimi-K2.7-Code (rank 32)&lt;/td&gt;
&lt;td&gt;30.53&lt;/td&gt;
&lt;td&gt;30.53&lt;/td&gt;
&lt;td&gt;61.1&lt;/td&gt;
&lt;td&gt;0.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Seshat + GLM-5.3 (rank 1)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;90.3&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;68.96 (GLM-5.3)&lt;/td&gt;
&lt;td&gt;87.6 (GLM-5.3)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+21.34&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Official numbers are the &lt;code&gt;max&lt;/code&gt; reasoning-effort configuration of each model on the benchmark site's page, which was generated on 2026-09-22. 90.3 is 102 tasks, and GLM-5.3's official pass@4 is 99 tasks (87.6% of 113). The rank 1 row is above what four tries of the plain model reach on the official scaffold.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522GLM-5.3%2522%252C%2522Kimi-K3%2522%252C%2522GLM-5.3-Flash%2522%252C%2522GLM-5.2%2522%252C%2522Kimi-K2.7-Code%2522%252C%2522Seshat%2520row%2520%2528GLM-5.3%2520base%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522board%2520row%2522%252C%2522data%2522%253A%255B66.9%252C67.5%252C63.4%252C46.2%252C30.53%252C90.3%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522official%2520pass%25401%2520%2528mean%2520of%25204%2520attempts%2529%2522%252C%2522data%2522%253A%255B68.96%252C68.51%252C63.39%252C43.78%252C30.53%252C68.96%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%252C%257B%2522label%2522%253A%2522official%2520pass%25404%2520%2528best%2520of%25204%2520attempts%2529%2522%252C%2522data%2522%253A%255B87.6%252C89.4%252C85.0%252C77.0%252C61.1%252C87.6%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Board%2520rows%2520vs%2520the%2520benchmark%2527s%2520own%2520site%253A%2520five%2520rows%2520match%2520pass%25401%2520within%25202.4%2520points%252C%2520the%2520sixth%2520sits%2520above%2520pass%25404%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520of%2520113%2520tasks%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522GLM-5.3%2522%252C%2522Kimi-K3%2522%252C%2522GLM-5.3-Flash%2522%252C%2522GLM-5.2%2522%252C%2522Kimi-K2.7-Code%2522%252C%2522Seshat%2520row%2520%2528GLM-5.3%2520base%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522board%2520row%2522%252C%2522data%2522%253A%255B66.9%252C67.5%252C63.4%252C46.2%252C30.53%252C90.3%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522official%2520pass%25401%2520%2528mean%2520of%25204%2520attempts%2529%2522%252C%2522data%2522%253A%255B68.96%252C68.51%252C63.39%252C43.78%252C30.53%252C68.96%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%252C%257B%2522label%2522%253A%2522official%2520pass%25404%2520%2528best%2520of%25204%2520attempts%2529%2522%252C%2522data%2522%253A%255B87.6%252C89.4%252C85.0%252C77.0%252C61.1%252C87.6%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Board%2520rows%2520vs%2520the%2520benchmark%2527s%2520own%2520site%253A%2520five%2520rows%2520match%2520pass%25401%2520within%25202.4%2520points%252C%2520the%2520sixth%2520sits%2520above%2520pass%25404%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520of%2520113%2520tasks%2522%257D%257D%255D%257D%257D%257D" alt="Board rows against the benchmark's own pass@1 and pass@4, with the rank 1 row on the right" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;One more detail from the repo's own README: it puts the starting agent at 59.3%, which is below both the model card's 66.9 and the official 68.96. The uplift the repo quotes (31.0 points) is therefore larger than the gap to the official base (21.3 points). Both are consistent with a real improvement, and both are consistent with a different baseline setup. The public files do not settle which.&lt;/p&gt;

&lt;h2&gt;
  
  
  Do the other DeepSWE rows agree with the official site?
&lt;/h2&gt;

&lt;p&gt;Yes, and that is the useful finding about the board itself. Five rows can be matched one to one to an official configuration, and the differences are -2.06, -1.01, +0.01, +2.42 and 0.00 points. Kimi-K2.7-Code's row copies the official figure exactly, and it is the only row whose note says how the number was made: "Pass@1 over 452 attempts on 113 tasks, mini-swe-agent harness".&lt;/p&gt;

&lt;p&gt;The model-card numbers are close to the official site's pass@1 which fits labs running the same open scaffold. The GLM-5.3 card, for example, states the &lt;code&gt;mini-swe-agent&lt;/code&gt; harness with &lt;code&gt;temperature=0.95&lt;/code&gt;, &lt;code&gt;top_p=1.0&lt;/code&gt;, a 6-hour timeout and a 400K context window.&lt;/p&gt;

&lt;p&gt;The gap between the two official numbers is where a column with no metric field loses information. Across the 65 model, scaffold and effort configurations on the official page, pass@4 is a median of 21.9 points above pass@1, and 57 of the 65 are more than 15 points apart. A row that reports a best-of-k figure and a row that reports a mean over attempts differ by about twice the span between ranks 2 and 12 (77.0 down to 66.9 is 10.1 points). Nothing on the board says which one a given row is.&lt;/p&gt;

&lt;h2&gt;
  
  
  How many of the 34 rows are comparable with each other?
&lt;/h2&gt;

&lt;p&gt;Less than the sort order suggests. Four checks, all computed from the board JSON and the YAML files it points to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Scaffold.&lt;/strong&gt; 12 of 34 notes name one (7 &lt;code&gt;mini-swe-agent&lt;/code&gt;, plus Claude Code, SWE-agent, LingCode, NexAU, and one "Claude Code / mini-SWE-agent (best of two)"). 22 name none, including rank 1 and one row that says only "the authors' preferred agent harness". The benchmark's own site runs one scaffold for everything, so a card score from another harness is a different experiment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Version.&lt;/strong&gt; The official site has separate v1.1 and v1 leaderboards. 13 notes say v1.1, and 21 do not say. Every YAML uses the same &lt;code&gt;task_id&lt;/code&gt;, &lt;code&gt;deep_swe&lt;/code&gt;, so the schema cannot separate them either.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Source.&lt;/strong&gt; 20 of 34 rows come from pull requests on model repos that are still open at fetch time, so the model owner has not merged them. 14 are on &lt;code&gt;main&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Duplicates and lineage.&lt;/strong&gt; Ranks 14 and 15 both show 63.4. Rank 15 is &lt;code&gt;inference-optimization/GLM-5.3-Flash-MXFP4xFP8&lt;/code&gt;, a quantized checkpoint whose YAML points at the original model's card as its source, with the same date and the same note. The board lists it with the original's score, and nothing says the quantized weights were run. The official site's number for the original is 63.39. Five of the 34 rows are DeepSeek snapshots, and ranks 21 and 22 also tie at 58.4.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522tie%2520%25280.00%2529%2522%252C%25220.01%2520to%25200.89%2522%252C%25220.9%2520to%25201.0%2522%252C%25221.1%2520to%25202.0%2522%252C%25222.1%2520to%25205.0%2522%252C%2522over%25205.0%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522steps%2520between%2520neighbouring%2520board%2520rows%2522%252C%2522data%2522%253A%255B2%252C13%252C1%252C8%252C5%252C4%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%252233%2520steps%2520between%2520neighbouring%2520DeepSWE%2520rows%252C%2520by%2520size%2520in%2520points%2520%2528one%2520task%2520of%2520113%2520%253D%25200.88%2520point%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522plugins%2522%253A%257B%2522datalabels%2522%253A%257B%2522display%2522%253Atrue%252C%2522anchor%2522%253A%2522end%2522%252C%2522align%2522%253A%2522end%2522%252C%2522font%2522%253A%257B%2522size%2522%253A12%257D%257D%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522gap%2520to%2520the%2520next%2520row%2520down%252C%2520points%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A16%252C%2522stepSize%2522%253A4%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522number%2520of%2520steps%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522tie%2520%25280.00%2529%2522%252C%25220.01%2520to%25200.89%2522%252C%25220.9%2520to%25201.0%2522%252C%25221.1%2520to%25202.0%2522%252C%25222.1%2520to%25205.0%2522%252C%2522over%25205.0%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522steps%2520between%2520neighbouring%2520board%2520rows%2522%252C%2522data%2522%253A%255B2%252C13%252C1%252C8%252C5%252C4%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%252233%2520steps%2520between%2520neighbouring%2520DeepSWE%2520rows%252C%2520by%2520size%2520in%2520points%2520%2528one%2520task%2520of%2520113%2520%253D%25200.88%2520point%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522plugins%2522%253A%257B%2522datalabels%2522%253A%257B%2522display%2522%253Atrue%252C%2522anchor%2522%253A%2522end%2522%252C%2522align%2522%253A%2522end%2522%252C%2522font%2522%253A%257B%2522size%2522%253A12%257D%257D%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522gap%2520to%2520the%2520next%2520row%2520down%252C%2520points%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A16%252C%2522stepSize%2522%253A4%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522number%2520of%2520steps%2522%257D%257D%255D%257D%257D%257D" alt="Number of steps between neighbouring board rows, grouped by size in points" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Is a 1-point gap on a 113-task board a real difference?
&lt;/h2&gt;

&lt;p&gt;No. One task is 0.88 point. 16 of the 33 steps between neighbouring rows are 1.0 point or less, and 2 are exact ties. A pass rate of 66.9% on 113 tasks (76 tasks) has a Wilson 95% interval of 58.2 to 75.2, about 17 points wide. Rank 1's 102 tasks gives 83.4 to 94.5. The official site reports a run-to-run 95% interval of plus or minus 3.0 points for GLM-5.3 (65.9 to 72.0, from four repeated passes), and 28 of the 33 steps on the board are 3.0 points or less.&lt;/p&gt;

&lt;p&gt;The ordering that survives these checks is coarse: a top group of cards between 71.7 and 77.0, a middle between 53 and 68, and a tail below 47. Within each group, rank is mostly noise.&lt;/p&gt;

&lt;h2&gt;
  
  
  How should I read the DeepSWE board?
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Open the repo behind the row.&lt;/strong&gt; Check whether it holds weights or only a result file, and read the notes field before the number.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Find the metric.&lt;/strong&gt; Mean over attempts, best of k and a single run are different quantities. On this benchmark the first two differ by about 22 points.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check the scaffold and version.&lt;/strong&gt; If the notes do not name them, treat the row as unmatched.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compare to the benchmark's own site for the same model.&lt;/strong&gt; Five of six matched here, and the exception is the one that deserves the question.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Merge duplicates.&lt;/strong&gt; A quantized checkpoint that carries the base model's card score is one measurement, not two rows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not rank within 2 points.&lt;/strong&gt; On 113 tasks that is two to three problems.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The check takes a few lines of Python against the public API:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"User-Agent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Mozilla/5.0"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"https://huggingface.co/api/datasets/datacurve/deep-swe/leaderboard"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;named&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"mini[- ]swe|claude code|swe-agent|lingcode|nexau"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"rows;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pullRequest"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"from open PRs;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;named&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"notes"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"name no harness"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;top&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rank 1:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"modelId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"|"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"notes"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;wilson&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.96&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;
    &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"modelId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"zai-org/GLM-5.3"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"PCL-RSI-lab/Seshat-agent"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;113&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"rank"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"modelId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"= {k}/113"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"95% CI"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;wilson&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;113&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output on November 18, 2026:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;34 rows; 20 from open PRs; 22 name no harness
rank 1: PCL-RSI-lab/Seshat-agent 90.3 | Seshat + GLM-5.3; DeepSWE-v1.1 (113), resolve rate 102/113=90.3%
1 PCL-RSI-lab/Seshat-agent 90.3 = 102/113 95% CI (83.4, 94.5)
12 zai-org/GLM-5.3 66.9 = 76/113 95% CI (58.2, 75.2)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;pullRequest&lt;/code&gt; field is present for rows read from a pull request. I confirmed their status separately through the discussions API, which showed all 20 as open.&lt;/p&gt;

&lt;h2&gt;
  
  
  What would make the board harder to misread?
&lt;/h2&gt;

&lt;p&gt;The information needed is mostly already in the files and gets dropped when the board is assembled. Four optional fields in the eval YAML would remove most of the ambiguity: the &lt;strong&gt;metric&lt;/strong&gt; (&lt;code&gt;pass@1&lt;/code&gt;, &lt;code&gt;pass@k&lt;/code&gt;, single run), the &lt;strong&gt;scaffold&lt;/strong&gt;, the &lt;strong&gt;benchmark version&lt;/strong&gt;, and a &lt;strong&gt;row type&lt;/strong&gt; (model checkpoint or agent system). A board could then sort within a row type, show system rows in their own band, and fold a derivative checkpoint that reuses its base model's source into that model's entry. None of that changes what any author publishes; it changes what the reader is told.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is the DeepSWE leaderboard on Hugging Face?&lt;/strong&gt;&lt;br&gt;
It is the board on the dataset page for &lt;code&gt;datacurve/deep-swe&lt;/code&gt;, a benchmark of 113 long-horizon software tasks. Model repos publish YAML result files and the board sorts the values. As of November 18, 2026 it has 34 rows from 34 repos.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Who is at rank 1 on the Hugging Face DeepSWE board?&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;PCL-RSI-lab/Seshat-agent&lt;/code&gt; with 90.3 (102 of 113 tasks), a repo that holds a README and result files and describes an agent layer around GLM-5.3. The plain GLM-5.3 model is at rank 12 with 66.9.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the 90.3 DeepSWE score wrong?&lt;/strong&gt;&lt;br&gt;
The public files do not show that. The evidence dataset contains patches and pass/fail flags but no trajectories, so a reader cannot re-derive it. What can be said is that it is 21.3 points above the official pass@1 of the base model and 3 tasks above its official pass@4, so it is not comparable with a plain model score.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is the difference between pass@1 and pass@4 on DeepSWE?&lt;/strong&gt;&lt;br&gt;
On the official site, pass@1 is the pass rate over all scored attempts, and pass@4 is the share of the 113 tasks solved by at least one of four attempts. Across 65 configurations pass@4 is a median of 21.9 points higher.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do Hugging Face model-card DeepSWE scores match the official site?&lt;/strong&gt;&lt;br&gt;
For the five matched rows, yes, within 2.4 points (mean 1.1). The mismatch on this board comes from a row of a different kind, not from card scores drifting.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How many tasks separate two DeepSWE rows that are 1 point apart?&lt;/strong&gt;&lt;br&gt;
About one. A single task is 0.88 point of 113, and a 66.9% result has a 95% interval of 58.2 to 75.2.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-verified-on-hugging-face-9-self-reported-rows-top-the-official-tables-best-and-6-repos-3173"&gt;SWE-bench Verified on Hugging Face: 9 Self-Reported Rows Top the Official Table's Best, and 6 Repos Fill 16 of 78 Rows&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/wildclawbench-on-hugging-face-rank-1-is-a-runtime-because-scores-minutes-and-dollars-share-one-18io"&gt;WildClawBench on Hugging Face: Rank 1 Is a Runtime, Because Scores, Minutes and Dollars Share One Ranked Column&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Board API: &lt;a href="https://huggingface.co/api/datasets/datacurve/deep-swe/leaderboard" rel="noopener noreferrer"&gt;https://huggingface.co/api/datasets/datacurve/deep-swe/leaderboard&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Benchmark site: &lt;a href="https://deepswe.datacurve.ai/" rel="noopener noreferrer"&gt;https://deepswe.datacurve.ai/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Rank 1 repo: &lt;a href="https://huggingface.co/PCL-RSI-lab/Seshat-agent" rel="noopener noreferrer"&gt;https://huggingface.co/PCL-RSI-lab/Seshat-agent&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Rank 1 evidence dataset: &lt;a href="https://huggingface.co/datasets/PCL-RSI-lab/DeepSWE-v1.1-Evaluation-Evidence" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/PCL-RSI-lab/DeepSWE-v1.1-Evaluation-Evidence&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GLM-5.3 model card: &lt;a href="https://huggingface.co/zai-org/GLM-5.3" rel="noopener noreferrer"&gt;https://huggingface.co/zai-org/GLM-5.3&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>benchmark</category>
      <category>llm</category>
    </item>
    <item>
      <title>WildClawBench on Hugging Face: Rank 1 Is a Runtime, Because Scores, Minutes and Dollars Share One Ranked Column</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Wed, 11 Nov 2026 02:34:39 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/wildclawbench-on-hugging-face-rank-1-is-a-runtime-because-scores-minutes-and-dollars-share-one-18io</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/wildclawbench-on-hugging-face-rank-1-is-a-runtime-because-scores-minutes-and-dollars-share-one-18io</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On November 11, 2026 the Hugging Face board for the WildClawBench agent benchmark lists &lt;strong&gt;52 rows from 23 open-weight models, and the rows are three different measurements in one column:&lt;/strong&gt; 24 are overall scores in percent, 16 are total minutes and 12 are total dollars. The board ranks them together, high to low, and all 52 rows carry &lt;code&gt;lower_is_better: false&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ranks 1 to 16 are all minutes.&lt;/strong&gt; Rank 1 is 708, the slowest model's runtime for the 60 tasks. The 24 score rows sit at ranks 17 to 42 mixed with 2 cost rows, and the 10 cheapest-looking rows fill ranks 43 to 52, so the cheapest run is ranked last.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The values themselves are right.&lt;/strong&gt; For 14 of the 15 models that also appear in the benchmark's own README table, every value on the board matches it. The problem is the ranking and the units, not the data. The one exception is a model-card score for Hy3 that is 3.9 points above the README's.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read one metric at a time, the picture is useful.&lt;/strong&gt; Among the 12 models that have all three rows, GLM-5.2 scores 54.2 against Kimi-K3's 54.5 at 43% of the total cost ($17.10 against $40.08). Score tracks cost (rank correlation 0.82) and barely tracks time (0.16).&lt;/li&gt;
&lt;li&gt;Score gaps are small for a 60-task suite: 13 of the 23 steps between neighbouring score rows are 1.0 point or less, and the README shows a model moving by up to 18 points when only the agent harness changes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%252C%252235%2522%252C%252236%2522%252C%252237%2522%252C%252238%2522%252C%252239%2522%252C%252240%2522%252C%252241%2522%252C%252242%2522%252C%252243%2522%252C%252244%2522%252C%252245%2522%252C%252246%2522%252C%252247%2522%252C%252248%2522%252C%252249%2522%252C%252250%2522%252C%252251%2522%252C%252252%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522total%2520minutes%2520%252816%2520rows%2529%2522%252C%2522data%2522%253A%255B708%252C674%252C605%252C551%252C542%252C541%252C516%252C515%252C488%252C459%252C451%252C442%252C430%252C421%252C406%252C373%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522score%252C%2520%2525%2520%252824%2520rows%2529%2522%252C%2522data%2522%253A%255Bnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C61.7%252C56.2%252C54.5%252C54.2%252C53.6%252C53.5%252C52.2%252C48.2%252C48.0152%252C47.6%252C46.9%252C44.7%252C43.7%252C43.2%252C43%252C42.6%252C41.5%252Cnull%252C39.2%252C35%252Cnull%252C34.5%252C33.8%252C30.8%252C27.1%252C26.7%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%252C%257B%2522label%2522%253A%2522total%2520cost%252C%2520USD%2520%252812%2520rows%2529%2522%252C%2522data%2522%253A%255Bnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C40.08%252Cnull%252Cnull%252C34.8%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C22.2%252C20.91%252C17.1%252C12.6%252C12%252C11.4%252C9.6%252C7.2%252C6.6%252C6.6%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522WildClawBench%2520board%2520JSON%252C%2520Nov%252011%252C%25202026%253A%2520all%252052%2520rows%2520in%2520board%2520order%252C%2520one%2520column%252C%2520three%2520different%2520units%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522stacked%2522%253Atrue%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522stacked%2522%253Atrue%252C%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A750%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522value%2520as%2520listed%2520%2528no%2520common%2520unit%2529%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%252C%252235%2522%252C%252236%2522%252C%252237%2522%252C%252238%2522%252C%252239%2522%252C%252240%2522%252C%252241%2522%252C%252242%2522%252C%252243%2522%252C%252244%2522%252C%252245%2522%252C%252246%2522%252C%252247%2522%252C%252248%2522%252C%252249%2522%252C%252250%2522%252C%252251%2522%252C%252252%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522total%2520minutes%2520%252816%2520rows%2529%2522%252C%2522data%2522%253A%255B708%252C674%252C605%252C551%252C542%252C541%252C516%252C515%252C488%252C459%252C451%252C442%252C430%252C421%252C406%252C373%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522score%252C%2520%2525%2520%252824%2520rows%2529%2522%252C%2522data%2522%253A%255Bnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C61.7%252C56.2%252C54.5%252C54.2%252C53.6%252C53.5%252C52.2%252C48.2%252C48.0152%252C47.6%252C46.9%252C44.7%252C43.7%252C43.2%252C43%252C42.6%252C41.5%252Cnull%252C39.2%252C35%252Cnull%252C34.5%252C33.8%252C30.8%252C27.1%252C26.7%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%252C%257B%2522label%2522%253A%2522total%2520cost%252C%2520USD%2520%252812%2520rows%2529%2522%252C%2522data%2522%253A%255Bnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C40.08%252Cnull%252Cnull%252C34.8%252Cnull%252Cnull%252Cnull%252Cnull%252Cnull%252C22.2%252C20.91%252C17.1%252C12.6%252C12%252C11.4%252C9.6%252C7.2%252C6.6%252C6.6%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522WildClawBench%2520board%2520JSON%252C%2520Nov%252011%252C%25202026%253A%2520all%252052%2520rows%2520in%2520board%2520order%252C%2520one%2520column%252C%2520three%2520different%2520units%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522stacked%2522%253Atrue%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522stacked%2522%253Atrue%252C%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A750%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522value%2520as%2520listed%2520%2528no%2520common%2520unit%2529%2522%257D%257D%255D%257D%257D%257D" alt="WildClawBench board rows in board order, colored by what they measure: total minutes, score, and total cost" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What does the WildClawBench board on Hugging Face rank?
&lt;/h2&gt;

&lt;p&gt;WildClawBench is a benchmark of 60 hand-built agent tasks in six categories (productivity, code, social interaction, search, creative synthesis and safety). Each task runs in its own Docker container inside an agent harness, and each metric is scored from 0.00 to 1.00. The README says judge-based metrics use an LLM judge, defaulting to &lt;code&gt;openai/gpt-5.4&lt;/code&gt;. The reported "overall" score follows a weighted multimodal and pure-text breakdown, so it is not a count of passed tasks.&lt;/p&gt;

&lt;p&gt;The dataset page carries a leaderboard built the usual way: model repos publish a YAML file under &lt;code&gt;.eval_results/&lt;/code&gt; and the board collects them. The board JSON is public at &lt;code&gt;/api/datasets/internlm/WildClawBench/leaderboard&lt;/code&gt;. This dataset's &lt;code&gt;eval.yaml&lt;/code&gt; declares three tasks, not one:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;code&gt;task_id&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Name in &lt;code&gt;eval.yaml&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;Unit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;overall&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Overall&lt;/td&gt;
&lt;td&gt;score, percent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;avg_time&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Avg Time (min)&lt;/td&gt;
&lt;td&gt;minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;avg_cost&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Avg Cost ($)&lt;/td&gt;
&lt;td&gt;US dollars&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each model repo's YAML has up to three entries, one per &lt;code&gt;task_id&lt;/code&gt;. The board JSON has no &lt;code&gt;task_id&lt;/code&gt; field. Each row carries a &lt;code&gt;value&lt;/code&gt;, a rank and a &lt;code&gt;lower_is_better&lt;/code&gt; flag, and the rows are sorted by value, descending.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is rank 1 a runtime and not a model?
&lt;/h2&gt;

&lt;p&gt;Because the sort does not know the units. Minutes for a 60-task suite run into the hundreds and dollars into the tens, while scores top out near 60, so a descending sort sorts by unit first. The first 16 ranks are minutes, then the 24 scores arrive with two dollar rows (ranks 34 and 37) interleaved, and the rest is dollars.&lt;/p&gt;

&lt;p&gt;The same model shows up three times. Kimi-K3 is at rank 9 (488 minutes), rank 19 (54.5 percent) and rank 34 ($40.08). Qwen3.8-2.4T-A95B is rank 1 for 708 minutes and rank 18 for its 56.2 score. A reader who scans for the top of the column sees a list of the slowest runs.&lt;/p&gt;

&lt;p&gt;There is also a direction problem. Time and cost are lower-is-better, and every row says &lt;code&gt;lower_is_better: false&lt;/code&gt;. The cheapest runs on the board, Kimi-K2.5 and Step-3.5-Flash at $6.60 each, are ranks 51 and 52, the bottom of the column, for being cheap.&lt;/p&gt;

&lt;p&gt;I also tried the obvious workaround. Adding &lt;code&gt;?task_id=avg_time&lt;/code&gt; or &lt;code&gt;?task=overall&lt;/code&gt; to the board endpoint returns the same 52 rows, with 708 first. The dataset page embeds the same list in the props of its leaderboard component, also starting with 708, and I found no task selector in those props. I did not render the page in a browser, so I cannot say what a visitor sees beyond that.&lt;/p&gt;

&lt;h2&gt;
  
  
  How can you check which unit each row is in?
&lt;/h2&gt;

&lt;p&gt;Read each row's source file and match the row's value to the entry whose &lt;code&gt;task_id&lt;/code&gt; it came from. All 52 values resolved to exactly one entry. This reproduces every number in this post:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;yaml&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"User-Agent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Mozilla/5.0"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"utf-8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="no"&gt;DS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"internlm/WildClawBench"&lt;/span&gt;
&lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"https://huggingface.co/api/datasets/{DS}/leaderboard"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"rows;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"lower_is_better"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"marked lower-is-better"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;declared&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;yaml&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;safe_load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"https://huggingface.co/datasets/{DS}/raw/main/eval.yaml"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"tasks"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"declared tasks:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;declared&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"modelId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pullRequest"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"filename"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;ref&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"refs%2Fpr%2F{key[1]}"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"main"&lt;/span&gt;
        &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;yaml&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;safe_load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"https://huggingface.co/{key[0]}/raw/{ref}/{key[2]}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;hit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"dataset"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"task_id"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"rank"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"modelId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"ambiguous"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rows per task_id:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ranks 1-16 by task_id:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rank 1:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rank 52:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;by_model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;rank&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;by_model&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setdefault&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rank&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Kimi-K3 ranks by task_id:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;by_model&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"moonshotai/Kimi-K3"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;per_task_min&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;per_task_usd&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"zai-org/GLM-5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;6.22&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.19&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"MiniMaxAI/MiniMax-M2.7"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;9.18&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"avg_time"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"avg_cost"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"board:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"min,"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"USD | README per-task x 60:"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;per_task_min&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"min,"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;per_task_usd&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"USD"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On November 11 it prints:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;52 rows; 0 marked lower-is-better
declared tasks: {'overall': 'Overall', 'avg_time': 'Avg Time (min)', 'avg_cost': 'Avg Cost ($)'}
rows per task_id: {'avg_time': 16, 'overall': 24, 'avg_cost': 12}
ranks 1-16 by task_id: {'avg_time': 16}
rank 1: (1, 'Qwen/Qwen3.8-2.4T-A95B', 708, 'avg_time')
rank 52: (52, 'stepfun-ai/Step-3.5-Flash', 6.6, 'avg_cost')
Kimi-K3 ranks by task_id: {'avg_time': [9], 'overall': [19], 'avg_cost': [34]}
zai-org/GLM-5 board: 373 min, 11.4 USD | README per-task x 60: 373 min, 11.4 USD
MiniMaxAI/MiniMax-M2.7 board: 551 min, 7.2 USD | README per-task x 60: 551 min, 7.2 USD
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(The last two lines use per-task averages from the harness table in the dataset README. It needs &lt;code&gt;pip install pyyaml&lt;/code&gt;.)&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Kimi-K3%2522%252C%2522GLM-5.2%2522%252C%2522GLM-5.1%2522%252C%2522DeepSeek-V4-Pro%2522%252C%2522Qwen3.6-27B%2522%252C%2522MiMo-V2.5-Pro%2522%252C%2522GLM-5%2522%252C%2522Qwen3.5-397B%2522%252C%2522MiniMax-M2.7%2522%252C%2522Kimi-K2.5%2522%252C%2522MiniMax-M2.5%2522%252C%2522Step-3.5-Flash%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rank%2520of%2520its%2520total-minutes%2520row%2522%252C%2522data%2522%253A%255B9%252C12%252C8%252C3%252C14%252C11%252C16%252C10%252C4%252C15%252C5%252C13%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522rank%2520of%2520its%2520score%2520row%2522%252C%2522data%2522%253A%255B19%252C20%252C24%252C29%252C30%252C31%252C32%252C38%252C39%252C40%252C41%252C42%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%252C%257B%2522label%2522%253A%2522rank%2520of%2520its%2520total-cost%2520row%2522%252C%2522data%2522%253A%255B34%252C45%252C37%252C47%252C44%252C46%252C48%252C43%252C50%252C51%252C49%252C52%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522One%2520model%252C%2520three%2520board%2520ranks%2520%252812%2520models%2520with%2520all%2520three%2520rows%253B%2520lower%2520bar%2520%253D%2520higher%2520on%2520the%2520board%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%252C%2522maxRotation%2522%253A45%252C%2522minRotation%2522%253A45%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A55%252C%2522reverse%2522%253Atrue%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2520%25281%2520at%2520top%2529%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Kimi-K3%2522%252C%2522GLM-5.2%2522%252C%2522GLM-5.1%2522%252C%2522DeepSeek-V4-Pro%2522%252C%2522Qwen3.6-27B%2522%252C%2522MiMo-V2.5-Pro%2522%252C%2522GLM-5%2522%252C%2522Qwen3.5-397B%2522%252C%2522MiniMax-M2.7%2522%252C%2522Kimi-K2.5%2522%252C%2522MiniMax-M2.5%2522%252C%2522Step-3.5-Flash%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rank%2520of%2520its%2520total-minutes%2520row%2522%252C%2522data%2522%253A%255B9%252C12%252C8%252C3%252C14%252C11%252C16%252C10%252C4%252C15%252C5%252C13%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522rank%2520of%2520its%2520score%2520row%2522%252C%2522data%2522%253A%255B19%252C20%252C24%252C29%252C30%252C31%252C32%252C38%252C39%252C40%252C41%252C42%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%252C%257B%2522label%2522%253A%2522rank%2520of%2520its%2520total-cost%2520row%2522%252C%2522data%2522%253A%255B34%252C45%252C37%252C47%252C44%252C46%252C48%252C43%252C50%252C51%252C49%252C52%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522One%2520model%252C%2520three%2520board%2520ranks%2520%252812%2520models%2520with%2520all%2520three%2520rows%253B%2520lower%2520bar%2520%253D%2520higher%2520on%2520the%2520board%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%252C%2522maxRotation%2522%253A45%252C%2522minRotation%2522%253A45%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A55%252C%2522reverse%2522%253Atrue%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2520%25281%2520at%2520top%2529%2522%257D%257D%255D%257D%257D%257D" alt="One model, three board ranks for the 12 models that have a score, a minutes and a cost row" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Are the numbers on the board correct?
&lt;/h2&gt;

&lt;p&gt;Yes, as far as I can check, and that is what makes this a presentation problem rather than a data problem. The dataset README has its own model table for the OpenClaw harness. Fifteen of its 31 models also have rows on the board (the other 16 are mostly closed models, plus a few open ones without a row). For 14 of those 15, every value the board carries equals the README's score, minutes or dollars. Two small catches:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The label says average, the values are totals.&lt;/strong&gt; &lt;code&gt;eval.yaml&lt;/code&gt; calls the tasks "Avg Time (min)" and "Avg Cost ($)", but the board's 373 minutes for GLM-5 is the README's 6.22 minutes per task multiplied by 60, and its $11.40 is $0.19 multiplied by 60. The README states this ("multiplied by 60 for the full 60-task suite"). A reader who trusts the label reads 708 as an average of 11.8 hours per task. It is the total, 11.8 hours across the suite, or about 11.8 minutes per task.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One score does not match.&lt;/strong&gt; Hy3 is listed at 53.6 from its model card, while the README's OpenClaw table has 49.7. That is 3.9 points between two figures for the same model, and the board does not say what harness the card used.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two other README values are absent from the board: Kimi-K2.7-Code has a total cost of $72.31 there but no cost row on the board, and four models (Qwen3.8-2.4T-A95B, Kimi-K2.7-Code, Intern-S2-Preview-397B and Qwen3.8-27B) have a score and minutes but no dollar row. Only 12 of the 23 models have all three rows.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does each column look like when read on its own?
&lt;/h2&gt;

&lt;p&gt;Split by &lt;code&gt;task_id&lt;/code&gt;, each metric has a sensible ordering. The score column, highest first, starts with dots3-note-prev at 61.7, then Qwen3.8-2.4T-A95B at 56.2, Kimi-K3 at 54.5, GLM-5.2 at 54.2 and Hy3 at 53.6.&lt;/p&gt;

&lt;p&gt;Where the scores come from matters. 18 of the 24 score rows come from a source named "WildClawBench", the benchmark team's own site, and the other 6 from model cards (dots3-note-prev, Hy3, Moderato-V1-Pro, Muse-Glimmer-30B and two openPangu-2.0-Flash rows). The top of the score column, 61.7, is a model-card figure. The highest row with the benchmark team as its source is 56.2. Two of the 24 score rows are the same model: openPangu-2.0-Flash appears at 41.5 and 35, and only the &lt;code&gt;notes&lt;/code&gt; field ("Avg@3, Thinking" and "Avg@3, Non-Thinking") tells them apart. Those two are the only rows with notes on the whole board.&lt;/p&gt;

&lt;p&gt;Provenance is also thin. All 52 rows are &lt;code&gt;verified: false&lt;/code&gt;. 50 of them come from 21 pull requests that are still open on the model repos, and 2 (Nex-N2-Pro and Moderato-V1-Pro) from a main branch. The board JSON has no date, but the YAML files do: 29 rows are dated May 22, 2026, 21 are dated between July 1 and August 16, and 2 have none.&lt;/p&gt;

&lt;h3&gt;
  
  
  How big is a gap on a 60-task suite?
&lt;/h3&gt;

&lt;p&gt;Smaller than the ranking suggests. Scores are weighted rubric means, not pass counts, so I cannot convert them to tasks. As a scale, if the 60 tasks carried equal weight, one task would be 1.67 points, and the README describes the overall as a weighted multimodal and pure-text breakdown, so treat that as a rough unit rather than an exact one. Among the 24 score rows, 13 of the 23 steps between neighbours are 1.0 point or less: Kimi-K3 to GLM-5.2 is 0.3, Hy3 to Nex-N2-Pro is 0.1, and GLM-5.1 (48.2) to Qwen3.8-27B (48.0152) is 0.18. The board lists no run-to-run spread and no per-task data, so nothing on it shows whether a 0.3 point gap is real. The dataset has a separate trajectories repo, which is where per-task evidence would have to come from.&lt;/p&gt;

&lt;p&gt;Qwen3.8-27B's 48.0152 is also the only value on the board with more than two decimals.&lt;/p&gt;

&lt;h3&gt;
  
  
  What does cost add?
&lt;/h3&gt;

&lt;p&gt;Cost and time answer different questions, and the board's column cannot show it. Across the 12 models with all three rows, the rank correlation between score and total cost is 0.82, and between score and total time only 0.16. A slow run is not a better run, and a better run is usually a pricier one, with exceptions.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522scatter%2522%252C%2522data%2522%253A%257B%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522open-weight%2520rows%2520with%2520score%252C%2520minutes%2520and%2520cost%2522%252C%2522data%2522%253A%255B%257B%2522x%2522%253A40.08%252C%2522y%2522%253A54.5%257D%252C%257B%2522x%2522%253A17.1%252C%2522y%2522%253A54.2%257D%252C%257B%2522x%2522%253A34.8%252C%2522y%2522%253A48.2%257D%252C%257B%2522x%2522%253A12%252C%2522y%2522%253A43.7%257D%252C%257B%2522x%2522%253A20.91%252C%2522y%2522%253A43.2%257D%252C%257B%2522x%2522%253A12.6%252C%2522y%2522%253A43%257D%252C%257B%2522x%2522%253A11.4%252C%2522y%2522%253A42.6%257D%252C%257B%2522x%2522%253A22.2%252C%2522y%2522%253A34.5%257D%252C%257B%2522x%2522%253A7.2%252C%2522y%2522%253A33.8%257D%252C%257B%2522x%2522%253A6.6%252C%2522y%2522%253A30.8%257D%252C%257B%2522x%2522%253A9.6%252C%2522y%2522%253A27.1%257D%252C%257B%2522x%2522%253A6.6%252C%2522y%2522%253A26.7%257D%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%252C%2522pointRadius%2522%253A6%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Read%2520one%2520metric%2520at%2520a%2520time%253A%2520score%2520vs%2520total%2520cost%2520for%2520the%252012%2520models%2520with%2520all%2520three%2520rows%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522plugins%2522%253A%257B%2522datalabels%2522%253A%257B%2522display%2522%253Atrue%252C%2522align%2522%253A%2528ctx%2529%253D%253E%2528%257B%2522GLM-5%2522%253A%2520%2522left%2522%252C%2520%2522DeepSeek-V4-Pro%2522%253A%2520%2522top%2522%252C%2520%2522MiMo-V2.5-Pro%2522%253A%2520%2522bottom%2522%252C%2520%2522Step-3.5-Flash%2522%253A%2520%2522left%2522%257D%2529%255B%255B%2522Kimi-K3%2522%252C%2520%2522GLM-5.2%2522%252C%2520%2522GLM-5.1%2522%252C%2520%2522DeepSeek-V4-Pro%2522%252C%2520%2522Qwen3.6-27B%2522%252C%2520%2522MiMo-V2.5-Pro%2522%252C%2520%2522GLM-5%2522%252C%2520%2522Qwen3.5-397B%2522%252C%2520%2522MiniMax-M2.7%2522%252C%2520%2522Kimi-K2.5%2522%252C%2520%2522MiniMax-M2.5%2522%252C%2520%2522Step-3.5-Flash%2522%255D%255Bctx.dataIndex%255D%255D%257C%257C%2527right%2527%252C%2522font%2522%253A%257B%2522size%2522%253A10%257D%252C%2522formatter%2522%253A%2528v%252Cctx%2529%253D%253E%255B%2522Kimi-K3%2522%252C%2520%2522GLM-5.2%2522%252C%2520%2522GLM-5.1%2522%252C%2520%2522DeepSeek-V4-Pro%2522%252C%2520%2522Qwen3.6-27B%2522%252C%2520%2522MiMo-V2.5-Pro%2522%252C%2520%2522GLM-5%2522%252C%2520%2522Qwen3.5-397B%2522%252C%2520%2522MiniMax-M2.7%2522%252C%2520%2522Kimi-K2.5%2522%252C%2520%2522MiniMax-M2.5%2522%252C%2520%2522Step-3.5-Flash%2522%255D%255Bctx.dataIndex%255D%257D%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A48%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522total%2520cost%2520for%252060%2520tasks%252C%2520USD%2520%2528listed%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A20%252C%2522max%2522%253A60%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522overall%2520score%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522scatter%2522%252C%2522data%2522%253A%257B%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522open-weight%2520rows%2520with%2520score%252C%2520minutes%2520and%2520cost%2522%252C%2522data%2522%253A%255B%257B%2522x%2522%253A40.08%252C%2522y%2522%253A54.5%257D%252C%257B%2522x%2522%253A17.1%252C%2522y%2522%253A54.2%257D%252C%257B%2522x%2522%253A34.8%252C%2522y%2522%253A48.2%257D%252C%257B%2522x%2522%253A12%252C%2522y%2522%253A43.7%257D%252C%257B%2522x%2522%253A20.91%252C%2522y%2522%253A43.2%257D%252C%257B%2522x%2522%253A12.6%252C%2522y%2522%253A43%257D%252C%257B%2522x%2522%253A11.4%252C%2522y%2522%253A42.6%257D%252C%257B%2522x%2522%253A22.2%252C%2522y%2522%253A34.5%257D%252C%257B%2522x%2522%253A7.2%252C%2522y%2522%253A33.8%257D%252C%257B%2522x%2522%253A6.6%252C%2522y%2522%253A30.8%257D%252C%257B%2522x%2522%253A9.6%252C%2522y%2522%253A27.1%257D%252C%257B%2522x%2522%253A6.6%252C%2522y%2522%253A26.7%257D%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%252C%2522pointRadius%2522%253A6%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Read%2520one%2520metric%2520at%2520a%2520time%253A%2520score%2520vs%2520total%2520cost%2520for%2520the%252012%2520models%2520with%2520all%2520three%2520rows%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522plugins%2522%253A%257B%2522datalabels%2522%253A%257B%2522display%2522%253Atrue%252C%2522align%2522%253A%2528ctx%2529%253D%253E%2528%257B%2522GLM-5%2522%253A%2520%2522left%2522%252C%2520%2522DeepSeek-V4-Pro%2522%253A%2520%2522top%2522%252C%2520%2522MiMo-V2.5-Pro%2522%253A%2520%2522bottom%2522%252C%2520%2522Step-3.5-Flash%2522%253A%2520%2522left%2522%257D%2529%255B%255B%2522Kimi-K3%2522%252C%2520%2522GLM-5.2%2522%252C%2520%2522GLM-5.1%2522%252C%2520%2522DeepSeek-V4-Pro%2522%252C%2520%2522Qwen3.6-27B%2522%252C%2520%2522MiMo-V2.5-Pro%2522%252C%2520%2522GLM-5%2522%252C%2520%2522Qwen3.5-397B%2522%252C%2520%2522MiniMax-M2.7%2522%252C%2520%2522Kimi-K2.5%2522%252C%2520%2522MiniMax-M2.5%2522%252C%2520%2522Step-3.5-Flash%2522%255D%255Bctx.dataIndex%255D%255D%257C%257C%2527right%2527%252C%2522font%2522%253A%257B%2522size%2522%253A10%257D%252C%2522formatter%2522%253A%2528v%252Cctx%2529%253D%253E%255B%2522Kimi-K3%2522%252C%2520%2522GLM-5.2%2522%252C%2520%2522GLM-5.1%2522%252C%2520%2522DeepSeek-V4-Pro%2522%252C%2520%2522Qwen3.6-27B%2522%252C%2520%2522MiMo-V2.5-Pro%2522%252C%2520%2522GLM-5%2522%252C%2520%2522Qwen3.5-397B%2522%252C%2520%2522MiniMax-M2.7%2522%252C%2520%2522Kimi-K2.5%2522%252C%2520%2522MiniMax-M2.5%2522%252C%2520%2522Step-3.5-Flash%2522%255D%255Bctx.dataIndex%255D%257D%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A48%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522total%2520cost%2520for%252060%2520tasks%252C%2520USD%2520%2528listed%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A20%252C%2522max%2522%253A60%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522overall%2520score%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" alt="Score against total cost for the 12 models that have all three rows" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;GLM-5.2 is the clearest exception: 54.2 against Kimi-K3's 54.5, at $17.10 against $40.08 and 442 against 488 minutes. A 0.3 point score gap is inside any plausible noise on 60 tasks, while the cost gap is 2.3 times. Cost here is tokens multiplied by prices at evaluation time, so it mixes how many tokens a model burns with what its provider charges. The README documents the rates it used for only two models, Claude Opus 4.8 and Kimi-K2.7-Code, so I would treat the dollar column as indicative.&lt;/p&gt;

&lt;h2&gt;
  
  
  Does the harness change the order?
&lt;/h2&gt;

&lt;p&gt;It does, and the board does not say which harness a row used. The README says its model leaderboard runs on the OpenClaw harness, so the rows sourced from the benchmark team's site are presumably OpenClaw. The 6 model-card rows state none. The README's harness table runs four models on four scaffolds (OpenClaw, Claude Code, Codex and Hermes Agent) with the same 60 tasks and the same grading:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Lowest score&lt;/th&gt;
&lt;th&gt;Highest score&lt;/th&gt;
&lt;th&gt;Spread&lt;/th&gt;
&lt;th&gt;Best harness&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;GPT-5.4&lt;/td&gt;
&lt;td&gt;48.4&lt;/td&gt;
&lt;td&gt;56.8&lt;/td&gt;
&lt;td&gt;8.4&lt;/td&gt;
&lt;td&gt;Codex&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM 5&lt;/td&gt;
&lt;td&gt;31.0&lt;/td&gt;
&lt;td&gt;46.4&lt;/td&gt;
&lt;td&gt;15.4&lt;/td&gt;
&lt;td&gt;Hermes Agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MiMo V2 Pro&lt;/td&gt;
&lt;td&gt;29.9&lt;/td&gt;
&lt;td&gt;48.1&lt;/td&gt;
&lt;td&gt;18.2&lt;/td&gt;
&lt;td&gt;Hermes Agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MiniMax M2.7&lt;/td&gt;
&lt;td&gt;32.0&lt;/td&gt;
&lt;td&gt;37.1&lt;/td&gt;
&lt;td&gt;5.1&lt;/td&gt;
&lt;td&gt;Hermes Agent&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On OpenClaw, GLM 5 (42.6) is ahead of MiMo V2 Pro (40.2). On Hermes Agent the order flips: 48.1 against 46.4. A spread of 5 to 18 points from the scaffold alone is bigger than almost every gap on the score column, so a model-card figure and a benchmark-team figure are not the same experiment unless the harness is given.&lt;/p&gt;

&lt;h2&gt;
  
  
  What else on the board is worth knowing?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ties get different ranks.&lt;/strong&gt; Kimi-K2.5 and Step-3.5-Flash are both $6.60 and hold ranks 51 and 52.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Closed models are absent.&lt;/strong&gt; That is expected for a board built from model-repo YAML files, but it means the README's top 7 (all closed) are missing, and the best score on the board (61.7) is not the best on the benchmark (67.2 in the README).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No row has a task field.&lt;/strong&gt; The page's embedded data has the same keys as the API (rank, value, source, notes, &lt;code&gt;lower_is_better&lt;/code&gt; and a few more), so the unit has to be inferred from the source file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Some rows are old.&lt;/strong&gt; 29 of the 52 rows are dated May 22, 2026, and the newest is dated August 16, so the board is a snapshot of two or three batches, not a live view.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What can a reader do with this board?
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Check the unit before the rank.&lt;/strong&gt; If a "score" column has values above 100, it is not a score.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Split by &lt;code&gt;task_id&lt;/code&gt;.&lt;/strong&gt; Fetch the YAML, or use the code above, and sort each metric in its own direction.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat time and cost as lower-is-better.&lt;/strong&gt; The board's flag says otherwise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look for the harness.&lt;/strong&gt; Without one, a card figure and a table figure can differ by 5 to 18 points for reasons unrelated to the model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not over-read sub-1-point gaps&lt;/strong&gt; on a 60-task suite with a judge-graded score.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For the board, three changes would remove most of the confusion: group rows by &lt;code&gt;task_id&lt;/code&gt;, carry the direction (&lt;code&gt;lower_is_better&lt;/code&gt;) per task, and show the harness. The YAML already carries the &lt;code&gt;task_id&lt;/code&gt;, so the information exists in the files and is dropped when the board is assembled.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is the WildClawBench leaderboard on Hugging Face?&lt;/strong&gt;&lt;br&gt;
It is the board on the dataset page for &lt;code&gt;internlm/WildClawBench&lt;/code&gt;, a suite of 60 agent tasks. Model repos publish YAML result files and the board collects them. As of November 11, 2026 it has 52 rows from 23 models.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why is rank 1 on the WildClawBench board a time of 708?&lt;/strong&gt;&lt;br&gt;
Because the board mixes three tasks (overall score, total minutes and total cost) in one descending column. 708 is the total minutes of the slowest run, not a score, and all 16 time rows rank above all 24 score rows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are the WildClawBench numbers on Hugging Face wrong?&lt;/strong&gt;&lt;br&gt;
No. For 14 of the 15 models that also appear in the benchmark's own README table, every value matches. The ordering and the missing direction flag are the problem, and so is one model-card score (Hy3) that is 3.9 points above the README's.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is "Avg Time" on the board an average per task?&lt;/strong&gt;&lt;br&gt;
The label says average, but the values are totals for the 60-task suite. The README multiplies the per-task averages by 60, and the board's 373 minutes for GLM-5 equals 6.22 minutes per task times 60.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How should I compare models on WildClawBench?&lt;/strong&gt;&lt;br&gt;
Compare scores only within one harness, ignore gaps under about 1 point, and read cost and time as lower-is-better in their own columns. The benchmark's README table gives the harness and the per-task averages, and for the open-weight models it matches the board.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/olmocr-bench-on-hugging-face-three-rows-use-a-different-average-and-one-moves-from-13th-to-7th-or-5gih"&gt;olmOCR-bench on Hugging Face: Three Rows Use a Different Average, and One Moves From 13th to 7th or 19th&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/open-asr-leaderboard-on-hugging-face-ranks-1-and-2-swap-on-a-common-mean-and-8-of-65-rows-skip-453n"&gt;Open ASR Leaderboard on Hugging Face: Ranks 1 and 2 Swap on a Common Mean, and 8 of 65 Rows Skip TED-LIUM&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>benchmark</category>
      <category>agents</category>
    </item>
    <item>
      <title>SWE-bench Multilingual on Hugging Face: 15 of 34 Scores Are Not a Count Out of 300 Tasks</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Mon, 09 Nov 2026 13:47:14 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-multilingual-on-hugging-face-15-of-34-scores-are-not-a-count-out-of-300-tasks-2ig7</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-multilingual-on-hugging-face-15-of-34-scores-are-not-a-count-out-of-300-tasks-2ig7</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On November 9, 2026 the Hugging Face SWE-bench Multilingual board lists &lt;strong&gt;34 rows from 34 repos, none verified.&lt;/strong&gt; 30 rows are read from a pull request that is still open (all 30 opened by one of two accounts) and 4 from a repo's main branch. 32 rows cite a model card and 2 cite a paper.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The benchmark has exactly 300 tasks, so a single run can only score a multiple of 0.333 points. 15 of the 34 listed values are not one&lt;/strong&gt;, including ranks 1 and 2 (88.5 and 82.9). They may be averages over several runs or use a different denominator. The files do not say which.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two models also appear on the benchmark's official table, where every row ran the same open agent.&lt;/strong&gt; GLM-5 is listed at 73.3 on the board and 69.7 officially (220 against 209 tasks). Kimi-K2.5 is 73.0 against 67.3 (219 against 202). The board names a harness for 2 of its 34 rows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On the official table's per-task data, the top two rows are 2 tasks apart (218 and 216 of 300), and the paired standard error of that gap is 2.45 points.&lt;/strong&gt; A 7 point spread across ranks 2 to 12 on the board is about 21 tasks, which is a tier more than a ranking.&lt;/li&gt;
&lt;li&gt;The board pools nine languages that have between 12 and 44 tasks each. On the official table five different rows lead or tie for the lead in at least one language.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522value%2522%252C%2522data%2522%253A%255B88.5%252C82.9%252C81.67%252C81%252C79.6%252C78.9%252C78.3%252C77.3%252C76.7%252C76.5%252C76.2%252C75.8%252C75.7%252C73.3%252C73.3%252C73%252C72.4%252C71.7%252C71.4%252C71.3%252C69.3%252C69.3%252C67.72%252C67.7%252C67.2%252C64.3%252C63.1%252C57.7%252C54.4%252C52%252C47.3%252C45.8%252C39.33%252C36.47%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522SWE-bench%2520Multilingual%2520board%252C%2520Nov%25209%252C%25202026%253A%2520red%2520%253D%252015%2520of%252034%2520values%2520cannot%2520be%2520a%2520pass%2520count%2520out%2520of%2520300%2520tasks%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520resolved%2520as%2520listed%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522value%2522%252C%2522data%2522%253A%255B88.5%252C82.9%252C81.67%252C81%252C79.6%252C78.9%252C78.3%252C77.3%252C76.7%252C76.5%252C76.2%252C75.8%252C75.7%252C73.3%252C73.3%252C73%252C72.4%252C71.7%252C71.4%252C71.3%252C69.3%252C69.3%252C67.72%252C67.7%252C67.2%252C64.3%252C63.1%252C57.7%252C54.4%252C52%252C47.3%252C45.8%252C39.33%252C36.47%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522SWE-bench%2520Multilingual%2520board%252C%2520Nov%25209%252C%25202026%253A%2520red%2520%253D%252015%2520of%252034%2520values%2520cannot%2520be%2520a%2520pass%2520count%2520out%2520of%2520300%2520tasks%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520resolved%2520as%2520listed%2522%257D%257D%255D%257D%257D%257D" alt="SWE-bench Multilingual board listed values with the 15 that cannot be a count of 300 tasks in red" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What does the SWE-bench Multilingual board on Hugging Face rank?
&lt;/h2&gt;

&lt;p&gt;SWE-bench Multilingual is a set of 300 real GitHub issue and pull request pairs from 41 repositories in 9 languages. A system gets the repository at the commit before the fix and the issue text, produces a patch, and the task counts as resolved if the repository's own tests pass afterwards. The score is the share of the 300 tasks resolved.&lt;/p&gt;

&lt;p&gt;The Hugging Face page for the dataset carries a leaderboard assembled from files that model repos publish themselves: a YAML file under &lt;code&gt;.eval_results/&lt;/code&gt; that gives a value, a source link and, optionally, a date and notes. The board JSON is public at &lt;code&gt;/api/datasets/SWE-bench/SWE-bench_Multilingual/leaderboard&lt;/code&gt;, so anyone can pull the same rows I used. All values are higher-is-better percentages. Every row has &lt;code&gt;verified: false&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Where the rows come from is worth a sentence. 30 of the 34 sit on pull requests that I checked through the discussions API and that are all still open. Two accounts opened all 30, 23 by one and 7 by the other. The other 4 rows (ranks 7, 14, 17 and 34) are on main branches. The source is a model card for 32 rows and a paper table for the other two (ranks 3 and 26). Six of the 34 rows come from one organization, five from a second and five from a third, so three organizations hold 16 of the 34 rows, on a board with 15 organizations.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can a score on a 300-task benchmark be 88.5?
&lt;/h2&gt;

&lt;p&gt;Only if it is not a single run. Each task is pass or fail, so one run produces k/300 for a whole number k, and k/300 expressed as a percentage moves in steps of 0.333. A value reported to one decimal place can come from a single run only if some whole k rounds to it.&lt;/p&gt;

&lt;p&gt;Here is the check, along with the official comparison used later in this post:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="n"&gt;N&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt;  &lt;span class="c1"&gt;# tasks in SWE-bench Multilingual
&lt;/span&gt;&lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/datasets/SWE-bench/SWE-bench_Multilingual/leaderboard&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;counts_that_round_to&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;dec&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;repr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;  &lt;span class="c1"&gt;# 81 is stored as 81.0, one decimal
&lt;/span&gt;    &lt;span class="n"&gt;half&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;dec&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;N&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;N&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;half&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mf"&gt;1e-9&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rank&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;counts_that_round_to&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cannot be k/300 at their stated precision&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;LB&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;official&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;LB&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;leaderboards&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Multilingual&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;results&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;off&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;official&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;zai-org/GLM-5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;GLM 5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;moonshotai/Kimi-K2.5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Kimi K2.5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
    &lt;span class="n"&gt;hf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;counts_that_round_to&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hf&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: board &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;hf&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; (= &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/300), official &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;off&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;resolved&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; (= &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;off&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;resolved&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;N&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/300)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;off&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;per_instance_details&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Gemini 3 Flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Claude 4.6 Opus&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;only_a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resolved&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resolved&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;only_b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resolved&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resolved&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;se&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;only_a&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;only_b&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;only_a&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;only_b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;N&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;N&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;official 1 vs 2: only-first&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;only_a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;only-second&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;only_b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;paired SE&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;se&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;points&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On November 9 it prints:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;34 rows; 15 cannot be k/300 at their stated precision
   1 inclusionAI/Ling-3.1-flash 88.5
   2 tencent/Hy4-preview 82.9
   5 ornith-ai/Ornith-1.5-397B 79.6
   6 ornith-ai/Ornith-1.0-397B 78.9
   10 MiniMaxAI/MiniMax-M2.7 76.5
   11 deepseek-ai/DeepSeek-V4-Pro 76.2
   12 tencent/Hy3 75.8
   17 inclusionAI/Ling-3.0-flash 72.4
   19 ornith-ai/Ornith-1.5-35B-A3B 71.4
   23 nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 67.72
   25 Qwen/Qwen3.6-35B-A3B 67.2
   27 poolside/Laguna-XS-2.1 63.1
   29 ornith-ai/Ornith-1.5-9B 54.4
   32 nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 45.8
   34 nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 36.47
zai-org/GLM-5: board 73.3 (= 220/300), official 69.7 (= 209/300)
moonshotai/Kimi-K2.5: board 73 (= 219/300), official 67.3 (= 202/300)
official 1 vs 2: only-first 28 only-second 26 paired SE 2.45 points
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Seven of the top twelve ranks and four of the top six are on that list. A miss is informative and a hit is not: a one-decimal number lands on some k/300 about 3 times in 10 by luck, so the 19 rows that do fit prove nothing on their own. The 15 that do not fit cannot be one pass count.&lt;/p&gt;

&lt;p&gt;That does not make them wrong. A model card often reports the mean of several runs, and a mean of five runs steps in units of 0.067 points. A card may also use a different denominator, for instance after dropping tasks whose environment failed. Both are legitimate, and neither is visible in the file. The YAML has a &lt;code&gt;notes&lt;/code&gt; field and the two rows that fill it with harness details are ranks 4 (mini-SWE-agent, 256K context) and 26 (OpenHands). Nothing on the board tells a reader whether 88.5 is one run, an average, or something else, and that matters because a mean of several runs is usually a less noisy number than a single run.&lt;/p&gt;

&lt;h2&gt;
  
  
  How large is a gap of a few points on 300 tasks?
&lt;/h2&gt;

&lt;p&gt;Smaller than the ranking implies. For a single run at 245 of 300 (81.67, rank 3), an unpaired 95% Wilson interval runs from 76.9 to 85.6. The row at rank 13, 227 of 300 or 75.7, has an interval of 70.5 to 80.2. The two overlap by more than 3 points even though 10 ranks separate them.&lt;/p&gt;

&lt;p&gt;The two rows are different models scored on the same tasks, so an unpaired interval is the conservative choice. The official table lets me measure how much smaller the paired number is, because it publishes per-task results. For its top two rows, Gemini 3 Flash with 218 tasks and Claude 4.6 Opus with 216, 190 tasks are solved by both, 28 only by the first and 26 only by the second. The paired standard error of the 0.67 point difference is 2.45 points. A lead has to exceed roughly 5 points, about 15 tasks, before it clears 1.96 standard errors for two rows this similar. On the board, ranks 3 to 13 span 18 tasks.&lt;/p&gt;

&lt;p&gt;The same pattern holds one level down. Among all 13 official rows, 30 of the 300 tasks (10%) are solved by none and 69 (23%) by every row, so the rows differ on the other 201 tasks. A ranking built from rows that agree on a third of the benchmark is mostly reading disagreement on the rest.&lt;/p&gt;

&lt;h2&gt;
  
  
  Do the same models score the same on the official table?
&lt;/h2&gt;

&lt;p&gt;Two repos on the board also have rows on the benchmark's own table, which I read from the leaderboard data file in the benchmark's website repository. Every row there ran the same agent, mini-SWE-agent, and the file includes per-task outcomes. Its latest entry is dated February 20, 2026, so any new submission since then is not in it. It lists 13 rows, and 12 of them carry the maintainers' checked flag.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Board value&lt;/th&gt;
&lt;th&gt;Implied tasks&lt;/th&gt;
&lt;th&gt;Official value&lt;/th&gt;
&lt;th&gt;Official tasks&lt;/th&gt;
&lt;th&gt;Gap&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;GLM-5&lt;/td&gt;
&lt;td&gt;73.3&lt;/td&gt;
&lt;td&gt;220&lt;/td&gt;
&lt;td&gt;69.7&lt;/td&gt;
&lt;td&gt;209&lt;/td&gt;
&lt;td&gt;3.6 points, 11 tasks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kimi-K2.5&lt;/td&gt;
&lt;td&gt;73.0&lt;/td&gt;
&lt;td&gt;219&lt;/td&gt;
&lt;td&gt;67.3&lt;/td&gt;
&lt;td&gt;202&lt;/td&gt;
&lt;td&gt;5.7 points, 17 tasks&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The model-card figures are higher in both cases, and the board does not say what harness produced them. A different agent, a different tool budget or a different prompt can all move a score by several points, and a model card is free to report the setup that suits it best. None of that is an error. It means the board mixes numbers measured under unlike conditions and presents them in one column.&lt;/p&gt;

&lt;p&gt;The ordering is the more interesting part. Officially GLM-5 resolves 7 tasks more than Kimi-K2.5. On the board the same two rows are 1 task apart. The paired standard error between those two official rows is 2.47 points, so even the 7 task official gap is under one standard error. Neither table can say which is better. The other two open-weight rows on the official table, MiniMax M2.5 and DeepSeek V3.2, have no row on the board, which lists newer versions of both, so the overlap is two models.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522GLM-5%2522%252C%2522Kimi-K2.5%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522HF%2520board%2520%2528model-card%2520figure%252C%2520harness%2520not%2520stated%2529%2522%252C%2522data%2522%253A%255B73.3%252C73.0%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522official%2520table%2520%2528mini-SWE-agent%252C%2520per-task%2520data%2529%2522%252C%2522data%2522%253A%255B69.7%252C67.3%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Same%2520two%2520models%252C%2520two%2520tables%253A%2520%2525%2520resolved%2520on%2520the%2520same%2520300%2520tasks%2520%2528axis%2520starts%2520at%252050%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A50%252C%2522max%2522%253A80%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520resolved%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522GLM-5%2522%252C%2522Kimi-K2.5%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522HF%2520board%2520%2528model-card%2520figure%252C%2520harness%2520not%2520stated%2529%2522%252C%2522data%2522%253A%255B73.3%252C73.0%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522official%2520table%2520%2528mini-SWE-agent%252C%2520per-task%2520data%2529%2522%252C%2522data%2522%253A%255B69.7%252C67.3%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Same%2520two%2520models%252C%2520two%2520tables%253A%2520%2525%2520resolved%2520on%2520the%2520same%2520300%2520tasks%2520%2528axis%2520starts%2520at%252050%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A50%252C%2522max%2522%253A80%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520resolved%2522%257D%257D%255D%257D%257D%257D" alt="GLM-5 and Kimi-K2.5 listed on the board next to their official table scores" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why do nine languages matter for one pooled number?
&lt;/h2&gt;

&lt;p&gt;Because the languages do not carry equal weight, and each is a small sample. The dataset card lists the task counts: Ruby 44, Java 43, Rust 43, PHP 43, Go 42, C 30, JavaScript 26, TypeScript 17 and C++ 12. Five languages hold 215 of the 300 tasks, so the pooled score is mostly a Ruby, Java, Rust, PHP and Go score. C++ is 4 percent of the benchmark, with 11 of its 12 tasks coming from one repository.&lt;/p&gt;

&lt;p&gt;I mapped each of the 41 repos to the languages in the card, and I checked that the counts reproduce the card's table exactly. Then I computed per-language rates for every official row from its per-task data. The chart shows the top two.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Ruby%2520%252844%2529%2522%252C%2522Java%2520%252843%2529%2522%252C%2522Rust%2520%252843%2529%2522%252C%2522PHP%2520%252843%2529%2522%252C%2522Go%2520%252842%2529%2522%252C%2522C%2520%252830%2529%2522%252C%2522JavaScript%2520%252826%2529%2522%252C%2522TypeScript%2520%252817%2529%2522%252C%2522C%252B%252B%2520%252812%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522Gemini%25203%2520Flash%2520%2528218%252F300%2520overall%2529%2522%252C%2522data%2522%253A%255B68.2%252C81.4%252C81.4%252C69.8%252C61.9%252C76.7%252C69.2%252C58.8%252C91.7%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522Claude%25204.6%2520Opus%2520%2528216%252F300%2520overall%2529%2522%252C%2522data%2522%253A%255B65.9%252C72.1%252C81.4%252C76.7%252C54.8%252C73.3%252C92.3%252C70.6%252C58.3%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Official%2520table%2527s%2520top%2520two%2520rows%2520by%2520language%2520%2528tasks%2520per%2520language%2520in%2520brackets%2529%253A%2520the%2520leader%2520changes%2520with%2520the%2520language%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520resolved%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Ruby%2520%252844%2529%2522%252C%2522Java%2520%252843%2529%2522%252C%2522Rust%2520%252843%2529%2522%252C%2522PHP%2520%252843%2529%2522%252C%2522Go%2520%252842%2529%2522%252C%2522C%2520%252830%2529%2522%252C%2522JavaScript%2520%252826%2529%2522%252C%2522TypeScript%2520%252817%2529%2522%252C%2522C%252B%252B%2520%252812%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522Gemini%25203%2520Flash%2520%2528218%252F300%2520overall%2529%2522%252C%2522data%2522%253A%255B68.2%252C81.4%252C81.4%252C69.8%252C61.9%252C76.7%252C69.2%252C58.8%252C91.7%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522Claude%25204.6%2520Opus%2520%2528216%252F300%2520overall%2529%2522%252C%2522data%2522%253A%255B65.9%252C72.1%252C81.4%252C76.7%252C54.8%252C73.3%252C92.3%252C70.6%252C58.3%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Official%2520table%2527s%2520top%2520two%2520rows%2520by%2520language%2520%2528tasks%2520per%2520language%2520in%2520brackets%2529%253A%2520the%2520leader%2520changes%2520with%2520the%2520language%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520resolved%2522%257D%257D%255D%257D%257D%257D" alt="Official table top two rows compared language by language with task counts in brackets" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Claude 4.6 Opus is 92.3 on JavaScript (24 of 26) against 69.2 (18 of 26) for Gemini 3 Flash. On C++ the order flips: 91.7 (11 of 12) against 58.3 (7 of 12). Both are differences of 4 to 6 tasks, and each of them is larger than the 2 task gap that separates the two rows overall. Across all 13 official rows, five different rows lead or tie for the lead in at least one of the nine languages: Gemini 3 Flash, Claude 4.6 Opus, GLM 5, Kimi K2.5 and GPT 5.2 Codex. The overall leader leads or ties in 4 of the 9.&lt;/p&gt;

&lt;p&gt;None of that says the pooled number is wrong. It says that a user who writes Go or C++ and picks from a pooled ranking is reading a number dominated by other languages, and that the per-language cells are small enough that one task moves them by 2 to 8 points.&lt;/p&gt;

&lt;h2&gt;
  
  
  What else on the board is worth knowing?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ties get different ranks.&lt;/strong&gt; Ranks 14 and 15 are both 73.3, and ranks 21 and 22 are both 69.3. At 300 tasks that is the same count, 220 and 208.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The size column is not always right.&lt;/strong&gt; The board's &lt;code&gt;num_parameters&lt;/code&gt; for &lt;code&gt;ornith-ai/Ornith-1.0-35B&lt;/code&gt; is 664,944 and for &lt;code&gt;ornith-ai/Ornith-1.0-9B&lt;/code&gt; it is 1,469,680. The Hub's own safetensors metadata gives the same numbers for these two repos, which have 16 and 4 weight shards. Treat the size column as a hint.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dates are mostly missing.&lt;/strong&gt; The &lt;code&gt;date&lt;/code&gt; field is filled for 9 of 34 rows, so a reader cannot tell a February score from an October one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The top row is a different generation from the official table's top row.&lt;/strong&gt; Rank 1 at 88.5 sits 15.8 points above the official table's best of 72.7. The rows are different models, run by different parties with different agents, and the official table ends in February, so this is not a like for like gap.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What can a reader do with this board?
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Convert to task counts.&lt;/strong&gt; Multiply by 3 to get tasks out of 300. If the result is not close to a whole number, the row is an average or uses another denominator.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read gaps in tasks, not points.&lt;/strong&gt; Two rows 5 points apart are 15 tasks apart. For two models that agree on most tasks, the paired noise on that gap is around 2.5 points.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look for a harness.&lt;/strong&gt; Only 2 of 34 rows name one. Without it, a model-card figure and a table row are not the same experiment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cross-check with per-task data where it exists.&lt;/strong&gt; The official file publishes which tasks each row solved, and it is the only place to see where two models actually differ.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Weight languages by what you write.&lt;/strong&gt; If your code is in Go or C++, the pooled number tells you little about it, and C++ has 12 tasks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat 0 verified as 0 verified.&lt;/strong&gt; All 34 rows are self-reported, and all 30 pull request rows are still open.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For the board itself, three small changes would remove most of the ambiguity: show the number of runs and the denominator next to each value, show the harness, and show the date for every row.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is the SWE-bench Multilingual leaderboard on Hugging Face?&lt;/strong&gt;&lt;br&gt;
It is a board on the dataset page for SWE-bench Multilingual, which has 300 real issue and pull request tasks in 9 programming languages. Model repos publish YAML result files and the board lists them, sorted by percentage resolved.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How many tasks does SWE-bench Multilingual have, and why does it matter for scores?&lt;/strong&gt;&lt;br&gt;
300 tasks. A single run therefore gives a score in steps of 0.333 points, and a gap of 1 point is 3 tasks. A listed value that is not a multiple of that step is an average over runs or uses another denominator.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are the Hugging Face scores comparable with the official SWE-bench table?&lt;/strong&gt;&lt;br&gt;
Not directly. The official table runs every submission with the same open agent and publishes per-task results. The board takes figures from model cards, names a harness for 2 of 34 rows, and has no verified rows. For GLM-5 and Kimi-K2.5, the two models I could compare, the board values are 3.6 and 5.7 points higher.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does the top row on the board mean that model is the best at multilingual code repair?&lt;/strong&gt;&lt;br&gt;
It means the model's card reports the highest number. The top row's 88.5 cannot be a single count out of 300, and a lead of a few points between top rows is within the noise on 300 tasks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I check a score on a small benchmark myself?&lt;/strong&gt;&lt;br&gt;
Multiply the percentage by the number of tasks and see whether you get a whole number. Then, where per-task data exists, count the tasks only one of two models solved and use that count to size the noise on their gap. The code above does both.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-verified-on-hugging-face-9-self-reported-rows-top-the-official-tables-best-and-6-repos-3173"&gt;SWE-bench Verified on Hugging Face: 9 Self-Reported Rows Top the Official Table's Best, and 6 Repos Fill 16 of 78 Rows&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-pro-has-two-versions-what-the-hugging-face-leaderboards-47-rows-mix-together-37co"&gt;SWE-Bench Pro Has Two Versions: What the Hugging Face Leaderboard's 47 Rows Mix Together&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>benchmark</category>
      <category>llm</category>
    </item>
    <item>
      <title>olmOCR-bench on Hugging Face: Three Rows Use a Different Average, and One Moves From 13th to 7th or 19th</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Sun, 08 Nov 2026 13:47:17 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/olmocr-bench-on-hugging-face-three-rows-use-a-different-average-and-one-moves-from-13th-to-7th-or-5gih</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/olmocr-bench-on-hugging-face-three-rows-use-a-different-average-and-one-moves-from-13th-to-7th-or-5gih</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On November 8, 2026 the Hugging Face olmOCR-bench board lists &lt;strong&gt;24 rows from 24 repos, all higher-is-better pass rates, 0 verified.&lt;/strong&gt; 18 rows sit on a repo's main branch and 6 come from pull requests that are still open, all opened by one account.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The board's &lt;code&gt;overall&lt;/code&gt; is the plain average of eight columns (seven test categories plus a &lt;code&gt;baseline&lt;/code&gt; column) for 15 of the 24 rows.&lt;/strong&gt; Three rows, ranks 13, 20 and 23, use a different average that leaves out &lt;code&gt;headers_footers&lt;/code&gt;, and their notes say so. Six more rows cannot be rechecked from their files: one is gated, one stores only a mean, four store fewer than eight columns.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rank 13, LightOnOCR-2-1B, is listed at 83.2.&lt;/strong&gt; Its own eight columns average 75.24. Drop &lt;code&gt;headers_footers&lt;/code&gt; from everyone and it is 7th of 22. Keep it and the row is 19th of 21. The board's 13th is a mix of the two.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rank 1 survives every recomputation I tried, and rank 2 survives all but a test-count weighting.&lt;/strong&gt; Rank 1 leads 3 of the 8 columns, is 16th of 17 on &lt;code&gt;baseline&lt;/code&gt;, and among the rows I could read, six different repos lead the eight columns between them.&lt;/li&gt;
&lt;li&gt;Ranks 1 to 3 are 0.78 points apart. The dataset card's own results table puts a ± of 0.9 to 1.2 on every Overall it lists, so treat the top of this board as a tier.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2520OXR-1.0%2522%252C%25222%2520Infinity-Parser2-Pro%2522%252C%25224%2520LightOnOCR-3-4B%2522%252C%25228%2520dots.mocr%2522%252C%252214%2520chandra%2522%252C%252221%2520DeepSeek-OCR%2522%252C%252223%2520GLM-OCR%2522%252C%252213%2520LightOnOCR-2-1B%2522%252C%252220%2520LightOnOCR-1B-1025%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522board%2520overall%2520%2528as%2520listed%2529%2522%252C%2522data%2522%253A%255B87.84%252C87.6%252C86.3%252C83.9%252C83.1%252C75.7%252C75.2%252C83.2%252C76.1%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522mean%2520of%2520the%25207%2520test%2520categories%2520%2528headers_footers%2520kept%252C%2520baseline%2520left%2520out%2529%2522%252C%2522data%2522%253A%255B86.48%252C85.83%252C84.41%252C81.6%252C80.74%252C72.31%252C74.8%252C71.76%252C66.97%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522olmOCR-bench%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25208%252C%25202026%253A%2520listed%2520overall%2520vs%2520a%2520common%25207-category%2520mean%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A60%252C%2522max%2522%253A90%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522pass%2520rate%252C%2520%2525%2520%2528axis%2520starts%2520at%252060%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2520OXR-1.0%2522%252C%25222%2520Infinity-Parser2-Pro%2522%252C%25224%2520LightOnOCR-3-4B%2522%252C%25228%2520dots.mocr%2522%252C%252214%2520chandra%2522%252C%252221%2520DeepSeek-OCR%2522%252C%252223%2520GLM-OCR%2522%252C%252213%2520LightOnOCR-2-1B%2522%252C%252220%2520LightOnOCR-1B-1025%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522board%2520overall%2520%2528as%2520listed%2529%2522%252C%2522data%2522%253A%255B87.84%252C87.6%252C86.3%252C83.9%252C83.1%252C75.7%252C75.2%252C83.2%252C76.1%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522mean%2520of%2520the%25207%2520test%2520categories%2520%2528headers_footers%2520kept%252C%2520baseline%2520left%2520out%2529%2522%252C%2522data%2522%253A%255B86.48%252C85.83%252C84.41%252C81.6%252C80.74%252C72.31%252C74.8%252C71.76%252C66.97%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522olmOCR-bench%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25208%252C%25202026%253A%2520listed%2520overall%2520vs%2520a%2520common%25207-category%2520mean%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A60%252C%2522max%2522%253A90%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522pass%2520rate%252C%2520%2525%2520%2528axis%2520starts%2520at%252060%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" alt="olmOCR-bench board listed overall next to a common 7-category mean for 9 rows" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What does the olmOCR-bench board on Hugging Face rank?
&lt;/h2&gt;

&lt;p&gt;olmOCR-bench is a dataset of 1,403 PDF pages plus 7,010 unit tests. A test checks one property of an OCR system's markdown output: a sentence is present, a header is absent, two spans appear in the right order, a table cell sits next to its neighbour, a formula is rendered. A model's score is the share of tests it passes.&lt;/p&gt;

&lt;p&gt;The Hugging Face page for the dataset carries a leaderboard. It is assembled from files that model repos publish themselves: a small YAML file under &lt;code&gt;.eval_results/&lt;/code&gt;, read either from the repo's main branch or from an open pull request. The board JSON is public at &lt;code&gt;/api/datasets/allenai/olmOCR-bench/leaderboard&lt;/code&gt;, so anyone can pull the same 24 rows I used.&lt;/p&gt;

&lt;p&gt;The dataset card splits the 7,010 tests into seven categories: &lt;code&gt;arxiv_math&lt;/code&gt; (2,927 tests), &lt;code&gt;table_tests&lt;/code&gt; (1,020), &lt;code&gt;multi_column&lt;/code&gt; (884), &lt;code&gt;headers_footers&lt;/code&gt; (753), &lt;code&gt;old_scans&lt;/code&gt; (526), &lt;code&gt;old_scans_math&lt;/code&gt; (458) and &lt;code&gt;long_tiny_text&lt;/code&gt; (442). The row's headline number is stored under the task id &lt;code&gt;overall&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  How is the board's overall computed?
&lt;/h2&gt;

&lt;p&gt;The card's results table answers part of it. It lists a &lt;code&gt;Base&lt;/code&gt; column next to the seven categories, and its Overall is the unweighted mean of those eight numbers: for the first row, GOT OCR, (52.7 + 94.0 + 93.6 + 29.9 + 42.0 + 22.1 + 52.0 + 0.2) / 8 is 48.3, the listed Overall. The card's table of 7,010 tests has no &lt;code&gt;Base&lt;/code&gt; category, and the repo's &lt;code&gt;bench_data&lt;/code&gt; folder has seven test files. One board row's notes describe &lt;code&gt;baseline&lt;/code&gt; as an automatic category of 1,394 checks that is "not a transcription-accuracy metric".&lt;/p&gt;

&lt;p&gt;So the convention is the card's, and the question is whether each row on the board follows it. Most YAML files carry the seven categories plus an entry called &lt;code&gt;baseline&lt;/code&gt;, and averaging all eight entries without weights reproduces &lt;code&gt;overall&lt;/code&gt; to within 0.06 points for 15 rows. Here is the whole check:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;yaml&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;COLS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;arxiv_math&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;headers_footers&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;long_tiny_text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;multi_column&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;old_scans&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;old_scans_math&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;table_tests&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;baseline&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/datasets/allenai/olmOCR-bench/leaderboard&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;pr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pullRequest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ref&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;refs%2Fpr%2F&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;pr&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;pr&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;main&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/raw/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;filename&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;                                  &lt;span class="c1"&gt;# gated repos answer 401
&lt;/span&gt;    &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dataset&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;task_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;yaml&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;safe_load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;COLS&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;                                  &lt;span class="c1"&gt;# a column is missing: cannot recompute
&lt;/span&gt;    &lt;span class="n"&gt;mean8&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;COLS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mean8&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mf"&gt;0.06&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rank&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;board&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;8-column mean&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mean8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On November 8 it prints three lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;13 lightonai/LightOnOCR-2-1B board 83.2 8-column mean 75.24
20 lightonai/LightOnOCR-1B-1025 board 76.1 8-column mean 71.05
23 zai-org/GLM-OCR board 75.2 8-column mean 77.8
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An unweighted mean of categories is a choice. The same 7,010 tests weighted by test count would give &lt;code&gt;arxiv_math&lt;/code&gt; 42 percent of the score. Neither is wrong, but a reader comparing a row against its own breakdown needs to know which one is in use, and for these three rows it is not the card's.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which rows use a different overall?
&lt;/h2&gt;

&lt;p&gt;The three mismatches are not errors. Each file says what it did.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Board rank&lt;/th&gt;
&lt;th&gt;Repo&lt;/th&gt;
&lt;th&gt;Listed overall&lt;/th&gt;
&lt;th&gt;Mean of its 8 columns&lt;/th&gt;
&lt;th&gt;Mean without &lt;code&gt;headers_footers&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;Note in the file&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;13&lt;/td&gt;
&lt;td&gt;LightOnOCR-2-1B&lt;/td&gt;
&lt;td&gt;83.2&lt;/td&gt;
&lt;td&gt;75.24&lt;/td&gt;
&lt;td&gt;83.17&lt;/td&gt;
&lt;td&gt;H&amp;amp;F rewards omission, so it is excluded&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;LightOnOCR-1B-1025&lt;/td&gt;
&lt;td&gt;76.1&lt;/td&gt;
&lt;td&gt;71.05&lt;/td&gt;
&lt;td&gt;76.13&lt;/td&gt;
&lt;td&gt;Excluding Headers &amp;amp; Footers category&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;23&lt;/td&gt;
&lt;td&gt;GLM-OCR&lt;/td&gt;
&lt;td&gt;75.2&lt;/td&gt;
&lt;td&gt;77.80&lt;/td&gt;
&lt;td&gt;75.23&lt;/td&gt;
&lt;td&gt;Excluding Headers &amp;amp; Footers category, run through an API&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Dropping &lt;code&gt;headers_footers&lt;/code&gt; and averaging the other seven columns, &lt;code&gt;baseline&lt;/code&gt; included, reproduces all three listed numbers to within 0.04. These rows are correctly labelled in their own files. The board just sorts them next to rows that kept the category.&lt;/p&gt;

&lt;p&gt;The effect is large for the LightOn rows. Their &lt;code&gt;headers_footers&lt;/code&gt; scores are 19.7 and 35.5, against a median of 92.3 for the 20 rows that report the column. The listed 83.2 is 8.0 points above the same row's eight-column mean. GLM-OCR goes the other way: its &lt;code&gt;headers_footers&lt;/code&gt; score is 95.8, so leaving it out costs it 2.6 points.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522headers_footers%2522%252C%2522data%2522%253A%255B94.34%252C95.8%252C89.2%252C91.4%252C88.2%252C87.6%252C94.0%252C88.7%252C92.5%252C92.5%252C19.7%252C90.8%252C88.7%252C94.0%252C92.2%252C94.1%252C35.5%252C96.1%252C96.6%252C95.8%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522headers_footers%2520score%2520by%2520board%2520rank%2520%252820%2520rows%252C%2520Kraken%2520excluded%2529%253A%2520ranks%252013%2520and%252020%2520sit%2520far%2520below%2520the%2520rest%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522pass%2520rate%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522headers_footers%2522%252C%2522data%2522%253A%255B94.34%252C95.8%252C89.2%252C91.4%252C88.2%252C87.6%252C94.0%252C88.7%252C92.5%252C92.5%252C19.7%252C90.8%252C88.7%252C94.0%252C92.2%252C94.1%252C35.5%252C96.1%252C96.6%252C95.8%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522headers_footers%2520score%2520by%2520board%2520rank%2520%252820%2520rows%252C%2520Kraken%2520excluded%2529%253A%2520ranks%252013%2520and%252020%2520sit%2520far%2520below%2520the%2520rest%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522board%2520rank%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522pass%2520rate%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" alt="Headers and footers score by board rank for 20 rows, with ranks 13 and 20 at 19.7 and 35.5" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is headers_footers the column that causes the trouble?
&lt;/h2&gt;

&lt;p&gt;Because its 753 tests all ask for text to be absent. A page header, a footer, a page number: the test passes if the model leaves it out. The LightOn authors argue in the file that this "rewards omission, not transcription thus a model that outputs nothing scores perfectly". Their model is trained for full-page transcription and keeps headers on purpose, which lowers the score under the original scoring.&lt;/p&gt;

&lt;p&gt;That is a reasonable position, and it is a position. The category is also the one with the widest spread on the board, 19.7 to 96.6, so any choice about it moves ranks. The &lt;code&gt;baseline&lt;/code&gt; column has the opposite character: across the 17 rows that store it, Kraken excluded, the median is 99.6 and the range is 93.7 to 99.9. Including it lifts a row's average by 1.4 to 4.1 points, about 2.3 at the median, and it separates rows far less than any test category does.&lt;/p&gt;

&lt;p&gt;Neither the exclusion nor the inclusion is hidden. What the board does not do is mark which rows chose which, so the sort key hides the choice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Does the order change when every row is scored the same way?
&lt;/h2&gt;

&lt;p&gt;I built three common yardsticks from the stored per-category values. They are my constructions, not board metrics. Rows that cannot be recomputed (the gated file at rank 3, the mean-only file at rank 9) drop out, so every rank below is among the rows that remain.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Row&lt;/th&gt;
&lt;th&gt;Board rank (of 24)&lt;/th&gt;
&lt;th&gt;6 categories, no &lt;code&gt;headers_footers&lt;/code&gt; (of 22)&lt;/th&gt;
&lt;th&gt;7 categories, no &lt;code&gt;baseline&lt;/code&gt; (of 21)&lt;/th&gt;
&lt;th&gt;7 categories weighted by test count (of 21)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;OXR-1.0&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Infinity-Parser2-Pro&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LightOnOCR-3-4B&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LightOnOCR-2-1B&lt;/td&gt;
&lt;td&gt;13&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;td&gt;17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LightOnOCR-1B-1025&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-OCR&lt;/td&gt;
&lt;td&gt;23&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Rank 1 holds under all three yardsticks. Rank 2 holds under two and drops to 3rd when tests are weighted by count, because &lt;code&gt;arxiv_math&lt;/code&gt; then carries 42 percent of the score. The distortion that matters is in the middle of the table. LightOnOCR-2-1B is 7th when nobody is scored on &lt;code&gt;headers_footers&lt;/code&gt; and 19th when everybody is. The board's 13th matches neither.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%252213%2520LightOnOCR-2-1B%2522%252C%252220%2520LightOnOCR-1B-1025%2522%252C%252223%2520GLM-OCR%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522board%2520rank%2520%2528of%252024%2529%2522%252C%2522data%2522%253A%255B13%252C20%252C23%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522rank%2520on%25206%2520common%2520categories%252C%2520no%2520headers_footers%2520%2528of%252022%2529%2522%252C%2522data%2522%253A%255B7%252C18%252C19%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%252C%257B%2522label%2522%253A%2522rank%2520on%25207%2520common%2520categories%252C%2520headers_footers%2520kept%2520%2528of%252021%2529%2522%252C%2522data%2522%253A%255B19%252C20%252C16%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522The%2520three%2520rows%2520with%2520a%2520different%2520overall%253A%2520rank%2520depends%2520on%2520which%2520columns%2520you%2520average%2520%2528lower%2520bar%2520%253D%2520better%2520rank%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A25%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rank%2520%25281%2520%253D%2520best%2529%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%252213%2520LightOnOCR-2-1B%2522%252C%252220%2520LightOnOCR-1B-1025%2522%252C%252223%2520GLM-OCR%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522board%2520rank%2520%2528of%252024%2529%2522%252C%2522data%2522%253A%255B13%252C20%252C23%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522rank%2520on%25206%2520common%2520categories%252C%2520no%2520headers_footers%2520%2528of%252022%2529%2522%252C%2522data%2522%253A%255B7%252C18%252C19%255D%252C%2522backgroundColor%2522%253A%2522%25232f855a%2522%257D%252C%257B%2522label%2522%253A%2522rank%2520on%25207%2520common%2520categories%252C%2520headers_footers%2520kept%2520%2528of%252021%2529%2522%252C%2522data%2522%253A%255B19%252C20%252C16%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522The%2520three%2520rows%2520with%2520a%2520different%2520overall%253A%2520rank%2520depends%2520on%2520which%2520columns%2520you%2520average%2520%2528lower%2520bar%2520%253D%2520better%2520rank%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A25%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rank%2520%25281%2520%253D%2520best%2529%2522%257D%257D%255D%257D%257D%257D" alt="Rank of three rows under the board's listed overall and under two common yardsticks" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Which other rows cannot be checked?
&lt;/h2&gt;

&lt;p&gt;Six rows cannot be recomputed, and their problems differ.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rank 3, ProKope-4B-Eagle-Eye (87.06):&lt;/strong&gt; the file sits in a gated repo, so the raw YAML answers 401. The board JSON shows a note that describes the number as an overall "across 7 document OCR splits", which may or may not include &lt;code&gt;baseline&lt;/code&gt;. I cannot tell.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rank 9, Surya OCR 2.5 (83.7):&lt;/strong&gt; the file, from an open pull request, stores only &lt;code&gt;overall&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ranks 14, 18 and 21 (chandra, dots.ocr, DeepSeek-OCR):&lt;/strong&gt; all seven categories are stored and &lt;code&gt;baseline&lt;/code&gt; is not. The listed overall minus the seven categories implies a baseline of 99.6, 99.6 and 99.4, which is right where stored baselines sit, so the headline is probably an eight-column mean with one column left out of the file. That is an inference, not a check.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rank 19, DeepSeek-OCR-2 (76.3):&lt;/strong&gt; six categories stored, no &lt;code&gt;headers_footers&lt;/code&gt; and no &lt;code&gt;baseline&lt;/code&gt;, and the source is a tweet.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two more details. Ranks 11 and 12, Surya OCR 2 and its ONNX conversion, carry identical values in all nine entries. That is one measurement stored twice, so the board has 23 distinct results, not 24. And the Falcon-OCR file marks every entry "English-subset only" while the dataset card lists the language as English, so I cannot tell which pages were left out.&lt;/p&gt;

&lt;p&gt;Rank 24 is a different kind of row. It is a reproducible run of a segmentation and OCR pipeline with no table reconstruction, no LaTeX generation and no header suppression, scored by the official scorer. It lists unrounded values, 36.77414 overall, and is the only row that reports an interval, a 95% bootstrap range of 35.9 to 37.7. It works as a floor for the benchmark, and it shows what an interval on this board looks like.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where does rank 1 actually lead?
&lt;/h2&gt;

&lt;p&gt;SoMarkAI/OXR-1.0 is first on &lt;code&gt;arxiv_math&lt;/code&gt; (92.76), &lt;code&gt;old_scans&lt;/code&gt; (59.32) and &lt;code&gt;table_tests&lt;/code&gt; (93.05), and second on &lt;code&gt;old_scans_math&lt;/code&gt;. Among the rows that store the column it is 5th of 20 on &lt;code&gt;headers_footers&lt;/code&gt;, 7th of 21 on &lt;code&gt;multi_column&lt;/code&gt;, 11th of 21 on &lt;code&gt;long_tiny_text&lt;/code&gt; and 16th of 17 on &lt;code&gt;baseline&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Among the rows I could read, the other five column leaders are five different repos: MinerU2.5 on &lt;code&gt;headers_footers&lt;/code&gt; (96.6), LightOnOCR-3-0.8B on &lt;code&gt;long_tiny_text&lt;/code&gt; (94.1), Falcon-OCR on &lt;code&gt;multi_column&lt;/code&gt; (87.1, with the subset note above), Infinity-Parser2-Pro on &lt;code&gt;old_scans_math&lt;/code&gt; (91.3) and jina-ocr-v1 on &lt;code&gt;baseline&lt;/code&gt; (99.9). A single overall number hides that the best row for tables, old scans and math is not the best row for multi-column layouts or headers.&lt;/p&gt;

&lt;p&gt;The rank 1 file is also the only readable one with no source and no notes. Its model card states the result and points to a benchmark document with reproduction instructions. I did not run that reproduction, and a card pointing to one is not the same as a file carrying its source. Every other readable row names one, such as a model card, a technical report or the benchmark's own repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  How big is a 0.24-point lead on 7,010 tests?
&lt;/h2&gt;

&lt;p&gt;Rank 1 is 87.84 and rank 2 is 87.60. The gap to the gated rank 3 at 87.06 is 0.78, and to rank 4 at 86.30 it is 1.54.&lt;/p&gt;

&lt;p&gt;The card's own results table is the best guide, because it was produced on these same pages. Every Overall in it carries a ± between 0.9 and 1.2, and the card does not define the symbol in the text. The only interval on the board itself, for the last row, is a 95% bootstrap range of 35.9 to 37.7 around 36.77, a half-width near 0.9. Those are different scores from different pipelines, so I would not apply them row by row. As an order of magnitude they say the same thing: the first three or four rows of this board are a tier, and the board gives them four distinct ranks.&lt;/p&gt;

&lt;h2&gt;
  
  
  What can a reader do with this board?
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Recompute from the categories.&lt;/strong&gt; Pick the columns you care about and average those. If you do not care about &lt;code&gt;headers_footers&lt;/code&gt; or &lt;code&gt;baseline&lt;/code&gt;, leave them out for every row, not only some.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read the &lt;code&gt;notes&lt;/code&gt; field.&lt;/strong&gt; It is in the board JSON. Three of the 24 rows tell you the overall was computed differently, and one tells you it is a subset.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Count the columns before you compare.&lt;/strong&gt; A row with seven stored columns, six, or one cannot be compared with an eight-column row without an assumption.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat a tie as one measurement.&lt;/strong&gt; Identical values under two repo names are a copy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prefer rows with a reproducible source.&lt;/strong&gt; 6 of 24 rows come from open pull requests and none is marked verified.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For the board itself, two small changes would remove most of the ambiguity: define &lt;code&gt;overall&lt;/code&gt; in &lt;code&gt;eval.yaml&lt;/code&gt;, and show which columns each row's overall includes.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is the olmOCR-bench leaderboard on Hugging Face?&lt;/strong&gt;&lt;br&gt;
It is a board on the dataset page for olmOCR-bench, which has 1,403 PDF pages and 7,010 pass-fail tests. Model repos publish YAML result files and the board lists them, sorted by &lt;code&gt;overall&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How is &lt;code&gt;overall&lt;/code&gt; computed on this board?&lt;/strong&gt;&lt;br&gt;
For 15 of 24 rows it equals the unweighted mean of eight columns: seven test categories and a &lt;code&gt;baseline&lt;/code&gt; column. Three rows leave out &lt;code&gt;headers_footers&lt;/code&gt;, and six cannot be rechecked from their files.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does the board's rank 13 deserve to be 13th?&lt;/strong&gt;&lt;br&gt;
It depends on &lt;code&gt;headers_footers&lt;/code&gt;. LightOnOCR-2-1B is listed at 83.2 with that category excluded. With it excluded for every row the row is 7th of 22, and with it included for every row it is 19th of 21.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are these scores verified?&lt;/strong&gt;&lt;br&gt;
No. &lt;code&gt;verified&lt;/code&gt; is false on all 24 rows. Eighteen files sit on the repo's main branch, six come from open pull requests, and the scores are whatever each file says.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the rank 1 row clearly ahead?&lt;/strong&gt;&lt;br&gt;
On the stored numbers it is first under every recomputation I tried, but the margin over rank 2 is 0.24 points and over rank 3 is 0.78, while the card's own table reports a ± of 0.9 to 1.2 on Overall. I would treat the top three as a tier.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why is &lt;code&gt;headers_footers&lt;/code&gt; controversial?&lt;/strong&gt;&lt;br&gt;
Its 753 tests check that text is absent, so a model that outputs nothing passes them. The LightOn files argue that this rewards omission and leave it out of their overall, and so does the GLM-OCR file. Most other rows keep it, so the board's rows disagree about whether to count it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/open-asr-leaderboard-on-hugging-face-ranks-1-and-2-swap-on-a-common-mean-and-8-of-65-rows-skip-453n"&gt;Open ASR Leaderboard on Hugging Face: Ranks 1 and 2 Swap on a Common Mean, and 8 of 65 Rows Skip TED-LIUM&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/how-hugging-face-official-benchmark-leaderboards-actually-work-evalresults-yaml-the-basemodel-29bo"&gt;How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Field and More&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Data pulled November 8, 2026 (KST) from the board JSON and the YAML files named above. All numbers are the files' own values or my arithmetic on them. The common yardsticks are my constructions, not leaderboard metrics, and I did not run any model.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>ocr</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>Open ASR Leaderboard on Hugging Face: Ranks 1 and 2 Swap on a Common Mean, and 8 of 65 Rows Skip TED-LIUM</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Sat, 07 Nov 2026 00:03:10 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/open-asr-leaderboard-on-hugging-face-ranks-1-and-2-swap-on-a-common-mean-and-8-of-65-rows-skip-453n</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/open-asr-leaderboard-on-hugging-face-ranks-1-and-2-swap-on-a-common-mean-and-8-of-65-rows-skip-453n</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On November 7, 2026 the Hugging Face Open ASR Leaderboard board lists &lt;strong&gt;65 rows from 65 repos, all lower-is-better word error rate (WER), 0 verified.&lt;/strong&gt; 45 rows come from pull requests that are still open, all opened by one account, and 20 sit on a repo's main branch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The board's &lt;code&gt;mean_wer&lt;/code&gt; is not the same average for every row.&lt;/strong&gt; 54 rows average 8 datasets. &lt;strong&gt;8 rows, including ranks 2, 3 and 4, have no TED-LIUM score and average 7.&lt;/strong&gt; TED-LIUM is the easiest column for all 54 rows that report it, so leaving it out raises a row's mean by about 0.3 to 0.6 points.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On the 7 datasets every row has, ranks 1 and 2 swap.&lt;/strong&gt; The board gives 4.76 to ARK-ASR-3B and 4.87 to MOSS-Transcribe-preview-2B. On the common 7 the numbers are 5.04 and 4.87.&lt;/li&gt;
&lt;li&gt;The board's rank 1 is first in only 2 of 8 columns, and five different checkpoints lead the eight columns. Ranks 3 and 4 are one measurement stored twice.&lt;/li&gt;
&lt;li&gt;One row's mean does not match its own parts (7.12 listed, 6.94 recomputed), the rank 6 file sits in a gated repo I could not open, and two rows carry only a mean.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2520ARK-ASR-3B%2522%252C%25222%2520MOSS-Tr-preview-2B%2522%252C%25223%2520MOSS-Tr-Diarize%2522%252C%25225%2520parakeet-ultra%2522%252C%25227%2520ARK-ASR-0.6B%2522%252C%25228%2520Zipformer-cr-XL%2522%252C%25229%2520Qwen3-ASR-1.7B%2522%252C%252210%2520Phi-4-multimodal-instruct%2522%252C%252211%2520parakeet-tdt-0.6b-v2%2522%252C%252212%2520parakeet-redux%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522board%2520mean_wer%2520%25287%2520or%25208%2520sets%252C%2520as%2520listed%2529%2522%252C%2522data%2522%253A%255B4.76%252C4.87%252C5.17%252C5.32%252C5.53%252C5.56%252C5.76%252C6.02%252C6.05%252C6.05%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522mean%2520over%2520the%25207%2520sets%2520every%2520row%2520has%2522%252C%2522data%2522%253A%255B5.04%252C4.87%252C5.17%252C5.8%252C5.97%252C5.56%252C6.25%252C6.46%252C6.44%252C6.55%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Open%2520ASR%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25207%252C%25202026%253A%2520board%2520mean%2520vs%2520common%25207-set%2520mean%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A4%252C%2522max%2522%253A7%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522mean%2520WER%252C%2520%2525%2520%2528lower%2520is%2520better%253B%2520axis%2520starts%2520at%25204%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2520ARK-ASR-3B%2522%252C%25222%2520MOSS-Tr-preview-2B%2522%252C%25223%2520MOSS-Tr-Diarize%2522%252C%25225%2520parakeet-ultra%2522%252C%25227%2520ARK-ASR-0.6B%2522%252C%25228%2520Zipformer-cr-XL%2522%252C%25229%2520Qwen3-ASR-1.7B%2522%252C%252210%2520Phi-4-multimodal-instruct%2522%252C%252211%2520parakeet-tdt-0.6b-v2%2522%252C%252212%2520parakeet-redux%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522board%2520mean_wer%2520%25287%2520or%25208%2520sets%252C%2520as%2520listed%2529%2522%252C%2522data%2522%253A%255B4.76%252C4.87%252C5.17%252C5.32%252C5.53%252C5.56%252C5.76%252C6.02%252C6.05%252C6.05%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%252C%257B%2522label%2522%253A%2522mean%2520over%2520the%25207%2520sets%2520every%2520row%2520has%2522%252C%2522data%2522%253A%255B5.04%252C4.87%252C5.17%252C5.8%252C5.97%252C5.56%252C6.25%252C6.46%252C6.44%252C6.55%255D%252C%2522backgroundColor%2522%253A%2522%2523dd6b20%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Open%2520ASR%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25207%252C%25202026%253A%2520board%2520mean%2520vs%2520common%25207-set%2520mean%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A4%252C%2522max%2522%253A7%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522mean%2520WER%252C%2520%2525%2520%2528lower%2520is%2520better%253B%2520axis%2520starts%2520at%25204%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" alt="Open ASR board mean WER next to the mean over the 7 datasets every row has, for 10 of the top 12 rows" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Blue bars are the numbers on the board. Orange bars are my recomputation over AMI, Earnings-22, GigaSpeech, LibriSpeech clean, LibriSpeech other, SPGISpeech and VoxPopuli, the seven columns that all 62 rows with per-dataset values share. For the rows that already average those seven, the bars are equal. The axis starts at 4, so lengths exaggerate the gaps; read the numbers.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does the Open ASR board on Hugging Face contain?
&lt;/h2&gt;

&lt;p&gt;The Open ASR Leaderboard scores speech recognition models on a fixed set of public English test sets and reports word error rate, the share of reference words a transcript gets wrong. Hugging Face lists &lt;code&gt;hf-audio/open-asr-leaderboard&lt;/code&gt; as a benchmark dataset, so a model repo can publish results in &lt;code&gt;.eval_results/*.yaml&lt;/code&gt; and the board collects them.&lt;/p&gt;

&lt;p&gt;I pulled the board on November 7, 2026 (KST) from the JSON behind the leaderboard view. It has 65 rows, one per repo, and &lt;code&gt;verified&lt;/code&gt; is false on every one. Source labels: 61 rows say &lt;code&gt;open-asr-leaderboard&lt;/code&gt;, two say &lt;code&gt;moondream&lt;/code&gt;, and two cite their own model card.&lt;/p&gt;

&lt;p&gt;The dataset's &lt;code&gt;eval.yaml&lt;/code&gt; names 12 task ids: &lt;code&gt;mean_wer&lt;/code&gt;, &lt;code&gt;rtfx&lt;/code&gt;, and ten WER sets (LibriSpeech clean and other, Common Voice, VoxPopuli, TED-LIUM, GigaSpeech, SPGISpeech, Earnings-22, AMI, and a cleaned VoxPopuli). It does not say which of those feed &lt;code&gt;mean_wer&lt;/code&gt;, and it does not say that all of them must be present. No readable file reports Common Voice or the cleaned VoxPopuli, so the working definition is a mean over up to eight sets that nobody wrote down in one place.&lt;/p&gt;

&lt;p&gt;Where do the 65 rows come from? 45 are open pull requests, every one opened by the same account (SaylorTwift). 43 of those were opened in April 2026 and two on October 28. The other 20 are YAML files already on the model repo's main branch. A row from an open pull request is a claim that the repo owner has not yet accepted, and the board shows it next to accepted ones with no marker.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is every &lt;code&gt;mean_wer&lt;/code&gt; an average of the same datasets?
&lt;/h2&gt;

&lt;p&gt;No. I fetched the YAML behind every row, from the pull request ref when the row has one and from main otherwise. 64 of 65 files opened. The rank 6 file is in a gated repo and answered HTTP 401 without accepting its terms, so I could not check it. Of the 64:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Group&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;th&gt;What the file holds&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Eight datasets plus mean (RTFx on most)&lt;/td&gt;
&lt;td&gt;54&lt;/td&gt;
&lt;td&gt;AMI, Earnings-22, GigaSpeech, LS clean, LS other, SPGISpeech, TED-LIUM, VoxPopuli&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Seven datasets, no TED-LIUM&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;the same list minus TED-LIUM&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mean only&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;one &lt;code&gt;mean_wer&lt;/code&gt; number, no per-dataset values&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The eight rows without TED-LIUM sit at ranks 2, 3, 4, 8, 20, 23, 45 and 48. Their &lt;code&gt;mean_wer&lt;/code&gt; is the average of the seven they have. That is a fair way to report a model, but it is not the number the other 54 rows carry, and the board sorts them in one column.&lt;/p&gt;

&lt;p&gt;Does the difference matter? TED-LIUM is the easiest column. For every one of the 54 rows that report it, the TED-LIUM WER is lower than that row's own mean over the other seven. The median TED-LIUM score is 4.1. Adding an easy set pulls a mean down, so a row that skips it starts about 0.56 points behind at the median, and about 0.3 to 0.5 points behind among the top rows. The rank 1 row moves from 5.04 to 4.76 when its TED-LIUM score (2.79) joins the average.&lt;/p&gt;

&lt;p&gt;moondream's own file header makes the same point from the other side. It says its card figure of 5.80 "is the mean of the other seven" and the board value of 5.32 averages all eight. My recomputation of that row gives 5.803 over seven, which matches its note.&lt;/p&gt;

&lt;h2&gt;
  
  
  Do ranks 1 and 2 survive a common mean?
&lt;/h2&gt;

&lt;p&gt;Not on the numbers I can compute. The board's top two:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Board rank&lt;/th&gt;
&lt;th&gt;Repo&lt;/th&gt;
&lt;th&gt;Board mean&lt;/th&gt;
&lt;th&gt;Sets averaged&lt;/th&gt;
&lt;th&gt;Mean over the common 7&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Edge0/ARK-ASR-3B&lt;/td&gt;
&lt;td&gt;4.76&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;5.044&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;OpenMOSS-Team/MOSS-Transcribe-preview-2B&lt;/td&gt;
&lt;td&gt;4.87&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;4.866&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the board, rank 1 leads by 0.11. On the seven columns both rows have, rank 2 leads by 0.18. Across all 62 rows with per-dataset values, the common-7 order starts with the MOSS preview model, then ARK-ASR-3B, then the MOSS Diarize pair, then Zipformer-cr-ctc-transducer-XL-290M, then moondream's parakeet-ultra. The board has parakeet-ultra at rank 5 and the Zipformer at rank 8, so those two swap as well.&lt;/p&gt;

&lt;p&gt;I do not claim rank 2 is the better model. Here is the fair reading. If the MOSS preview model had a TED-LIUM score of at most 4.04, its 8-set mean would stay below ARK-ASR-3B's 4.76. The rows ranked 1, 5, 7 and 9 report TED-LIUM scores of 2.79, 1.94, 2.45 and 2.28, so a model this strong would probably clear that bar and also be first on eight sets. But that score is not on the board, and the board's order of the top two comes from comparing a 7-set mean with an 8-set mean. So the board cannot say which is first.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522AMI%2522%252C%2522Earnings-22%2522%252C%2522GigaSpeech%2522%252C%2522LS%2520clean%2522%252C%2522LS%2520other%2522%252C%2522SPGISpeech%2522%252C%2522TED-LIUM%2522%252C%2522VoxPopuli%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rank%2522%252C%2522data%2522%253A%255B4%252C3%252C2%252C1%252C1%252C14%252C6%252C6%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232f855a%2522%252C%2522%25232f855a%2522%252C%2522%2523c53030%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Where%2520board%2520rank%25201%2520%2528ARK-ASR-3B%2529%2520places%2520in%2520each%2520dataset%2520%252862%2520rows%252C%252054%2520for%2520TED-LIUM%2529%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A16%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rank%2520within%2520the%2520dataset%2520%25281%2520%253D%2520lowest%2520WER%2529%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522AMI%2522%252C%2522Earnings-22%2522%252C%2522GigaSpeech%2522%252C%2522LS%2520clean%2522%252C%2522LS%2520other%2522%252C%2522SPGISpeech%2522%252C%2522TED-LIUM%2522%252C%2522VoxPopuli%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rank%2522%252C%2522data%2522%253A%255B4%252C3%252C2%252C1%252C1%252C14%252C6%252C6%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232f855a%2522%252C%2522%25232f855a%2522%252C%2522%2523c53030%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Where%2520board%2520rank%25201%2520%2528ARK-ASR-3B%2529%2520places%2520in%2520each%2520dataset%2520%252862%2520rows%252C%252054%2520for%2520TED-LIUM%2529%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A16%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rank%2520within%2520the%2520dataset%2520%25281%2520%253D%2520lowest%2520WER%2529%2522%257D%257D%255D%257D%257D%257D" alt="Rank of board row 1 in each of the eight datasets: 4, 3, 2, 1, 1, 14, 6, 6" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Who actually leads each column?
&lt;/h2&gt;

&lt;p&gt;The board's rank 1 is a good model by any column. It is just not first everywhere. Among the 62 rows with per-dataset values, ARK-ASR-3B ranks 4th on AMI, 3rd on Earnings-22, 2nd on GigaSpeech, 1st on both LibriSpeech sets, 14th on SPGISpeech, and 6th on TED-LIUM and VoxPopuli. Its SPGISpeech WER is 2.46 against a best of 1.63.&lt;/p&gt;

&lt;p&gt;Five different checkpoints lead the eight columns:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Column&lt;/th&gt;
&lt;th&gt;Lowest WER&lt;/th&gt;
&lt;th&gt;Repo&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;AMI, GigaSpeech, SPGISpeech&lt;/td&gt;
&lt;td&gt;8.37, 6.78, 1.63&lt;/td&gt;
&lt;td&gt;MOSS-Transcribe-preview-2B&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Earnings-22&lt;/td&gt;
&lt;td&gt;7.64&lt;/td&gt;
&lt;td&gt;Zipformer-cr-ctc-transducer-XL-290M&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LibriSpeech clean, LibriSpeech other&lt;/td&gt;
&lt;td&gt;1.03, 2.35&lt;/td&gt;
&lt;td&gt;ARK-ASR-3B&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;VoxPopuli&lt;/td&gt;
&lt;td&gt;3.58&lt;/td&gt;
&lt;td&gt;MOSS-Transcribe-Diarize (tied with its copy)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TED-LIUM&lt;/td&gt;
&lt;td&gt;1.94&lt;/td&gt;
&lt;td&gt;moondream/parakeet-ultra&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A single mean hides this, which is its job. Ranks 1 to 9 on the board fit inside 1.0 point, 4.76 to 5.76, while the per-column leaders are spread across that whole group. Which model to pick depends on whether your audio looks like meetings (AMI), earnings calls, read audiobooks or parliamentary speech. The board sorts by &lt;code&gt;mean_wer&lt;/code&gt;; the per-dataset columns you would need are only in each repo's YAML file.&lt;/p&gt;

&lt;p&gt;The board publishes no per-utterance output, so I cannot put an interval on any of these gaps. I can say that TED-LIUM has 1,155 test utterances and VoxPopuli 1,842, so the smaller columns are the noisier ones. A 0.11 gap in a mean of eight is a small thing to hang a rank on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why are ranks 3 and 4 the same row twice?
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;OpenMOSS-Team/MOSS-Transcribe-Diarize&lt;/code&gt; and &lt;code&gt;hf-audio/MOSS-Transcribe-Diarize-HF&lt;/code&gt; have the same parameter count (908,513,280), the same date (2026-07-25), and identical values in every field of their YAML files: the mean, RTFx and each per-dataset WER. The second repo sits under the &lt;code&gt;hf-audio&lt;/code&gt; organization, the same one that hosts the dataset, and looks like a converted copy of the first. Publishing a copy is fair. On the board it is two rows, and it moves everything below it down one place. If you read "top 5" as five models, you are reading four.&lt;/p&gt;

&lt;h2&gt;
  
  
  What do self-reported rows change?
&lt;/h2&gt;

&lt;p&gt;Two rows come from moondream (ranks 5 and 12), and they say so. Their YAML header reads "Self-reported", says the scores use the Open ASR Leaderboard's own pipeline with its normalizers as of September 2026, and carries a second note about TED-LIUM: the leaderboard's TED-LIUM split "is no longer distributed", so &lt;code&gt;tedlium_wer&lt;/code&gt; scores the eleven TED-LIUM 3 test talks whole. That is a different test from the leaderboard's own TED-LIUM split, which the dataset card lists at 1,155 utterances and which should sit behind the other 52 TED-LIUM values. So the eighth column of those two rows measures something else, and the column that is easiest for every row is exactly the one that differs.&lt;/p&gt;

&lt;p&gt;The header discloses this well, yet the value still lands as a plain number in a column that sorts it against the others.&lt;/p&gt;

&lt;p&gt;The two LiquidAI rows (ranks 13 and 15) are the opposite case. Each file holds a single &lt;code&gt;mean_wer&lt;/code&gt; (6.23 and 6.36), sourced from the model card. The card describes it as the average WER "on the ESB / Open ASR benchmark suite" and does not list the datasets. There are no per-dataset values to check, and no RTFx.&lt;/p&gt;

&lt;h2&gt;
  
  
  Does the mean match its parts, and is the speed column usable?
&lt;/h2&gt;

&lt;p&gt;For the 62 rows with per-dataset values, I recomputed the mean from the parts. 61 match their own &lt;code&gt;mean_wer&lt;/code&gt; to within 0.011. One does not: &lt;code&gt;nvidia/canary-180m-flash&lt;/code&gt; lists 7.12, and its eight values average 6.94. If the parts are right, that row belongs at rank 21 instead of 25. I do not know which number is stale. The pull request that added it has been open since April 7.&lt;/p&gt;

&lt;p&gt;The board also carries RTFx, the real-time factor, for 59 rows. It ranges from 42 to 6,399. Four rows are on the accuracy and speed frontier, meaning no other row is both faster and lower in WER: ARK-ASR-3B (4.76 at 491), parakeet-tdt-0.6b-v2 (6.05 at 3,386), parakeet-tdt_ctc-110m (7.49 at 5,345) and stt_en_fastconformer_ctc_large (8.97 at 6,399). Moving from rank 1 to parakeet-tdt-0.6b-v2 costs 1.29 points of WER and buys a 6.9 times faster row.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522scatter%2522%252C%2522data%2522%253A%257B%2522datasets%2522%253A%255B%257B%2522label%2522%253A%252259%2520rows%2520with%2520RTFx%2522%252C%2522data%2522%253A%255B%257B%2522x%2522%253A173.3%252C%2522y%2522%253A4.87%257D%252C%257B%2522x%2522%253A294.0%252C%2522y%2522%253A5.17%257D%252C%257B%2522x%2522%253A294.0%252C%2522y%2522%253A5.17%257D%252C%257B%2522x%2522%253A314.1%252C%2522y%2522%253A5.53%257D%252C%257B%2522x%2522%253A147.9%252C%2522y%2522%253A5.76%257D%252C%257B%2522x%2522%253A151.1%252C%2522y%2522%253A6.02%257D%252C%257B%2522x%2522%253A1045.8%252C%2522y%2522%253A6.35%257D%252C%257B%2522x%2522%253A88.4%252C%2522y%2522%253A6.4%257D%252C%257B%2522x%2522%253A166.2%252C%2522y%2522%253A6.42%257D%252C%257B%2522x%2522%253A235.3%252C%2522y%2522%253A6.50375%257D%252C%257B%2522x%2522%253A448.1%252C%2522y%2522%253A6.66%257D%252C%257B%2522x%2522%253A144.2%252C%2522y%2522%253A6.71%257D%252C%257B%2522x%2522%253A2390.6%252C%2522y%2522%253A7.015%257D%252C%257B%2522x%2522%253A145.3%252C%2522y%2522%253A7.03%257D%252C%257B%2522x%2522%253A741.1%252C%2522y%2522%253A7.03%257D%252C%257B%2522x%2522%253A109.9%252C%2522y%2522%253A7.05%257D%252C%257B%2522x%2522%253A1233.6%252C%2522y%2522%253A7.12%257D%252C%257B%2522x%2522%253A2053.1%252C%2522y%2522%253A7.12375%257D%252C%257B%2522x%2522%253A749%252C%2522y%2522%253A7.15%257D%252C%257B%2522x%2522%253A202.0%252C%2522y%2522%253A7.21%257D%252C%257B%2522x%2522%253A454.0%252C%2522y%2522%253A7.42%257D%252C%257B%2522x%2522%253A145.5%252C%2522y%2522%253A7.44%257D%252C%257B%2522x%2522%253A2815.7%252C%2522y%2522%253A7.495%257D%252C%257B%2522x%2522%253A4281.5%252C%2522y%2522%253A7.69%257D%252C%257B%2522x%2522%253A51.8%252C%2522y%2522%253A7.77%257D%252C%257B%2522x%2522%253A144.5%252C%2522y%2522%253A7.835%257D%252C%257B%2522x%2522%253A566.3%252C%2522y%2522%253A7.84%257D%252C%257B%2522x%2522%253A143.8%252C%2522y%2522%253A7.9425%257D%252C%257B%2522x%2522%253A182.1%252C%2522y%2522%253A8.09%257D%252C%257B%2522x%2522%253A502.0%252C%2522y%2522%253A8.12%257D%252C%257B%2522x%2522%253A4295.0%252C%2522y%2522%253A8.31875%257D%252C%257B%2522x%2522%253A42.2%252C%2522y%2522%253A8.48%257D%252C%257B%2522x%2522%253A268.9%252C%2522y%2522%253A8.59%257D%252C%257B%2522x%2522%253A4097.4%252C%2522y%2522%253A9.0575%257D%252C%257B%2522x%2522%253A2623.4%252C%2522y%2522%253A9.36%257D%252C%257B%2522x%2522%253A566.0%252C%2522y%2522%253A9.99%257D%252C%257B%2522x%2522%253A320.7%252C%2522y%2522%253A10.3175%257D%252C%257B%2522x%2522%253A2840.2%252C%2522y%2522%253A11.01%257D%252C%257B%2522x%2522%253A5686.9%252C%2522y%2522%253A11.15875%257D%252C%257B%2522x%2522%253A847.2%252C%2522y%2522%253A12%257D%252C%257B%2522x%2522%253A753.1%252C%2522y%2522%253A12.65%257D%252C%257B%2522x%2522%253A348.1%252C%2522y%2522%253A12.80625%257D%252C%257B%2522x%2522%253A451.2%252C%2522y%2522%253A14.3475%257D%252C%257B%2522x%2522%253A509.3%252C%2522y%2522%253A21.2725%257D%252C%257B%2522x%2522%253A230.8%252C%2522y%2522%253A22.53875%257D%252C%257B%2522x%2522%253A361.3%252C%2522y%2522%253A22.5475%257D%252C%257B%2522x%2522%253A495.9%252C%2522y%2522%253A22.69375%257D%252C%257B%2522x%2522%253A503.8%252C%2522y%2522%253A22.93125%257D%252C%257B%2522x%2522%253A470.2%252C%2522y%2522%253A23.21%257D%252C%257B%2522x%2522%253A607.9%252C%2522y%2522%253A23.28375%257D%252C%257B%2522x%2522%253A522.5%252C%2522y%2522%253A23.29125%257D%252C%257B%2522x%2522%253A516.6%252C%2522y%2522%253A26.7725%257D%252C%257B%2522x%2522%253A648.1%252C%2522y%2522%253A28.3025%257D%252C%257B%2522x%2522%253A686.0%252C%2522y%2522%253A29.4025%257D%252C%257B%2522x%2522%253A234.4%252C%2522y%2522%253A39.79625%257D%255D%252C%2522backgroundColor%2522%253A%2522%2523a0aec0%2522%252C%2522borderColor%2522%253A%2522%2523a0aec0%2522%252C%2522pointRadius%2522%253A4%257D%252C%257B%2522label%2522%253A%2522frontier%2520%2528no%2520row%2520is%2520both%2520faster%2520and%2520lower%2520WER%2529%2522%252C%2522data%2522%253A%255B%257B%2522x%2522%253A491.0%252C%2522y%2522%253A4.76%257D%252C%257B%2522x%2522%253A3386.0%252C%2522y%2522%253A6.05%257D%252C%257B%2522x%2522%253A5345.1%252C%2522y%2522%253A7.49%257D%252C%257B%2522x%2522%253A6399.3%252C%2522y%2522%253A8.965%257D%255D%252C%2522backgroundColor%2522%253A%2522%2523c53030%2522%252C%2522borderColor%2522%253A%2522%2523c53030%2522%252C%2522pointRadius%2522%253A7%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Accuracy%2520vs%2520speed%2520on%2520the%2520Open%2520ASR%2520board%253A%25204%2520of%252059%2520rows%2520are%2520on%2520the%2520frontier%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522type%2522%253A%2522logarithmic%2522%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522RTFx%2520%2528log%2520scale%252C%2520higher%2520is%2520faster%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A30%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522mean%2520WER%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522scatter%2522%252C%2522data%2522%253A%257B%2522datasets%2522%253A%255B%257B%2522label%2522%253A%252259%2520rows%2520with%2520RTFx%2522%252C%2522data%2522%253A%255B%257B%2522x%2522%253A173.3%252C%2522y%2522%253A4.87%257D%252C%257B%2522x%2522%253A294.0%252C%2522y%2522%253A5.17%257D%252C%257B%2522x%2522%253A294.0%252C%2522y%2522%253A5.17%257D%252C%257B%2522x%2522%253A314.1%252C%2522y%2522%253A5.53%257D%252C%257B%2522x%2522%253A147.9%252C%2522y%2522%253A5.76%257D%252C%257B%2522x%2522%253A151.1%252C%2522y%2522%253A6.02%257D%252C%257B%2522x%2522%253A1045.8%252C%2522y%2522%253A6.35%257D%252C%257B%2522x%2522%253A88.4%252C%2522y%2522%253A6.4%257D%252C%257B%2522x%2522%253A166.2%252C%2522y%2522%253A6.42%257D%252C%257B%2522x%2522%253A235.3%252C%2522y%2522%253A6.50375%257D%252C%257B%2522x%2522%253A448.1%252C%2522y%2522%253A6.66%257D%252C%257B%2522x%2522%253A144.2%252C%2522y%2522%253A6.71%257D%252C%257B%2522x%2522%253A2390.6%252C%2522y%2522%253A7.015%257D%252C%257B%2522x%2522%253A145.3%252C%2522y%2522%253A7.03%257D%252C%257B%2522x%2522%253A741.1%252C%2522y%2522%253A7.03%257D%252C%257B%2522x%2522%253A109.9%252C%2522y%2522%253A7.05%257D%252C%257B%2522x%2522%253A1233.6%252C%2522y%2522%253A7.12%257D%252C%257B%2522x%2522%253A2053.1%252C%2522y%2522%253A7.12375%257D%252C%257B%2522x%2522%253A749%252C%2522y%2522%253A7.15%257D%252C%257B%2522x%2522%253A202.0%252C%2522y%2522%253A7.21%257D%252C%257B%2522x%2522%253A454.0%252C%2522y%2522%253A7.42%257D%252C%257B%2522x%2522%253A145.5%252C%2522y%2522%253A7.44%257D%252C%257B%2522x%2522%253A2815.7%252C%2522y%2522%253A7.495%257D%252C%257B%2522x%2522%253A4281.5%252C%2522y%2522%253A7.69%257D%252C%257B%2522x%2522%253A51.8%252C%2522y%2522%253A7.77%257D%252C%257B%2522x%2522%253A144.5%252C%2522y%2522%253A7.835%257D%252C%257B%2522x%2522%253A566.3%252C%2522y%2522%253A7.84%257D%252C%257B%2522x%2522%253A143.8%252C%2522y%2522%253A7.9425%257D%252C%257B%2522x%2522%253A182.1%252C%2522y%2522%253A8.09%257D%252C%257B%2522x%2522%253A502.0%252C%2522y%2522%253A8.12%257D%252C%257B%2522x%2522%253A4295.0%252C%2522y%2522%253A8.31875%257D%252C%257B%2522x%2522%253A42.2%252C%2522y%2522%253A8.48%257D%252C%257B%2522x%2522%253A268.9%252C%2522y%2522%253A8.59%257D%252C%257B%2522x%2522%253A4097.4%252C%2522y%2522%253A9.0575%257D%252C%257B%2522x%2522%253A2623.4%252C%2522y%2522%253A9.36%257D%252C%257B%2522x%2522%253A566.0%252C%2522y%2522%253A9.99%257D%252C%257B%2522x%2522%253A320.7%252C%2522y%2522%253A10.3175%257D%252C%257B%2522x%2522%253A2840.2%252C%2522y%2522%253A11.01%257D%252C%257B%2522x%2522%253A5686.9%252C%2522y%2522%253A11.15875%257D%252C%257B%2522x%2522%253A847.2%252C%2522y%2522%253A12%257D%252C%257B%2522x%2522%253A753.1%252C%2522y%2522%253A12.65%257D%252C%257B%2522x%2522%253A348.1%252C%2522y%2522%253A12.80625%257D%252C%257B%2522x%2522%253A451.2%252C%2522y%2522%253A14.3475%257D%252C%257B%2522x%2522%253A509.3%252C%2522y%2522%253A21.2725%257D%252C%257B%2522x%2522%253A230.8%252C%2522y%2522%253A22.53875%257D%252C%257B%2522x%2522%253A361.3%252C%2522y%2522%253A22.5475%257D%252C%257B%2522x%2522%253A495.9%252C%2522y%2522%253A22.69375%257D%252C%257B%2522x%2522%253A503.8%252C%2522y%2522%253A22.93125%257D%252C%257B%2522x%2522%253A470.2%252C%2522y%2522%253A23.21%257D%252C%257B%2522x%2522%253A607.9%252C%2522y%2522%253A23.28375%257D%252C%257B%2522x%2522%253A522.5%252C%2522y%2522%253A23.29125%257D%252C%257B%2522x%2522%253A516.6%252C%2522y%2522%253A26.7725%257D%252C%257B%2522x%2522%253A648.1%252C%2522y%2522%253A28.3025%257D%252C%257B%2522x%2522%253A686.0%252C%2522y%2522%253A29.4025%257D%252C%257B%2522x%2522%253A234.4%252C%2522y%2522%253A39.79625%257D%255D%252C%2522backgroundColor%2522%253A%2522%2523a0aec0%2522%252C%2522borderColor%2522%253A%2522%2523a0aec0%2522%252C%2522pointRadius%2522%253A4%257D%252C%257B%2522label%2522%253A%2522frontier%2520%2528no%2520row%2520is%2520both%2520faster%2520and%2520lower%2520WER%2529%2522%252C%2522data%2522%253A%255B%257B%2522x%2522%253A491.0%252C%2522y%2522%253A4.76%257D%252C%257B%2522x%2522%253A3386.0%252C%2522y%2522%253A6.05%257D%252C%257B%2522x%2522%253A5345.1%252C%2522y%2522%253A7.49%257D%252C%257B%2522x%2522%253A6399.3%252C%2522y%2522%253A8.965%257D%255D%252C%2522backgroundColor%2522%253A%2522%2523c53030%2522%252C%2522borderColor%2522%253A%2522%2523c53030%2522%252C%2522pointRadius%2522%253A7%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Accuracy%2520vs%2520speed%2520on%2520the%2520Open%2520ASR%2520board%253A%25204%2520of%252059%2520rows%2520are%2520on%2520the%2520frontier%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Atrue%252C%2522position%2522%253A%2522bottom%2522%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522type%2522%253A%2522logarithmic%2522%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522RTFx%2520%2528log%2520scale%252C%2520higher%2520is%2520faster%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A30%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522mean%2520WER%252C%2520%2525%2522%257D%257D%255D%257D%257D%257D" alt="Mean WER against RTFx on a log scale for the 59 rows that report speed, with the 4-row frontier marked" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Neither the YAML files I read nor the dataset's README says what hardware RTFx was measured on, and five of the 64 readable files have no RTFx at all. A speed column without a stated machine is a hint, not a measurement.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to reproduce this audit
&lt;/h2&gt;

&lt;p&gt;The script below pulls the board, reads each row's file, and recomputes both the common-7 mean and the all-sets mean. It skips gated repos and mean-only rows and prints the five best rows on the common 7 plus any row whose mean disagrees with its parts. I ran it on November 7, 2026, and it printed the MOSS preview model first at 4.866, ARK-ASR-3B second at 5.044, and &lt;code&gt;canary-180m-flash&lt;/code&gt; as the one mismatch.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;yaml&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;SEVEN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ami&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;earnings22&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gigaspeech&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;librispeech_clean&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
         &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;librispeech_other&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spgispeech&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;voxpopuli&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/datasets/hf-audio/open-asr-leaderboard/leaderboard&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;pr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pullRequest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ref&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;refs%2Fpr%2F&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;pr&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;pr&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;main&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/raw/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;filename&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;                      &lt;span class="c1"&gt;# gated repos answer 401
&lt;/span&gt;    &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dataset&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;task_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;yaml&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;safe_load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;_wer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;SEVEN&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;                      &lt;span class="c1"&gt;# mean-only rows cannot be checked
&lt;/span&gt;    &lt;span class="n"&gt;m7&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;_wer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;SEVEN&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt;
    &lt;span class="n"&gt;sets&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;_wer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mean_wer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;m_all&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;sets&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sets&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rank&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m_all&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sets&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;board rank | repo | board mean | recomputed | common-7 | sets&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;])[:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mean does not match its parts:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mf"&gt;0.011&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What should a reader do with this board?
&lt;/h2&gt;

&lt;p&gt;A few habits that cost nothing:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Read the number of sets next to the mean.&lt;/strong&gt; A mean over 7 and a mean over 8 are different measurements, and on this board the gap is larger than the gap between several neighbouring ranks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sort by the column you care about.&lt;/strong&gt; Meetings, earnings calls, read speech and parliamentary speech rank the same models differently.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat a tie as one measurement.&lt;/strong&gt; Identical values under two repo names are a copy, not a replication.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check who opened the pull request and whether it is still open.&lt;/strong&gt; 45 of 65 rows have not been accepted by the repo owner.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ask what RTFx was measured on&lt;/strong&gt; before you trade accuracy for speed.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For the board itself, three small changes would remove most of the ambiguity: define which tasks feed &lt;code&gt;mean_wer&lt;/code&gt; in &lt;code&gt;eval.yaml&lt;/code&gt;, show the number of sets beside the mean, and mark rows that are not comparable on a column, such as a different TED-LIUM split.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is the Open ASR Leaderboard on Hugging Face?&lt;/strong&gt;&lt;br&gt;
It is a benchmark dataset page that collects word error rate results for speech recognition models on public English test sets such as LibriSpeech, AMI, GigaSpeech and TED-LIUM. Model repos publish results as YAML files and the board lists them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the board's &lt;code&gt;mean_wer&lt;/code&gt; the average of the same datasets for every model?&lt;/strong&gt;&lt;br&gt;
Not on November 7, 2026. 54 rows average eight datasets, and eight rows average seven because they have no TED-LIUM score. Two more rows list only a mean.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Which model is first on the Open ASR board?&lt;/strong&gt;&lt;br&gt;
The board lists Edge0/ARK-ASR-3B first with 4.76. On the seven datasets all rows share, OpenMOSS-Team/MOSS-Transcribe-preview-2B is lower at 4.866 against 5.044. The board cannot settle the order without the MOSS model's TED-LIUM score.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why does leaving out TED-LIUM change a mean?&lt;/strong&gt;&lt;br&gt;
TED-LIUM is the easiest column. For all 54 rows that report it, its WER is lower than the row's own mean over the other seven sets. A mean without it is higher by about 0.3 to 0.6 points.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does verified mean the scores were reproduced?&lt;/strong&gt;&lt;br&gt;
On this board, &lt;code&gt;verified&lt;/code&gt; is false on all 65 rows. The scores are whatever each YAML file says, whether it arrived through a pull request or a commit on the repo's main branch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I trust the RTFx column?&lt;/strong&gt;&lt;br&gt;
Use it as a rough guide. The files do not state the hardware, and five of the 64 readable files have no RTFx.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/how-hugging-face-official-benchmark-leaderboards-actually-work-evalresults-yaml-the-basemodel-29bo"&gt;How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Field and More&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/when-a-04-point-lead-means-nothing-reading-open-model-leaderboard-margins-like-a-statistician-5do7"&gt;When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Data pulled November 7, 2026 (KST) from the board JSON and the YAML files named above. All numbers are the files' own values or my arithmetic on them. The common-7 mean is my construction, not a leaderboard metric.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>speechrecognition</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>SWE-bench Verified on Hugging Face: 9 Self-Reported Rows Top the Official Table's Best, and 6 Repos Fill 16 of 78 Rows</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Thu, 05 Nov 2026 00:36:41 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-verified-on-hugging-face-9-self-reported-rows-top-the-official-tables-best-and-6-repos-3173</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-verified-on-hugging-face-9-self-reported-rows-top-the-official-tables-best-and-6-repos-3173</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On November 5, 2026 the Hugging Face SWE-bench Verified board lists &lt;strong&gt;78 rows from 68 repos. 51 rows come from pull requests that are still open, 27 sit on a repo's main branch, and 0 are verified.&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nine rows are above 79.2, the highest entry in the Verified table on swebench.com&lt;/strong&gt; (data file last updated February 26, 2026). All nine are self-reports. The rank 1 row (86.0) names its harness only in the model card's footnote, and the board row has no note.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Six repos take 16 of the 78 rows.&lt;/strong&gt; One gpt-oss repo appears three times at three reasoning efforts, and the 20B model's rows span 23.3 points. A Nemotron model and its re-upload carry the same three harness scores, so six rows are three measurements.&lt;/li&gt;
&lt;li&gt;Three rows cite "SWE-Bench official evaluation" and match the swebench.com mini-SWE-agent runs of February 17, 2026 exactly. One of those repos, GLM-5, also has the maker's own 77.8 on the board, 5.0 points higher and 19 ranks above.&lt;/li&gt;
&lt;li&gt;One task is 0.2 points, and a 95% interval on a single 80% run is about 3.5 points either side. Ranks 6 to 9 (80.6 to 80.2) are 2 tasks apart.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2520Ornith-1.5-397B%2522%252C%25222%2520Macaron-V1-Venti%2522%252C%25223%2520Macaron-V1-Coding-Venti%2522%252C%25224%2520Ornith-1.0-397B%2522%252C%25225%2520Beam-1.0-501B-A23B%2522%252C%25226%2520DeepSeek-V4-Pro%2522%252C%25227%2520MiniMax-M3%2522%252C%25228%2520Kimi-K2.6%2522%252C%25229%2520Inkling-Small%2522%252C%252210%2520DeepSeek-V4-Flash%2522%252C%252211%2520Ornith-1.5-35B-A3B%2522%252C%252212%2520MiMo-V2.5-Pro%2522%252C%252213%2520dots3-note-prev%2522%252C%252214%2520Hy3%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2522%252C%2522data%2522%253A%255B86%252C85.6%252C85.6%252C82.4%252C81.6%252C80.6%252C80.5%252C80.2%252C80.2%252C79%252C79%252C78.9%252C78.4%252C78%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522SWE-bench%2520Verified%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25205%252C%25202026%2520%2528orange%253A%2520open%2520PR%252C%2520blue%253A%2520on%2520main%253B%25200%2520verified%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A74%252C%2522max%2522%253A88%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520%2525%2520resolved%2520%2528swebench.com%2520Verified%2520table%2520max%253A%252079.2%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2520Ornith-1.5-397B%2522%252C%25222%2520Macaron-V1-Venti%2522%252C%25223%2520Macaron-V1-Coding-Venti%2522%252C%25224%2520Ornith-1.0-397B%2522%252C%25225%2520Beam-1.0-501B-A23B%2522%252C%25226%2520DeepSeek-V4-Pro%2522%252C%25227%2520MiniMax-M3%2522%252C%25228%2520Kimi-K2.6%2522%252C%25229%2520Inkling-Small%2522%252C%252210%2520DeepSeek-V4-Flash%2522%252C%252211%2520Ornith-1.5-35B-A3B%2522%252C%252212%2520MiMo-V2.5-Pro%2522%252C%252213%2520dots3-note-prev%2522%252C%252214%2520Hy3%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2522%252C%2522data%2522%253A%255B86%252C85.6%252C85.6%252C82.4%252C81.6%252C80.6%252C80.5%252C80.2%252C80.2%252C79%252C79%252C78.9%252C78.4%252C78%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522SWE-bench%2520Verified%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25205%252C%25202026%2520%2528orange%253A%2520open%2520PR%252C%2520blue%253A%2520on%2520main%253B%25200%2520verified%2529%2522%252C%2522fontSize%2522%253A13%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A74%252C%2522max%2522%253A88%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520%2525%2520resolved%2520%2528swebench.com%2520Verified%2520table%2520max%253A%252079.2%2529%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" alt="Top 14 rows of the SWE-bench Verified board on Hugging Face, orange for open pull requests and blue for main branch" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Orange bars are rows whose YAML file exists only in an open pull request. Blue bars are rows already on the model repo's main branch. The axis starts at 74, so bar lengths exaggerate the gaps; read the numbers.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does the SWE-bench Verified board on Hugging Face contain?
&lt;/h2&gt;

&lt;p&gt;SWE-bench Verified is a human-validated subset of 500 issue and pull request pairs from popular Python repositories. Evaluation runs the repo's unit tests against the patch an agent produces. Hugging Face lists it as a benchmark dataset, so a model repo can publish a result in &lt;code&gt;.eval_results/*.yaml&lt;/code&gt; and the dataset's board collects it.&lt;/p&gt;

&lt;p&gt;I pulled the board on November 5, 2026 (KST) from the JSON behind the leaderboard view. It has 78 rows. The dataset's own &lt;code&gt;eval.yaml&lt;/code&gt; defines two task ids, &lt;code&gt;swe_bench_%_resolved&lt;/code&gt; and &lt;code&gt;swe_bench_average_cost&lt;/code&gt;, and names the &lt;code&gt;swe-bench&lt;/code&gt; evaluation framework. It does not name an agent, a scaffold, a number of attempts or a context length. That matters because for this benchmark the scaffold is part of the measurement. A model alone does not edit a repository; an agent loop does, and each team brings its own.&lt;/p&gt;

&lt;p&gt;Of the 78 notes fields, 54 are empty. Nine name a harness or an official run. So most rows tell the reader nothing about how the number was produced, and the card behind each row is the only place to look.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is anything on the board verified?
&lt;/h2&gt;

&lt;p&gt;No row is. The &lt;code&gt;verified&lt;/code&gt; field is false on all 78. I fetched the discussion behind every row that carries a &lt;code&gt;pullRequest&lt;/code&gt; number.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Open PR by nielsr (Hugging Face staff account)&lt;/td&gt;
&lt;td&gt;23&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Open PR by SaylorTwift (Hugging Face staff account)&lt;/td&gt;
&lt;td&gt;28&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;File on the repo's main branch&lt;/td&gt;
&lt;td&gt;27&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Verified&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All 51 pull requests are open. The oldest was opened on February 16, 2026, which is 262 days before my pull; the median age is 153 days, and 36 of the 51 are older than 90 days. The top row is one of them: Ornith-1.5-397B (86.0) comes from a pull request opened on August 19, 2026 by a Hugging Face account, not by the model's owner.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Open%2520PR%2520by%2520nielsr%2522%252C%2522Open%2520PR%2520by%2520SaylorTwift%2522%252C%2522On%2520the%2520repo%2527s%2520main%2520branch%2522%252C%2522Verified%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rows%2522%252C%2522data%2522%253A%255B23%252C28%252C27%252C0%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Where%2520do%2520the%252078%2520SWE-bench%2520Verified%2520rows%2520come%2520from%253F%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A30%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rows%2520on%2520the%2520board%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Open%2520PR%2520by%2520nielsr%2522%252C%2522Open%2520PR%2520by%2520SaylorTwift%2522%252C%2522On%2520the%2520repo%2527s%2520main%2520branch%2522%252C%2522Verified%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rows%2522%252C%2522data%2522%253A%255B23%252C28%252C27%252C0%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Where%2520do%2520the%252078%2520SWE-bench%2520Verified%2520rows%2520come%2520from%253F%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A30%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rows%2520on%2520the%2520board%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" alt="Where the 78 SWE-bench Verified rows come from: open pull requests by two accounts, main branch, verified" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I am not criticizing the design. Opening a pull request with results taken from a model card is a reasonable way to bootstrap a board, and each file is labeled with its source. The cost is that the ranking blends results the owner has accepted with results someone else transcribed, and nothing in the rank column separates them. In the top ten, five rows are on main and five are open pull requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why do nine rows beat the official table's best?
&lt;/h2&gt;

&lt;p&gt;The swebench.com site publishes its own Verified table. The data file behind it has 180 entries, the newest dated February 26, 2026, and the highest value is 79.2, held by two agent submissions from December 2025 (Sonar Foundation Agent with Claude Opus 4.5, and live-SWE-agent with Claude Opus 4.5). Neither is marked as checked. The highest entry among the runs that use the site's bash-only mini-SWE-agent is 76.8 (Claude Opus 4.5, February 17, 2026).&lt;/p&gt;

&lt;p&gt;On the Hugging Face board, 9 rows are above 79.2 and 18 are above 76.8. None of that proves anything is wrong. Newer models are stronger, and the site's table has not moved since February. But the two tables are not measuring the same thing. The site's mini-SWE-agent runs give every model the same minimal agent with one bash tool. The board rows come from whatever harness each team used, and the cards say so when you read them:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Board rank&lt;/th&gt;
&lt;th&gt;Repo&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;What the card or note says about the setup&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Ornith-1.5-397B&lt;/td&gt;
&lt;td&gt;86.0&lt;/td&gt;
&lt;td&gt;OpenHands harness, temperature 1.0, top_p 0.95, 256k context, git history removed, network disabled&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Macaron-V1-Venti&lt;/td&gt;
&lt;td&gt;85.6&lt;/td&gt;
&lt;td&gt;Claude Code harness, retries up to 3 times only after an evaluation error (about 0.8%), reports the successful evaluated attempt&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;MiniMax-M3&lt;/td&gt;
&lt;td&gt;80.5&lt;/td&gt;
&lt;td&gt;Claude Code scaffolding, internal infrastructure, 4 runs averaged (in the row's note)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;MiniMax-M2.5&lt;/td&gt;
&lt;td&gt;75.8&lt;/td&gt;
&lt;td&gt;swebench.com mini-SWE-agent run, February 17, 2026&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;43&lt;/td&gt;
&lt;td&gt;Qwen3-Coder-Next&lt;/td&gt;
&lt;td&gt;70.6&lt;/td&gt;
&lt;td&gt;SWE-Agent as harness (in the row's note)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The anti-hacking measures in the rank 1 card are good practice. They also show that the number belongs to a specific environment. Among the top-ranked rows whose setup I could read, there are two different agent loops (OpenHands and Claude Code), and neither is the one the site uses.&lt;/p&gt;

&lt;h2&gt;
  
  
  How much does the harness move the same model's score?
&lt;/h2&gt;

&lt;p&gt;The board itself contains the answer, because some repos have several rows.&lt;/p&gt;

&lt;p&gt;NVIDIA's Nemotron-3-Super-120B card reports three SWE-bench Verified results for the same weights: 60.47 with the OpenHands harness, 59.20 with OpenCode and 53.73 with Codex. That is 6.74 points between harnesses, or about 34 tasks. The board shows them at ranks 60, 63 and 68. The &lt;code&gt;RedHatAI&lt;/code&gt; repo with the same name carries the same three rows, with the source URL pointing at NVIDIA's card, at ranks 61, 64 and 69. Six rows, three measurements, two repos.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522gpt-oss-20b%253A%2520low%2520vs%2520high%2520reasoning%2522%252C%2522gpt-oss-120b%253A%2520low%2520vs%2520high%2520reasoning%2522%252C%2522Nemotron-3-Super%253A%2520Codex%2520vs%2520OpenHands%2520harness%2522%252C%2522openPangu-2.0-Flash%253A%2520non-thinking%2520vs%2520thinking%2522%252C%2522GLM-5%253A%2520swebench.com%2520run%2520vs%2520maker%2527s%2520card%2522%252C%2522One%2520task%2520out%2520of%2520500%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522points%2522%252C%2522data%2522%253A%255B23.3%252C14.5%252C6.74%252C5.5%252C5.0%252C0.2%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Same%2520repo%252C%2520several%2520rows%253A%2520spread%2520between%2520its%2520own%2520rows%2520%2528points%2529%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A25%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522score%2520difference%2520on%2520the%2520same%2520500%2520tasks%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522gpt-oss-20b%253A%2520low%2520vs%2520high%2520reasoning%2522%252C%2522gpt-oss-120b%253A%2520low%2520vs%2520high%2520reasoning%2522%252C%2522Nemotron-3-Super%253A%2520Codex%2520vs%2520OpenHands%2520harness%2522%252C%2522openPangu-2.0-Flash%253A%2520non-thinking%2520vs%2520thinking%2522%252C%2522GLM-5%253A%2520swebench.com%2520run%2520vs%2520maker%2527s%2520card%2522%252C%2522One%2520task%2520out%2520of%2520500%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522points%2522%252C%2522data%2522%253A%255B23.3%252C14.5%252C6.74%252C5.5%252C5.0%252C0.2%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Same%2520repo%252C%2520several%2520rows%253A%2520spread%2520between%2520its%2520own%2520rows%2520%2528points%2529%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A25%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522score%2520difference%2520on%2520the%2520same%2520500%2520tasks%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" alt="Spread between rows of the same repo on the SWE-bench Verified board: reasoning effort, harness, mode, source" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The gpt-oss repos use a different axis. OpenAI's model card gives low, medium and high reasoning effort, and the board lists all three. gpt-oss-120b has 47.9, 52.6 and 62.4 at ranks 75, 72 and 56. gpt-oss-20b has 37.4, 53.2 and 60.7 at ranks 77, 70 and 59. The spread is 14.5 points for the larger model and 23.3 for the smaller one. Nobody is misreporting here; the effort level is in the file name. But a reader who looks up gpt-oss-20b gets a different answer depending on which of its three rows they land on, and the best and worst rows are 18 ranks apart. openPangu-2.0-Flash has the same shape in smaller form: 63.1 thinking and 57.6 non-thinking, 5.5 points.&lt;/p&gt;

&lt;p&gt;GLM-5 is the cleanest case. Its single YAML file contains two entries. One is 72.8, sourced to swebench.com with the note "high reasoning, official". The other is 77.8, sourced to the model card with the note "Z.ai reported number". The board shows 77.8 at rank 15 and 72.8 at rank 34, 19 ranks and 25 tasks apart.&lt;/p&gt;

&lt;p&gt;The 72.8 matches the swebench.com data file exactly: mini-SWE-agent v2.0.0 with glm-5 at high effort, dated February 17, 2026. The same is true of MiniMax-M2.5 (75.8) and Kimi-K2.5 (70.8). So these three rows are the only ones on the board I could trace to a run under a common harness, and for GLM-5 that run scores 5.0 points under the maker's own figure. The maker's number may well come from a better scaffold, and I did not try to reproduce either one. The point is that the board puts both on the same scale.&lt;/p&gt;

&lt;h2&gt;
  
  
  Are all the rows distinct models?
&lt;/h2&gt;

&lt;p&gt;Not always. Two more cases.&lt;/p&gt;

&lt;p&gt;Ranks 2 and 3 are &lt;code&gt;Macaron-V1-Venti&lt;/code&gt; (on main) and &lt;code&gt;Macaron-V1-Coding-Venti&lt;/code&gt; (an open pull request by nielsr), both 85.6. I did not check whether the two repos hold the same weights. If they do, two of the top three ranks are one result. If they do not, the identical value is a coincidence worth a footnote.&lt;/p&gt;

&lt;p&gt;Ranks 46 and 50 are &lt;code&gt;poolside/Laguna-XS.2&lt;/code&gt; at 69.9 and &lt;code&gt;MuVeraAI/Laguna-XS.2&lt;/code&gt; at 68.2. The second repo's YAML gives the poolside model card as its source. That card's table currently says 69.9, mean pass@1 over 4 runs, with a 262,144 token context. The MuVeraAI repo carries its own copy of the card, last modified April 28, 2026, with a 131,072 token context and 68.2 in the same table. Its downloads field reads 16, against 905,143 for the poolside repo. So the 68.2 looks like an earlier version of a card that was later updated, copied into another repo, and given its own rank. I can see the mismatch between the source link and the number; I cannot see why the copy exists.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can 500 tasks separate these rows?
&lt;/h2&gt;

&lt;p&gt;The benchmark has 500 tasks, so one task is 0.2 points. The arithmetic below is a sampling calculation for a single run on a fixed set. Prompt and scaffold variance is larger, as the Nemotron rows show.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Quantity&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;One task&lt;/td&gt;
&lt;td&gt;0.2 points&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rank 6 to rank 9 (80.6 to 80.2)&lt;/td&gt;
&lt;td&gt;0.4 points, 2 tasks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rank 1 to rank 6 (86.0 to 80.6)&lt;/td&gt;
&lt;td&gt;5.4 points, 27 tasks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;95% half-width at 80% accuracy, n=500 (illustrative)&lt;/td&gt;
&lt;td&gt;3.51 points&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;95% half-width at 86% accuracy, n=500 (illustrative)&lt;/td&gt;
&lt;td&gt;3.04 points&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nemotron-3-Super, Codex vs OpenHands&lt;/td&gt;
&lt;td&gt;6.74 points&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Ranks 8 and 9 are tied at 80.2, and rank 6 is 2 tasks above them. A paired comparison on the same 500 tasks has a narrower interval than the independent one I show, but it needs per-task results, which a board row does not carry. So the top 14 rows, from 86.0 to 78.0, contain only a few differences that I would call larger than noise, and none of them is established by a verified run.&lt;/p&gt;

&lt;h2&gt;
  
  
  What about the benchmark itself?
&lt;/h2&gt;

&lt;p&gt;In February 2026 OpenAI said it would no longer treat SWE-bench Verified as a measure of frontier coding ability. Third-party summaries of that post, from Pebblous and Decrypt among others, report that of 138 hard tasks audited, 59.4% had flawed tests or problem statements, 35.5% of them from overly narrow tests, and that some tasks showed signs of leakage from training data. OpenAI's page returned HTTP 403 to my fetcher, so I am relaying the summaries and have not read the original; check it before you cite the percentages. At the time, OpenAI pointed to SWE-bench Pro as the alternative. I looked at the Pro board in an earlier post in this series, and it has its own mixing problem.&lt;/p&gt;

&lt;p&gt;For a board reader, the implication is practical. A Verified score near 80 or 86 is a number on a benchmark whose main early adopter has questioned the grading. It can still rank models roughly, and the board is still useful as an index of claims. It is a weak basis for a procurement decision or a claim that one open model beats another by two points.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do I pull the board and check this myself?
&lt;/h2&gt;

&lt;p&gt;The script uses only the standard library. It counts rows by source, finds repos with several rows, and compares the board with the swebench.com data file.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/datasets/SWE-bench/SWE-bench_Verified/leaderboard&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;repos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;multi&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;repos&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;authors&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;pr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pullRequest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;pr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/models/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/discussions/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;pr&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;authors&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;author&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;| repos:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;repos&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;| verified:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verified&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;open/merged PR rows by author:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;authors&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;| main-branch rows:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pullRequest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;repos with 2+ rows:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;multi&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;carrying&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;multi&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;empty notes:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;notes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;one task = %.1f points&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;acc&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.70&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.86&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;accuracy %.2f: 95%% half-width = %.2f points (illustrative, independent runs)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.96&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;lb&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://raw.githubusercontent.com/swe-bench/swe-bench.github.io/master/data/leaderboards.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}),&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;ver&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;lb&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;leaderboards&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Verified&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;results&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;top&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resolved&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ver&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;swebench.com Verified table: %d entries, max %.1f, newest %s&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ver&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ver&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;HF rows above that max:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;top&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On November 5, 2026 it printed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;rows: 78 | repos: 68 | verified: 0
open/merged PR rows by author: {('open', 'SaylorTwift'): 28, ('open', 'nielsr'): 23} | main-branch rows: 27
repos with 2+ rows: 6 carrying 16 rows
empty notes: 54
one task = 0.2 points
accuracy 0.70: 95% half-width = 4.02 points (illustrative, independent runs)
accuracy 0.80: 95% half-width = 3.51 points (illustrative, independent runs)
accuracy 0.86: 95% half-width = 3.04 points (illustrative, independent runs)
swebench.com Verified table: 180 entries, max 79.2, newest 2026-02-26
HF rows above that max: 9
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The board endpoint is undocumented, so field names may change. To check a row, open its &lt;code&gt;pullRequest&lt;/code&gt; discussion, read the YAML, and then read the footnotes of the source card. Notes that name a harness are rare; the card usually has them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist for reading a Verified score
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Is the row verified, or an open pull request? Look at the badge, not the rank.&lt;/li&gt;
&lt;li&gt;Which harness and how many attempts? If the YAML note is empty, read the card footnotes.&lt;/li&gt;
&lt;li&gt;Does the same repo appear more than once? Pick the row whose setup you will actually run.&lt;/li&gt;
&lt;li&gt;Is a copy or re-upload carrying someone else's number? Follow the source URL and compare.&lt;/li&gt;
&lt;li&gt;Is the gap bigger than a few tasks and bigger than the harness spread? On this board, same-repo spreads run from 5 to 23 points.&lt;/li&gt;
&lt;li&gt;Does the benchmark still deserve the weight? Read what its own maintainers and its largest early adopter have said about the tests.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;How many models are on the Hugging Face SWE-bench Verified leaderboard?&lt;/strong&gt;&lt;br&gt;
On November 5, 2026 the board has 78 rows from 68 repos. Six repos have more than one row, carrying 16 rows between them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the Hugging Face SWE-bench Verified leaderboard verified?&lt;/strong&gt;&lt;br&gt;
No row has &lt;code&gt;verified&lt;/code&gt; set to true. 51 rows come from open pull requests, opened by two Hugging Face staff accounts, and 27 are on the model repos' main branches.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why does the Hugging Face board show higher scores than swebench.com?&lt;/strong&gt;&lt;br&gt;
Nine rows are above the 79.2 maximum in the site's Verified data file, last updated February 26, 2026. The board rows are self-reported under each team's own harness, while the site's mini-SWE-agent runs use one minimal bash-only agent. Different harnesses move scores by several points, as the three Nemotron rows show.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does the agent harness change a SWE-bench Verified score?&lt;/strong&gt;&lt;br&gt;
Yes. NVIDIA's Nemotron-3-Super-120B card gives 60.47 with OpenHands, 59.20 with OpenCode and 53.73 with Codex. GLM-5 has 77.8 from its maker and 72.8 from a swebench.com mini-SWE-agent run.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Should I still use SWE-bench Verified to compare coding models?&lt;/strong&gt;&lt;br&gt;
As a rough index, yes. As a deciding number, no. OpenAI said in February 2026 that it no longer treats it as a frontier measure, and the board mixes harnesses, efforts and copies. Run the model in your own agent on your own tasks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/gsm8k-on-hugging-face-4-of-the-top-12-rows-show-a-sibling-checkpoints-score-and-20-of-28-rows-1e50"&gt;GSM8K on Hugging Face: 4 of the Top 12 Rows Show a Sibling Checkpoint's Score, and 20 of 28 Rows Are Unmerged PRs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-pro-has-two-versions-what-the-hugging-face-leaderboards-47-rows-mix-together-37co"&gt;SWE-Bench Pro Has Two Versions: What the Hugging Face Leaderboard's 47 Rows Mix Together&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>GSM8K on Hugging Face: 4 of the Top 12 Rows Show a Sibling Checkpoint's Score, and 20 of 28 Rows Are Unmerged PRs</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Tue, 03 Nov 2026 14:11:34 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/gsm8k-on-hugging-face-4-of-the-top-12-rows-show-a-sibling-checkpoints-score-and-20-of-28-rows-1e50</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/gsm8k-on-hugging-face-4-of-the-top-12-rows-show-a-sibling-checkpoints-score-and-20-of-28-rows-1e50</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On November 3, 2026 the Hugging Face GSM8K board lists &lt;strong&gt;28 rows from 28 repos. 20 come from pull requests that are still open, 8 sit on the repo's main branch, and 0 are verified.&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Four of the top 12 rows show a number that the model card reports for a different checkpoint of the same family.&lt;/strong&gt; Rank 1 (99.6) is the MiMo-V2.5-Pro &lt;em&gt;Base&lt;/em&gt; score, registered on the post-trained repo. Rank 2 (96.8) is the Llama 3.1 405B &lt;em&gt;Instruct&lt;/em&gt; score, registered on the &lt;em&gt;base&lt;/em&gt; repo. Ranks 7 and 12 are DeepSeek base scores on the post-trained repos.&lt;/li&gt;
&lt;li&gt;The numbers are real self-reports. The labels are what is off, and the board has no column that would show it.&lt;/li&gt;
&lt;li&gt;GSM8K has 1,319 test problems, so one problem is 0.076 points. The rank 2 to rank 3 gap (0.74) is 10 problems, and a 95% interval at 95% accuracy is about 1.2 points wide on each side.&lt;/li&gt;
&lt;li&gt;A score of 99.6 means about 5 misses. The GSM8K-Platinum project relabeled 10 answers in the same test set. If those relabels are right, a score that high implies the model matched at least 5 of the 10 keys they call wrong.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2520MiMo-V2.5-Pro%2522%252C%25222%2520Llama-3.1-405B%2522%252C%25223%2520LLaDA2.0-flash%2522%252C%25224%2520granite-4.1-30b%2522%252C%25225%2520Motif-3-Base%2522%252C%25226%2520Llama-3.2-90B-Vision-Instruct%2522%252C%25227%2520DeepSeek-V4-Pro%2522%252C%25228%2520granite-4.1-8b%2522%252C%25229%2520MBD-Math-LLaDA2-mini-16B%2522%252C%252210%2520Phi-3-medium-4k-instruct%2522%252C%252211%2520Qwen2-72B%2522%252C%252212%2520DeepSeek-V3%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2522%252C%2522data%2522%253A%255B99.6%252C96.8%252C96.06%252C94.16%252C93.93%252C93.1%252C92.6%252C92.49%252C91.96%252C91.0%252C89.5%252C89.3%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522GSM8K%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25203%252C%25202026%2520%2528red%253A%2520card%2520gives%2520this%2520number%2520for%2520the%2520sibling%2520checkpoint%2529%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A80%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520score%252C%2520%2525%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2520MiMo-V2.5-Pro%2522%252C%25222%2520Llama-3.1-405B%2522%252C%25223%2520LLaDA2.0-flash%2522%252C%25224%2520granite-4.1-30b%2522%252C%25225%2520Motif-3-Base%2522%252C%25226%2520Llama-3.2-90B-Vision-Instruct%2522%252C%25227%2520DeepSeek-V4-Pro%2522%252C%25228%2520granite-4.1-8b%2522%252C%25229%2520MBD-Math-LLaDA2-mini-16B%2522%252C%252210%2520Phi-3-medium-4k-instruct%2522%252C%252211%2520Qwen2-72B%2522%252C%252212%2520DeepSeek-V3%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2522%252C%2522data%2522%253A%255B99.6%252C96.8%252C96.06%252C94.16%252C93.93%252C93.1%252C92.6%252C92.49%252C91.96%252C91.0%252C89.5%252C89.3%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523c53030%2522%252C%2522%2523c53030%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522GSM8K%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25203%252C%25202026%2520%2528red%253A%2520card%2520gives%2520this%2520number%2520for%2520the%2520sibling%2520checkpoint%2529%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A80%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520score%252C%2520%2525%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" alt="Top 12 rows of the GSM8K board on Hugging Face, with the four rows whose number belongs to a sibling checkpoint in red" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In the chart, red marks rows where the model card gives this number for a sibling checkpoint, blue marks a row whose card table matches the repo (LLaDA2.0-flash), and grey marks rows I did not check against the card.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does the GSM8K board on Hugging Face contain?
&lt;/h2&gt;

&lt;p&gt;GSM8K is a set of grade-school math word problems with a train split of 7,473 and a test split of 1,319. Hugging Face lists &lt;code&gt;openai/gsm8k&lt;/code&gt; as a benchmark dataset, so a model repo can publish a result in a &lt;code&gt;.eval_results/*.yaml&lt;/code&gt; file and the dataset's board collects it.&lt;/p&gt;

&lt;p&gt;I pulled the board on November 3, 2026 (KST) from the JSON behind the leaderboard view. It has 28 rows. Each row is from a different repo, &lt;code&gt;verified&lt;/code&gt; is false on every row, and the top of the page reads like a clean ranking: 99.6, 96.8, 96.06, 94.16, 93.93.&lt;/p&gt;

&lt;p&gt;The dataset's own &lt;code&gt;eval.yaml&lt;/code&gt; describes a specific protocol: an inspect-ai task, a zero-shot prompt template that asks for a boxed final answer, four epochs reduced to pass@1, and a model-graded scorer. The rows do not follow one protocol. Of the 28 notes, 15 are empty. Where a note gives a shot count it ranges across 0, 4, 5 and 8. Two rows from one lab give a complete setup (8-shot, temperature 0, n=1319, lm-evaluation-harness), and one row reports a strict-match score over 1,075 of the 1,319 problems. So I treat each row as a claim with a source link, not as a measurement the board produced.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why does rank 1 belong to a different checkpoint?
&lt;/h2&gt;

&lt;p&gt;The top row is &lt;code&gt;XiaomiMiMo/MiMo-V2.5-Pro&lt;/code&gt; at 99.6. Its model card has a section titled "Base Model Evaluation", and the GSM8K row there reads 8-shot with these values: 99.6 for MiMo-V2.5-Pro Base, 83.3 for MiMo-V2.5 Base, 92.6 for DeepSeek-V4-Pro Base. That is the only place GSM8K appears in the card. The card's own download table lists two repos: &lt;code&gt;MiMo-V2.5-Pro&lt;/code&gt; and &lt;code&gt;MiMo-V2.5-Pro-Base&lt;/code&gt;, and the second one exists on the Hub.&lt;/p&gt;

&lt;p&gt;So the board shows the base-model score on the post-trained repo. The pull request that registered it says the results "were extracted from the model card". It was opened on April 27, 2026 and is still open. Nothing in the YAML says "base", and the 8-shot setting is not in the note, which is empty.&lt;/p&gt;

&lt;p&gt;The card is not hiding anything. It labels its table as Base. The distinction is lost only in the step that turns a card into a board row.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is that one slip, or does it repeat?
&lt;/h2&gt;

&lt;p&gt;I checked the sources for the top rows. It repeats in four of the top 12, and in both directions.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Board rank&lt;/th&gt;
&lt;th&gt;Repo on the board&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Where the card puts this number&lt;/th&gt;
&lt;th&gt;Sibling repo&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;MiMo-V2.5-Pro&lt;/td&gt;
&lt;td&gt;99.6&lt;/td&gt;
&lt;td&gt;"Base Model Evaluation" table, Pro Base column, 8-shot&lt;/td&gt;
&lt;td&gt;MiMo-V2.5-Pro-Base&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Llama-3.1-405B&lt;/td&gt;
&lt;td&gt;96.8&lt;/td&gt;
&lt;td&gt;"Instruction tuned models" table, 405B Instruct column, 8-shot CoT, em_maj1@1&lt;/td&gt;
&lt;td&gt;Llama-3.1-405B-Instruct&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;DeepSeek-V4-Pro&lt;/td&gt;
&lt;td&gt;92.6&lt;/td&gt;
&lt;td&gt;Base table, DeepSeek-V4-Pro-Base column, 8-shot&lt;/td&gt;
&lt;td&gt;DeepSeek-V4-Pro-Base&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;DeepSeek-V3&lt;/td&gt;
&lt;td&gt;89.3&lt;/td&gt;
&lt;td&gt;"Base Model" section, DeepSeek-V3 column, 8-shot&lt;/td&gt;
&lt;td&gt;DeepSeek-V3-Base&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Rank 2 is the mirror image. The Llama 3.1 model card covers every size and both variants. Its "Base pretrained models" table has no GSM8K row. Its "Instruction tuned models" table has one, and the 405B Instruct value is 96.8. The pull request is titled "Add GSM8K eval result (96.8, 8-shot CoT)" and attaches that number to the base repo &lt;code&gt;meta-llama/Llama-3.1-405B&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;For DeepSeek-V3 the card has a Base Model section with GSM8K at 89.3 and a Chat Model section with no GSM8K row. For DeepSeek-V4-Pro the table header names the three columns DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base, and 92.6 is the last one.&lt;/p&gt;

&lt;p&gt;Why does it matter if the number is the same? Because the board is read by repo name. Someone choosing a model for math word problems sees MiMo-V2.5-Pro at the top and downloads that repo. The card reports no GSM8K number for that checkpoint. If the four rows were moved to the sibling repos, the post-trained MiMo, DeepSeek-V4-Pro and DeepSeek-V3 would have no GSM8K row at all, and the Llama 405B base repo would lose its rank 2.&lt;/p&gt;

&lt;p&gt;The correct placement would not change the ordering of the numbers: 99.6 would still lead, from the right repo. It would change which model each number tells you about.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Open%2520PR%2520from%2520an%2520HF-flagged%2520account%2522%252C%2522Open%2520PR%2520from%2520a%2520bot%2520account%2522%252C%2522On%2520the%2520repo%2527s%2520main%2520branch%2522%252C%2522Verified%2520%2528HF%2520Jobs%2520run%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rows%2522%252C%2522data%2522%253A%255B19%252C1%252C8%252C0%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Where%2520do%2520the%252028%2520GSM8K%2520rows%2520come%2520from%253F%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A20%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rows%2520on%2520the%2520board%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Open%2520PR%2520from%2520an%2520HF-flagged%2520account%2522%252C%2522Open%2520PR%2520from%2520a%2520bot%2520account%2522%252C%2522On%2520the%2520repo%2527s%2520main%2520branch%2522%252C%2522Verified%2520%2528HF%2520Jobs%2520run%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522rows%2522%252C%2522data%2522%253A%255B19%252C1%252C8%252C0%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Where%2520do%2520the%252028%2520GSM8K%2520rows%2520come%2520from%253F%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A20%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rows%2520on%2520the%2520board%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" alt="Where the 28 GSM8K rows come from: open pull requests, main branch, verified" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where do the rows come from, and what does "verified" mean here?
&lt;/h2&gt;

&lt;p&gt;I fetched the discussion for every row that carries a &lt;code&gt;pullRequest&lt;/code&gt; number, and listed &lt;code&gt;.eval_results&lt;/code&gt; on the main branch of every repo.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Open PR from an account flagged as Hugging Face (julien-c 9, nielsr 6, SaylorTwift 3, burtenshaw 1)&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Open PR from a bot account (EvalEvalBot)&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;File on the repo's main branch&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Verified&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All 20 pull requests are open, and for all 20 the YAML file is absent from main. For all 8 other rows it is present. Four of the 20 pull requests were opened on November 2, one day before I pulled the board, and one of them produced rank 3 (&lt;code&gt;LLaDA2.0-flash&lt;/code&gt;, 96.06). The board moves when someone opens a pull request, with no action from the model's owner.&lt;/p&gt;

&lt;p&gt;The Hugging Face documentation defines the badges. "Community" means a result submitted via an open PR that is not merged to main. "Verified" means a valid verify token, which requires the evaluation to have run in HF Jobs with inspect-ai. So 20 of 28 rows are community results by definition, and no row meets the verified bar. The documentation says community scores are visible while the PR is open. That is a reasonable design for bootstrapping a board. It also means a reader has to look at the badge to know which rows the model's owner stands behind, and a screenshot of the ranking does not show it.&lt;/p&gt;

&lt;p&gt;The eight main-branch rows are not uniformly clean either. Two Mellum2 repos show the same 81.73. The YAML on the &lt;code&gt;Base&lt;/code&gt; repo states that its numbers were measured on the pre-extension &lt;code&gt;Base-Pretrain&lt;/code&gt; checkpoint, so it is one measurement displayed twice. This one is disclosed in the file, and it still occupies two ranks.&lt;/p&gt;

&lt;h2&gt;
  
  
  How much can 1,319 problems separate?
&lt;/h2&gt;

&lt;p&gt;GSM8K is nearly saturated at the top, so the ordering is decided by a handful of problems.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Quantity&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;One problem&lt;/td&gt;
&lt;td&gt;0.076 points&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rank 1 to rank 2 gap (99.6 to 96.8)&lt;/td&gt;
&lt;td&gt;2.8 points, about 37 problems&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rank 2 to rank 3 gap (96.8 to 96.06)&lt;/td&gt;
&lt;td&gt;0.74 points, about 10 problems&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spread of ranks 3 to 6 (96.06 to 93.10)&lt;/td&gt;
&lt;td&gt;2.96 points&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;95% half-width at 95% accuracy, n=1,319&lt;/td&gt;
&lt;td&gt;1.18 points (illustrative)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;95% half-width at 99% accuracy, n=1,319&lt;/td&gt;
&lt;td&gt;0.54 points (illustrative)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The interval rows are a sampling calculation for a single run on a fixed test set. They ignore prompt variation, which is larger when rows use different shot counts. Even so, ranks 3 to 6 sit inside a band that a single run cannot separate, and the rows within it come from different protocols: 8-shot CoT for some, no note for others.&lt;/p&gt;

&lt;p&gt;One other check is exact arithmetic. The two rows with a full setup report 84.9886 and 78.4685, and with n=1319 these are 1,121 and 1,035 correct answers. ProKope-7B's note says 1,075 of 1,319, which is 81.50. Those rows can be audited back to a count. Most rows cannot.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522One%2520problem%2520out%2520of%25201%252C319%2522%252C%252295%2525%2520half-width%2520at%252099%2525%2520accuracy%2520%2528illustrative%2529%2522%252C%2522Ceiling%2520lost%2520if%2520the%252010%2520relabeled%2520keys%2520are%2520wrong%2522%252C%2522Rank%25202%2520to%2520rank%25203%2520gap%2522%252C%252295%2525%2520half-width%2520at%252095%2525%2520accuracy%2520%2528illustrative%2529%2522%252C%2522Rank%25201%2520to%2520rank%25202%2520gap%2522%252C%2522Spread%2520of%2520ranks%25203%2520to%25206%2522%252C%2522Spread%2520of%2520the%2520top%25205%2520rows%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522points%2522%252C%2522data%2522%253A%255B0.076%252C0.54%252C0.76%252C0.74%252C1.18%252C2.8%252C2.96%252C5.67%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522How%2520big%2520is%2520each%2520effect%252C%2520in%2520score%2520points%253F%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A6%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522points%2520on%2520a%25200-100%2520scale%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522One%2520problem%2520out%2520of%25201%252C319%2522%252C%252295%2525%2520half-width%2520at%252099%2525%2520accuracy%2520%2528illustrative%2529%2522%252C%2522Ceiling%2520lost%2520if%2520the%252010%2520relabeled%2520keys%2520are%2520wrong%2522%252C%2522Rank%25202%2520to%2520rank%25203%2520gap%2522%252C%252295%2525%2520half-width%2520at%252095%2525%2520accuracy%2520%2528illustrative%2529%2522%252C%2522Rank%25201%2520to%2520rank%25202%2520gap%2522%252C%2522Spread%2520of%2520ranks%25203%2520to%25206%2522%252C%2522Spread%2520of%2520the%2520top%25205%2520rows%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522points%2522%252C%2522data%2522%253A%255B0.076%252C0.54%252C0.76%252C0.74%252C1.18%252C2.8%252C2.96%252C5.67%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523c53030%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522How%2520big%2520is%2520each%2520effect%252C%2520in%2520score%2520points%253F%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A6%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522points%2520on%2520a%25200-100%2520scale%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" alt="How big is each effect: one problem, intervals, ceiling lost to relabeled keys, gaps between ranks" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Can a 99.6 be right if some answer keys are wrong?
&lt;/h2&gt;

&lt;p&gt;This part is arithmetic with one assumption, so I mark it as a question to check, not a finding.&lt;/p&gt;

&lt;p&gt;The GSM8K-Platinum project (MIT, Gradient Science) reviewed the 1,319 test problems. Its dataset card reports 219 flagged by models, 110 rejected as poorly written, 10 relabeled, 99 verified, leaving 1,209 questions. The 10 relabeled items are answer keys that the authors found to be wrong.&lt;/p&gt;

&lt;p&gt;Take a score of 99.6 on the original test set. 1,314 of 1,319 rounds to 99.6, so that is about 5 misses. Suppose the 10 relabels are correct and scoring is exact match against the original key. A model that solves an item correctly then loses it, because the key is wrong. A model that is right on all 10 would lose all 10 and top out at 99.24. To score 99.6 it must have matched the key on at least 5 of the 10, which means it gave the answer the key says, not the answer the authors say is right.&lt;/p&gt;

&lt;p&gt;Three explanations fit: the relabels are wrong for some of those items, the scoring is more lenient than exact match, or the model has seen the original keys. I have not run the model, and a base model's 8-shot number with no note on answer extraction does not let me tell these apart. What I can say is that for a benchmark with known key errors, a score above the ceiling is a reason to read the setup, not a reason to rank it first. Platinum also reports that cleaning the set separates models that look tied on the original, which is the same point from the other side.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do I pull the board and classify rows myself?
&lt;/h2&gt;

&lt;p&gt;This script needs only the standard library. It counts rows by source and prints the sampling arithmetic used above.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;DATASET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;openai/gsm8k&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/datasets/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;DATASET&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/leaderboard&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;by_source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;pr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pullRequest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;pr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;disc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/models/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/discussions/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;pr&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;by_source&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;open PR&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;disc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;open&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;disc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;by_source&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;main branch&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;| verified:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verified&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;by_source&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1319&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;one problem = %.3f points&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;acc&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.95&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.99&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;se&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;accuracy %.2f: 95%% half-width = %.2f points&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;acc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.96&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;se&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ceiling if 10 keys are wrong: %.2f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On November 3, 2026 it printed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;rows: 28 | verified: 0
{'open PR': 20, 'main branch': 8}
one problem = 0.076 points
accuracy 0.95: 95% half-width = 1.18 points
accuracy 0.99: 95% half-width = 0.54 points
ceiling if 10 keys are wrong: 99.24
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The board endpoint is undocumented, so the field names may change. To check a row, open its &lt;code&gt;pullRequest&lt;/code&gt; discussion, read the YAML, then open the source card and find the table the number came from.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist for reading a saturated board
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Open the badge before the rank.&lt;/strong&gt; Community means an open PR, and verified means a run in HF Jobs. Neither is a statement from the model's owner.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Match the checkpoint, not the family.&lt;/strong&gt; Find the card table the number came from and read its column header. Base, instruct and chat checkpoints sit in different tables.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ask how many problems separate two rows.&lt;/strong&gt; On GSM8K, 10 problems is 0.76 points.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prefer rows that state their setup.&lt;/strong&gt; Shot count, temperature, n and the answer-extraction rule decide the score more than rank does.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check for the same measurement on two repos.&lt;/strong&gt; A copied number is one result, not two.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Distrust a score above the key ceiling.&lt;/strong&gt; If a benchmark has known label errors, read the evaluation setup before you trust the lead.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;How many GSM8K results on Hugging Face are verified?&lt;/strong&gt;&lt;br&gt;
None. On November 3, 2026 all 28 rows on the board show verified as false. Verified requires a valid verify token from an inspect-ai run in HF Jobs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the 99.6 at rank 1 of the GSM8K board wrong?&lt;/strong&gt;&lt;br&gt;
Nothing here shows the number is wrong. It is the MiMo-V2.5-Pro Base score from the model card's Base Model table, 8-shot. The problem is that it is registered on the post-trained repo, which has its own separate row in the card's download table.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What does "community" mean on a Hugging Face leaderboard row?&lt;/strong&gt;&lt;br&gt;
It means the result was submitted through an open pull request that has not been merged to the repo's main branch. In the GSM8K board, 20 of 28 rows are in that state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why is the Llama 3.1 405B base repo ranked second on GSM8K?&lt;/strong&gt;&lt;br&gt;
Its row carries 96.8, which the shared model card gives for Llama 3.1 405B Instruct, 8-shot CoT. The card has no GSM8K row in its base table. The instruct repo is a separate repo.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How many problems does one point of GSM8K accuracy represent?&lt;/strong&gt;&lt;br&gt;
GSM8K has 1,319 test problems, so one point is about 13 problems and one problem is 0.076 points.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is GSM8K-Platinum?&lt;/strong&gt;&lt;br&gt;
It is a revised version of the GSM8K test set from Gradient Science. Of 1,319 test problems it rejected 110, relabeled 10 and kept 1,209. It exists because the original set has ambiguous questions and wrong keys.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/gpqa-diamond-has-198-questions-what-the-hugging-face-leaderboards-115-rows-can-and-cannot-rank-2dlm"&gt;GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/how-hugging-face-official-benchmark-leaderboards-actually-work-evalresults-yaml-the-basemodel-29bo"&gt;How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;GSM8K dataset and board: &lt;a href="https://huggingface.co/datasets/openai/gsm8k" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/openai/gsm8k&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GSM8K evaluation definition (&lt;code&gt;eval.yaml&lt;/code&gt;): &lt;a href="https://huggingface.co/datasets/openai/gsm8k/blob/main/eval.yaml" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/openai/gsm8k/blob/main/eval.yaml&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Hugging Face documentation, Evaluation Results (badges, community PRs): &lt;a href="https://huggingface.co/docs/hub/eval-results" rel="noopener noreferrer"&gt;https://huggingface.co/docs/hub/eval-results&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GSM8K-Platinum dataset card: &lt;a href="https://huggingface.co/datasets/madrylab/gsm8k-platinum" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/madrylab/gsm8k-platinum&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GSM8K-Platinum write-up: &lt;a href="https://gradientscience.org/gsm8k-platinum/" rel="noopener noreferrer"&gt;https://gradientscience.org/gsm8k-platinum/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Rank 1 model card (Base Model Evaluation table): &lt;a href="https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro" rel="noopener noreferrer"&gt;https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Rank 1 registering pull request: &lt;a href="https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro/discussions/1" rel="noopener noreferrer"&gt;https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro/discussions/1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Llama 3.1 model card (base and instruction tuned tables): &lt;a href="https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md" rel="noopener noreferrer"&gt;https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Rank 2 registering pull request: &lt;a href="https://huggingface.co/meta-llama/Llama-3.1-405B/discussions/37" rel="noopener noreferrer"&gt;https://huggingface.co/meta-llama/Llama-3.1-405B/discussions/37&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;DeepSeek-V4-Pro model card: &lt;a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro" rel="noopener noreferrer"&gt;https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;DeepSeek-V3 model card: &lt;a href="https://huggingface.co/deepseek-ai/DeepSeek-V3" rel="noopener noreferrer"&gt;https://huggingface.co/deepseek-ai/DeepSeek-V3&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All counts in this article are my own measurements from the board JSON, pull request discussions, dataset cards and the cited model cards on November 3, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. Which table a number came from is my reading of each card, and the board endpoint is undocumented and may change.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>Terminal-Bench 2.0 vs 2.1 on Hugging Face: A 2.1 Score Is Ranked First on the 2.0 Board</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Mon, 02 Nov 2026 13:46:59 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/terminal-bench-20-vs-21-on-hugging-face-a-21-score-is-ranked-first-on-the-20-board-2750</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/terminal-bench-20-vs-21-on-hugging-face-a-21-score-is-ranked-first-on-the-20-board-2750</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On November 2, 2026 the Hugging Face Terminal-Bench boards list &lt;strong&gt;39 rows for version 2.0 and 42 rows for 2.1, and 0 of the 81 are marked verified&lt;/strong&gt;. Both versions have 89 tasks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rank 1 of the 2.0 board (87.6) is a 2.1 score.&lt;/strong&gt; The model card says "TerminalBench 2.1", the result file is named &lt;code&gt;terminal_bench_2_1.yaml&lt;/code&gt;, and it is registered under the 2.0 dataset. The next row is 69.0, so the lead shown is 18.6 points. On the 2.1 board the same number would tie for rank 5.&lt;/li&gt;
&lt;li&gt;That matters because the benchmark maintainers fixed 28 of the 89 tasks between versions, and their own release post reports a gain of up to 12.1% for one unchanged agent and model pair. A version shift of that size is larger than the whole spread of the top 9 rows on the 2.1 board (5.4 points).&lt;/li&gt;
&lt;li&gt;Agent harness labels are patchy: &lt;strong&gt;17 of 39 notes on the 2.0 board and 29 of 42 on the 2.1 board name one&lt;/strong&gt;. Where the same model has a Terminus-2 row and a Claude Code row on 2.1, the gap is only 0.7 to 2.5 points and points both ways.&lt;/li&gt;
&lt;li&gt;One task is worth 1.12 points. Most of the ordering at the top of either board is a few tasks, not a measured difference.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Macaron-V1-Venti%2520%2528file%2520says%25202.1%2529%2522%252C%2522GLM-5.1%2520%2528Claude%2520Code%2529%2522%252C%2522GLM-5.1%2520MXFP4%2520%2528cites%2520GLM-5.1%2529%2522%252C%2522MiMo-V2.5-Pro%2522%252C%2522DeepSeek-V4-Pro%2522%252C%2522Kimi-K2.6%2522%252C%2522MiMo-V2.5%2522%252C%2522GLM-5.1%2520%2528Terminus-2%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2522%252C%2522data%2522%253A%255B87.6%252C69.0%252C69.0%252C68.4%252C67.9%252C66.7%252C65.8%252C63.5%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523c53030%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Top%25208%2520rows%2520of%2520the%2520Terminal-Bench%25202.0%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25202%252C%25202026%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520score%252C%2520%2525%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Macaron-V1-Venti%2520%2528file%2520says%25202.1%2529%2522%252C%2522GLM-5.1%2520%2528Claude%2520Code%2529%2522%252C%2522GLM-5.1%2520MXFP4%2520%2528cites%2520GLM-5.1%2529%2522%252C%2522MiMo-V2.5-Pro%2522%252C%2522DeepSeek-V4-Pro%2522%252C%2522Kimi-K2.6%2522%252C%2522MiMo-V2.5%2522%252C%2522GLM-5.1%2520%2528Terminus-2%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2522%252C%2522data%2522%253A%255B87.6%252C69.0%252C69.0%252C68.4%252C67.9%252C66.7%252C65.8%252C63.5%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523c53030%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Top%25208%2520rows%2520of%2520the%2520Terminal-Bench%25202.0%2520board%2520on%2520Hugging%2520Face%252C%2520Nov%25202%252C%25202026%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520score%252C%2520%2525%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" alt="Top 8 rows of the Terminal-Bench 2.0 board, with the 2.1 score highlighted" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What do the two Terminal-Bench boards on Hugging Face contain?
&lt;/h2&gt;

&lt;p&gt;Terminal-Bench is a set of 89 command-line tasks run by an agent inside a container: compile this, fix that server, recover this file. The score is the share of tasks whose hidden tests pass. Hugging Face lists both &lt;code&gt;harborframework/terminal-bench-2.0&lt;/code&gt; and &lt;code&gt;harborframework/terminal-bench-2.1&lt;/code&gt; as benchmark datasets, so a model repo can publish a result in a &lt;code&gt;.eval_results/*.yaml&lt;/code&gt; file and the board collects it.&lt;/p&gt;

&lt;p&gt;I pulled both boards on November 2, 2026 (KST) from the undocumented JSON behind the leaderboard view. The 2.0 board has 39 rows from 37 repos. The 2.1 board has 42 rows from 36 repos. No row is verified.&lt;/p&gt;

&lt;p&gt;The 2.0 dataset card is blunt about this. It opens with a warning: "Warning: The leaderboard above is unofficial." It points to the official leaderboard at tbench.ai, where entries are audited for configuration, show which agent harness was used, and have public trajectories. So the HF board is a collection of self-reports and third-party registrations, and I treat each row as a claim with a source link, not as a measurement the board produced.&lt;/p&gt;

&lt;p&gt;Nine of the 39 rows on the 2.0 board cite the official tbench.ai 2.0 leaderboard as their source, and the notes say "agent: Terminus 2". Those are among the best documented rows on the page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why is a 2.1 score at rank 1 of the 2.0 board?
&lt;/h2&gt;

&lt;p&gt;The top row of the 2.0 board is &lt;code&gt;mindlab-research/Macaron-V1-Venti&lt;/code&gt; at 87.6. Three things about it line up:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The model card's benchmark table has a row named "TerminalBench 2.1" with 87.6, and the protocol table says "Harbor framework, Claude Code Agent Harness, 4h timeout" with &lt;a href="mailto:pass@1"&gt;pass@1&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;The result file is &lt;code&gt;.eval_results/terminal_bench_2_1.yaml&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Inside that file, the dataset id is &lt;code&gt;harborframework/terminal-bench-2.0&lt;/code&gt; and the task id is &lt;code&gt;terminalbench_2&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the card says 2.1, the filename says 2.1, and the registration says 2.0. I cannot tell from the files whether this was a slip when the YAML was written; the fix would be a one-line change of the dataset id. What I can say is that the board has no check for it, and the result is a lead of 18.6 points over the next row (69.0), which is larger than any other gap on the page.&lt;/p&gt;

&lt;p&gt;The card itself is careful. It is built on GLM-5.2, marks imported comparison values with an asterisk, states that the coding and terminal rows use the Claude Code scaffold and not the production harness, and says "Whether reported gains come from specialization, routing, the harness, or their interaction is not resolved by any controlled experiment in this release." The issue is not the claim. It is where the claim is displayed.&lt;/p&gt;

&lt;p&gt;On the 2.1 board, 87.6 would sit at rank 5, tied with &lt;code&gt;XiaomiMiMo/MiMo-V2.6-Flash-RL&lt;/code&gt;, behind 90.6, 89.9, 88.2 and 87.9. On the 2.0 board it is first by a margin that no other row has.&lt;/p&gt;

&lt;h2&gt;
  
  
  How different are Terminal-Bench 2.0 and 2.1?
&lt;/h2&gt;

&lt;p&gt;The maintainers' release post (May 6, 2026) says Terminal-Bench 2.1 fixes issues in &lt;strong&gt;28 of the 89 tasks&lt;/strong&gt;, about 31%. It sorts the problems into three groups:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;External dependencies, 9 tasks.&lt;/strong&gt; 2.0 pinned pre-built Docker images, but tasks that touch the internet drifted after release.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resource mismatches, 8 tasks.&lt;/strong&gt; Some tasks had too little time or compute for at least one valid approach to finish consistently.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Misspecification.&lt;/strong&gt; In &lt;code&gt;query-optimize&lt;/code&gt; the tests expected Spark SQL output while the instructions asked for PostgreSQL, so the task was rewritten to use PostgreSQL throughout.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The post adds that no task is unsolved on 2.1, and that most representative agent and model pairs score higher on 2.1. The largest gain it names is Claude Code with Opus 4.6, "12.1%". The post does not say whether that is points or relative, so I read it as an order of magnitude, not a constant. The dataset repository's README counts the changes slightly differently (26 modified tasks), so treat 26 to 28 as the range.&lt;/p&gt;

&lt;p&gt;Two consequences follow. First, a 2.0 row and a 2.1 row are different tests, and the difference can be larger than the gaps people argue about. Second, a jump across model generations on different boards cannot be split into "better model" and "easier benchmark". GLM-5.1 shows 63.5 on the 2.0 board (Terminus-2). GLM-5.2 shows 81.0 on the 2.1 board (Terminus-2). That is 17.5 points, and the data cannot say how much of it is the model.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522One%2520task%2520out%2520of%252089%2522%252C%2522Harness%2520gap%252C%2520median%2520of%25206%2520pairs%2522%252C%2522Harness%2520gap%252C%2520largest%2520of%25206%2520pairs%2522%252C%2522Spread%2520of%2520the%2520top%25209%2520rows%2520%25282.1%2520board%2529%2522%252C%2522Single-run%252095%2525%2520half-width%2520%2528n%253D89%252C%2520illustrative%2529%2522%252C%25222.0%2520to%25202.1%2520gain%252C%2520one%2520agent%252Fmodel%2520pair%2520%2528post%253A%252012.1%2525%2529%2522%252C%2522Rank%25201%2520lead%2520on%2520the%25202.0%2520board%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522points%2522%252C%2522data%2522%253A%255B1.12%252C1.15%252C2.5%252C5.4%252C7.4%252C12.1%252C18.6%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523c53030%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522How%2520big%2520is%2520each%2520effect%252C%2520in%2520score%2520points%253F%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A20%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522points%2520on%2520a%25200-100%2520scale%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522One%2520task%2520out%2520of%252089%2522%252C%2522Harness%2520gap%252C%2520median%2520of%25206%2520pairs%2522%252C%2522Harness%2520gap%252C%2520largest%2520of%25206%2520pairs%2522%252C%2522Spread%2520of%2520the%2520top%25209%2520rows%2520%25282.1%2520board%2529%2522%252C%2522Single-run%252095%2525%2520half-width%2520%2528n%253D89%252C%2520illustrative%2529%2522%252C%25222.0%2520to%25202.1%2520gain%252C%2520one%2520agent%252Fmodel%2520pair%2520%2528post%253A%252012.1%2525%2529%2522%252C%2522Rank%25201%2520lead%2520on%2520the%25202.0%2520board%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522points%2522%252C%2522data%2522%253A%255B1.12%252C1.15%252C2.5%252C5.4%252C7.4%252C12.1%252C18.6%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523c53030%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522How%2520big%2520is%2520each%2520effect%252C%2520in%2520score%2520points%253F%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A20%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522points%2520on%2520a%25200-100%2520scale%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" alt="How big is each effect: one task, harness gaps, spread of the top 9, sampling interval, version shift, rank 1 lead" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Does the agent harness decide the ranking?
&lt;/h2&gt;

&lt;p&gt;Terminal-Bench scores belong to an agent and a model together. The most common agent on these boards is Terminus-2, the Terminal-Bench team's own agent. Others are Claude Code, a model vendor's own harness, or an in-house setup.&lt;/p&gt;

&lt;p&gt;I counted how many row notes name a harness, agent, scaffold or framework at all:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Board&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;th&gt;Notes naming a harness&lt;/th&gt;
&lt;th&gt;Notes silent&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Terminal-Bench 2.0&lt;/td&gt;
&lt;td&gt;39&lt;/td&gt;
&lt;td&gt;17 (all Terminus)&lt;/td&gt;
&lt;td&gt;22&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Terminal-Bench 2.1&lt;/td&gt;
&lt;td&gt;42&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;td&gt;13&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On 2.1 the 29 split into 11 Terminus, 8 Claude Code, 3 DeepSeek's own harness, and 7 others such as NexAU, an in-house setup or NVIDIA's evaluator. The 13 silent rows include ranks 2, 5 and 8. For those I cannot say which agent produced the number, only that the model card reports it.&lt;/p&gt;

&lt;p&gt;Three rows describe themselves as best-of: GLM-5.2 at 82.7 ("Best reported harness"), Inkling-Small at 64.7 ("Best harness") and Inkling at 63.8 ("Best Harness. Effort=0.99"). Picking the best harness per model is a maximum over several runs of a noisy test, so it is biased upward compared with a row that used one fixed harness. One more row, Qwen3.8-2.4T at 86.6, is reported "under the Qwen3.8-Max product tier ... may not be identical to the raw open checkpoint alone", which is a different artifact from the weights the repo name suggests.&lt;/p&gt;

&lt;p&gt;How large is the harness effect on this board? Six repos have both a Terminus-2 row and a Claude Code row on 2.1, so I can compare directly.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Terminus-2&lt;/th&gt;
&lt;th&gt;Claude Code&lt;/th&gt;
&lt;th&gt;Terminus-2 minus Claude Code&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ornith-1.5-397B&lt;/td&gt;
&lt;td&gt;86.1&lt;/td&gt;
&lt;td&gt;85.2&lt;/td&gt;
&lt;td&gt;+0.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GLM-5.2&lt;/td&gt;
&lt;td&gt;81.0&lt;/td&gt;
&lt;td&gt;82.7&lt;/td&gt;
&lt;td&gt;-1.7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ornith-1.0-397B&lt;/td&gt;
&lt;td&gt;77.5&lt;/td&gt;
&lt;td&gt;78.2&lt;/td&gt;
&lt;td&gt;-0.7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ornith-1.0-35B&lt;/td&gt;
&lt;td&gt;64.2&lt;/td&gt;
&lt;td&gt;62.8&lt;/td&gt;
&lt;td&gt;+1.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ornith-1.5-9B&lt;/td&gt;
&lt;td&gt;46.2&lt;/td&gt;
&lt;td&gt;47.0&lt;/td&gt;
&lt;td&gt;-0.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ornith-1.0-9B&lt;/td&gt;
&lt;td&gt;43.1&lt;/td&gt;
&lt;td&gt;40.6&lt;/td&gt;
&lt;td&gt;+2.5&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The gaps run from 0.7 to 2.5 points with a median of 1.15, and the sign is split three and three. Five of the six pairs come from one lab with the same sampling settings and "avg of 5 runs", so treat this as six pairs, not six independent labs. The honest reading is that here the harness choice moved scores by one or two tasks and in no consistent direction.&lt;/p&gt;

&lt;p&gt;That is not a general law. On the 2.0 board, GLM-5.1 has 69.0 under Claude Code and 63.5 under Terminus-2, a 5.5 point gap. The harness effect can be small or large depending on the model, and a board that does not record the harness cannot tell you which case you are in.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Ornith-1.5-397B%2522%252C%2522GLM-5.2%2522%252C%2522Ornith-1.0-397B%2522%252C%2522Ornith-1.0-35B%2522%252C%2522Ornith-1.5-9B%2522%252C%2522Ornith-1.0-9B%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522Terminus-2%2520minus%2520Claude%2520Code%2522%252C%2522data%2522%253A%255B0.9%252C-1.7%252C-0.7%252C1.4%252C-0.8%252C2.5%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Same%2520model%252C%2520two%2520harnesses%252C%2520Terminal-Bench%25202.1%2520board%2520%2528blue%253A%2520Terminus-2%2520higher%252C%2520orange%253A%2520Claude%2520Code%2520higher%2529%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A-3%252C%2522max%2522%253A3%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522Terminus-2%2520score%2520minus%2520Claude%2520Code%2520score%252C%2520points%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Ornith-1.5-397B%2522%252C%2522GLM-5.2%2522%252C%2522Ornith-1.0-397B%2522%252C%2522Ornith-1.0-35B%2522%252C%2522Ornith-1.5-9B%2522%252C%2522Ornith-1.0-9B%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522Terminus-2%2520minus%2520Claude%2520Code%2522%252C%2522data%2522%253A%255B0.9%252C-1.7%252C-0.7%252C1.4%252C-0.8%252C2.5%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Same%2520model%252C%2520two%2520harnesses%252C%2520Terminal-Bench%25202.1%2520board%2520%2528blue%253A%2520Terminus-2%2520higher%252C%2520orange%253A%2520Claude%2520Code%2520higher%2529%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A-3%252C%2522max%2522%253A3%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522Terminus-2%2520score%2520minus%2520Claude%2520Code%2520score%252C%2520points%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A12%257D%257D%255D%257D%257D%257D" alt="Terminus-2 minus Claude Code for six same-model pairs on the 2.1 board" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How much can 89 tasks separate?
&lt;/h2&gt;

&lt;p&gt;One task is 1.12 points. The top 9 rows of the 2.1 board run from 90.6 to 85.2, a spread of 5.4 points, or about 4.8 tasks.&lt;/p&gt;

&lt;p&gt;For scale, take a model that solves 85% of tasks and treat one run as 89 independent draws. The standard error is 3.8 points and the 95% interval is about plus or minus 7.4. At 50% the half-width is 10.4. That calculation is a ceiling for unpaired comparisons: every model faces the same 89 tasks, so the real noise in a head-to-head gap is smaller, but the board has no per-task results, so I cannot compute by how much. Averaging 5 runs, as several rows do, removes attempt-to-attempt variation but not the fact that the 89 tasks are a sample.&lt;/p&gt;

&lt;p&gt;So 90.6 against 88.2 against 87.9 is not an ordering the data establishes. It is a cluster. The same logic applies to the near-tie at the top of the 2.0 board once the version-mismatched row is set aside: 69.0, 69.0, 68.4, 67.9, 66.7 are within 2.3 points.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which rows repeat a number from another repo?
&lt;/h2&gt;

&lt;p&gt;Four rows on the 2.0 board, from three repos, cite a different Hugging Face repo as their source.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound&lt;/code&gt; shows 69.0 and 63.5, the same two values as &lt;code&gt;zai-org/GLM-5.1&lt;/code&gt;, with that repo as the source. Its card describes an MXFP4 quantization made with auto-round, and I found no Terminal-Bench line in it. A quantized checkpoint can score differently from its base, and these rows do not say it was measured. My October 31 post on the SWE-Bench Pro board found the same repo doing the same thing.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-BF16&lt;/code&gt; shows 31.0 with NVIDIA's repo as the source, the same value as the original.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;MuVeraAI/Laguna-XS.2&lt;/code&gt; shows 30.1 and cites &lt;code&gt;poolside/Laguna-XS.2&lt;/code&gt;, whose own row is 35.7. A 5.6 point disagreement with the cited source and no note about why.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The contrast is on the 2.1 board: NVIDIA's BF16 and NVFP4 Nemotron-3-Ultra rows show 56.4 and 53.9. When a quantized checkpoint is evaluated, the number moves, and the two rows say so.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do I check these things on any board?
&lt;/h2&gt;

&lt;p&gt;The leaderboard JSON endpoint is undocumented and may change, so treat this as a snapshot script. It reproduces the counts above.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;board&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;  &lt;span class="c1"&gt;# undocumented JSON behind the leaderboard view; may change
&lt;/span&gt;    &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/datasets/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/leaderboard&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;B20&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;board&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;harborframework/terminal-bench-2.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;B21&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;board&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;harborframework/terminal-bench-2.1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;B20&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows on 2.0;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;B21&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows on 2.1;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verified&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;B20&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;B21&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verified&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 1. a result file whose name says 2.1 but sits on the 2.0 board
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;B20&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2[_.]1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;filename&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]):&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;version mismatch:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rank&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;filename&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="c1"&gt;# 2. how many notes name an agent harness at all
&lt;/span&gt;&lt;span class="n"&gt;HARNESS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;terminus|claude code|harness|agent|scaffold|codex|openhands|framework&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;B20&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2.1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;B21&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
    &lt;span class="n"&gt;named&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;HARNESS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;notes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: harness named in &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;named&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; of &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; notes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 3. same repo, two harnesses, on the 2.1 board
&lt;/span&gt;&lt;span class="n"&gt;by&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;B21&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;notes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;
    &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;terminus&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;terminus&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;claude_code&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;claude code&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;by&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]][&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;by&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: Terminus-2 &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;terminus&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; vs Claude Code &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;claude_code&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; (&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;terminus&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;claude_code&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 4. rows whose cited source is a different Hugging Face repo
&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;B20&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface\.co/([^/]+/[^/]+)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source is&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On November 2, 2026 it prints:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;39 rows on 2.0; 42 rows on 2.1; 0 verified
version mismatch: 1 mindlab-research/Macaron-V1-Venti 87.6 .eval_results/terminal_bench_2_1.yaml
2.0: harness named in 17 of 39 notes
2.1: harness named in 29 of 42 notes
ornith-ai/Ornith-1.5-397B: Terminus-2 86.1 vs Claude Code 85.2 (+0.9)
zai-org/GLM-5.2: Terminus-2 81 vs Claude Code 82.7 (-1.7)
ornith-ai/Ornith-1.0-397B: Terminus-2 77.5 vs Claude Code 78.2 (-0.7)
ornith-ai/Ornith-1.0-35B: Terminus-2 64.2 vs Claude Code 62.8 (+1.4)
ornith-ai/Ornith-1.5-9B: Terminus-2 46.2 vs Claude Code 47 (-0.8)
ornith-ai/Ornith-1.0-9B: Terminus-2 43.1 vs Claude Code 40.6 (+2.5)
source is zai-org/GLM-5.1 -&amp;gt; INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound 69
source is zai-org/GLM-5.1 -&amp;gt; INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound 63.5
source is nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 -&amp;gt; RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 31
source is poolside/Laguna-XS.2 -&amp;gt; MuVeraAI/Laguna-XS.2 30.1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The harness classifier reads free text, so it is a lower bound on rows that name one, which is the same limit a reader faces.&lt;/p&gt;

&lt;h2&gt;
  
  
  What should a reader do with a Terminal-Bench row?
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Check the version before the number.&lt;/strong&gt; The filename, the model card and the dataset id can disagree, as the rank 1 row shows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never compare a 2.0 row with a 2.1 row&lt;/strong&gt; unless the same agent and model pair was run on both. Of the 89 tasks, 28 changed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Find the harness.&lt;/strong&gt; If the note is silent, open the card. If the card says "best harness", it is a maximum, not a typical run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat a few points as nothing.&lt;/strong&gt; One task is 1.12 points, and a single run has a sampling interval of several points.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look for quantized and mirrored repos&lt;/strong&gt; whose row cites another repo. A copied number is the base model's score.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use the official tbench.ai leaderboard for audited entries&lt;/strong&gt;, and use the HF board to find what to look at, not to rank.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Is Terminal-Bench 2.1 harder or easier than 2.0?&lt;/strong&gt;&lt;br&gt;
The release post says most representative agent and model pairs improve on 2.1 after 28 of 89 tasks were fixed, and that no task is unsolved. In that sense it is easier on average, but the shift differs by agent, model and task.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the 87.6 score at rank 1 of the 2.0 board wrong?&lt;/strong&gt;&lt;br&gt;
Nothing here shows the score is wrong. Its card documents the protocol and labels it Terminal-Bench 2.1. The problem is that it is registered under the 2.0 dataset, so the 2.0 board shows it as a 2.0 result.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does the harness matter on Terminal-Bench?&lt;/strong&gt;&lt;br&gt;
Sometimes. Six same-model pairs on the 2.1 board differ by 0.7 to 2.5 points with no consistent sign, while GLM-5.1 differs by 5.5 points on the 2.0 board. There is no single constant to subtract.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How many Terminal-Bench scores on Hugging Face are verified?&lt;/strong&gt;&lt;br&gt;
None. Across the 2.0 and 2.1 boards, all 81 rows show verified as false.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Which leaderboard is official?&lt;/strong&gt;&lt;br&gt;
The 2.0 dataset card says the HF leaderboard is unofficial and names tbench.ai as the official one, where entries are audited and the harness is shown.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I compare a Terminal-Bench 2.0 score with a 2.1 score?&lt;/strong&gt;&lt;br&gt;
Only if the same agent and model pair was run on both versions. Otherwise the difference mixes the model, the harness and the 28 changed tasks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-pro-has-two-versions-what-the-hugging-face-leaderboards-47-rows-mix-together-37co"&gt;SWE-Bench Pro Has Two Versions: What the Hugging Face Leaderboard's 47 Rows Mix Together&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/humanitys-last-exam-on-hugging-face-at-least-23-of-the-top-24-rows-use-tools-and-the-board-has-13ig"&gt;Humanity's Last Exam on Hugging Face: At Least 23 of the Top 24 Rows Use Tools, and the Board Has No Column for It&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Terminal-Bench 2.0 dataset and board: &lt;a href="https://huggingface.co/datasets/harborframework/terminal-bench-2.0" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/harborframework/terminal-bench-2.0&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Terminal-Bench 2.1 dataset and board: &lt;a href="https://huggingface.co/datasets/harborframework/terminal-bench-2.1" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/harborframework/terminal-bench-2.1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Terminal-Bench 2.1 release post: &lt;a href="https://www.tbench.ai/news/terminal-bench-2-1" rel="noopener noreferrer"&gt;https://www.tbench.ai/news/terminal-bench-2-1&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Official Terminal-Bench 2.0 leaderboard: &lt;a href="https://www.tbench.ai/leaderboard/terminal-bench/2.0" rel="noopener noreferrer"&gt;https://www.tbench.ai/leaderboard/terminal-bench/2.0&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Rank 1 row model card: &lt;a href="https://huggingface.co/mindlab-research/Macaron-V1-Venti" rel="noopener noreferrer"&gt;https://huggingface.co/mindlab-research/Macaron-V1-Venti&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GLM-5.1 model card: &lt;a href="https://huggingface.co/zai-org/GLM-5.1" rel="noopener noreferrer"&gt;https://huggingface.co/zai-org/GLM-5.1&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All counts in this article are my own measurements from the board JSON, dataset cards and the cited model cards on November 2, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. The classification of notes is mine, based on free text, and the board endpoint is undocumented and may change.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>Humanity's Last Exam on Hugging Face: At Least 23 of the Top 24 Rows Use Tools, and the Board Has No Column for It</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Sun, 01 Nov 2026 13:45:05 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/humanitys-last-exam-on-hugging-face-at-least-23-of-the-top-24-rows-use-tools-and-the-board-has-13ig</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/humanitys-last-exam-on-hugging-face-at-least-23-of-the-top-24-rows-use-tools-and-the-board-has-13ig</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On November 1, 2026 the Hugging Face board for Humanity's Last Exam (HLE) lists &lt;strong&gt;96 rows from 64 model repos, 0 marked verified&lt;/strong&gt;. Row 1 is &lt;strong&gt;63.9&lt;/strong&gt;, row 2 is &lt;strong&gt;62.5&lt;/strong&gt;, row 3 is &lt;strong&gt;61.0&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;The board has no field for "tools allowed". The only signal is the free-text &lt;code&gt;notes&lt;/code&gt; string. By my count &lt;strong&gt;34 rows say "with tools", 12 say "no tools", and 50 say nothing&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;21 of the top 24 rows say "with tools". The other 3 are unlabeled&lt;/strong&gt;, and for two of them (Kimi-K3 at 56.0, Inkling at 46.0) the model card shows the number is the with-tools score. The first row that says "no tools" is &lt;strong&gt;rank 25&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Where the same model has both a with-tools and a no-tools row, tools add &lt;strong&gt;7.8 to 15.1 points&lt;/strong&gt; (median 11.3, seven pairs). That is about four times the sampling noise of a 2,500-question test, and it is not a constant you can subtract.&lt;/li&gt;
&lt;li&gt;Among closed-book numbers the top six sit within about 2 points of each other, so their order is not established by this data.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Beam-1.0-501B%2522%252C%2522Hy4-preview%2522%252C%2522Ornith-1.5-397B%2522%252C%2522Nemotron-3-Ultra-NVFP4%2522%252C%2522Nemotron-3-Ultra-BF16%2522%252C%2522Ornith-1.5-9B%2522%252C%2522Ornith-1.5-35B%2522%252C%2522gpt-oss-20b%2520%2528high%2529%2522%252C%2522gpt-oss-120b%2520%2528high%2529%2522%252C%2522gpt-oss-120b%2520%2528low%2529%2522%252C%2522gpt-oss-120b%2520%2528medium%2529%2522%252C%2522gpt-oss-20b%2520%2528low%2529%2522%252C%2522gpt-oss-20b%2520%2528medium%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522uplift%2522%252C%2522data%2522%253A%255B15.1%252C12.0%252C11.5%252C11.3%252C10.7%252C10.3%252C7.8%252C6.4%252C4.1%252C3.9%252C2.7%252C2.1%252C1.8%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Points%2520added%2520by%2520tools%252C%2520same%2520model%2520on%2520the%2520HLE%2520board%2520%2528blue%253A%2520both%2520rows%2520labeled%252C%2520orange%253A%2520paired%2520by%2520reasoning%2520level%2529%2522%252C%2522fontSize%2522%253A15%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A16%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522with-tools%2520minus%2520no-tools%252C%2520points%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Beam-1.0-501B%2522%252C%2522Hy4-preview%2522%252C%2522Ornith-1.5-397B%2522%252C%2522Nemotron-3-Ultra-NVFP4%2522%252C%2522Nemotron-3-Ultra-BF16%2522%252C%2522Ornith-1.5-9B%2522%252C%2522Ornith-1.5-35B%2522%252C%2522gpt-oss-20b%2520%2528high%2529%2522%252C%2522gpt-oss-120b%2520%2528high%2529%2522%252C%2522gpt-oss-120b%2520%2528low%2529%2522%252C%2522gpt-oss-120b%2520%2528medium%2529%2522%252C%2522gpt-oss-20b%2520%2528low%2529%2522%252C%2522gpt-oss-20b%2520%2528medium%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522uplift%2522%252C%2522data%2522%253A%255B15.1%252C12.0%252C11.5%252C11.3%252C10.7%252C10.3%252C7.8%252C6.4%252C4.1%252C3.9%252C2.7%252C2.1%252C1.8%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Points%2520added%2520by%2520tools%252C%2520same%2520model%2520on%2520the%2520HLE%2520board%2520%2528blue%253A%2520both%2520rows%2520labeled%252C%2520orange%253A%2520paired%2520by%2520reasoning%2520level%2529%2522%252C%2522fontSize%2522%253A15%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A16%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522with-tools%2520minus%2520no-tools%252C%2520points%2522%257D%257D%255D%252C%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A11%257D%257D%255D%257D%257D%257D" alt="Points added by tools for 13 same-model pairs on the HLE board" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What is on the HLE board, and what does a row actually claim?
&lt;/h2&gt;

&lt;p&gt;HLE is a 2,500-question exam written by subject experts. The paper describes it as multimodal: roughly 14% of the questions need images, and about 24% are multiple choice, the rest short exact-match answers. Hugging Face lists the dataset as an official benchmark, so model repos can publish a score in a &lt;code&gt;.eval_results/*.yaml&lt;/code&gt; file and the board collects them.&lt;/p&gt;

&lt;p&gt;A row is therefore a self-report. In my last post on this series I walked through how those YAML files become a board; here the relevant part is small. A row has a value, a source link, a pull request or commit, and an optional &lt;code&gt;notes&lt;/code&gt; string. There is no structured field for tool access, test subset, judge model or number of runs.&lt;/p&gt;

&lt;p&gt;Two rows hint at what the board's default is. One submitter writes that their number is "with tools (full set)... not the no-tools default". Another says theirs is "evaluated with tool/browsing access enabled, not the closed-book default". I could not read the dataset's &lt;code&gt;eval.yaml&lt;/code&gt; (it is gated), so I cannot confirm the default from the source, but if those two notes are right, the top of the board is made of off-default runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  How many rows say whether tools were used?
&lt;/h2&gt;

&lt;p&gt;I classified every row by its note: "with tools", "w/ tool" or "with tool" means tools; "no tools" or "without tools" means closed-book; anything else is unlabeled.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Condition (from note text)&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;th&gt;Share of 96&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;With tools&lt;/td&gt;
&lt;td&gt;34&lt;/td&gt;
&lt;td&gt;35%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No tools&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;13%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unlabeled&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;52%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Half the board does not say. That is not a criticism of any single submitter: the YAML format never asked.&lt;/p&gt;

&lt;p&gt;Ten rows also say "text-only", which means they were scored on the subset without images. The Artificial Analysis HLE page uses 2,158 text-only questions out of the 2,500, so a text-only row and a full-set row are scored on different tests. Three rows state a judge: two say GPT-4o (one adds that the task default is o3-mini) and one says GPT-5.6-luna. The other 93 rows say nothing about the grader.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why does the top of the board look the way it does?
&lt;/h2&gt;

&lt;p&gt;Sorted by value, the top 24 rows break down like this: 21 say "with tools", 3 are unlabeled, none say "no tools". The first explicit no-tools row, Ornith-1.5-397B at 44.6, is &lt;strong&gt;rank 25&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%252C%252235%2522%252C%252236%2522%252C%252237%2522%252C%252238%2522%252C%252239%2522%252C%252240%2522%252C%252241%2522%252C%252242%2522%252C%252243%2522%252C%252244%2522%252C%252245%2522%252C%252246%2522%252C%252247%2522%252C%252248%2522%252C%252249%2522%252C%252250%2522%252C%252251%2522%252C%252252%2522%252C%252253%2522%252C%252254%2522%252C%252255%2522%252C%252256%2522%252C%252257%2522%252C%252258%2522%252C%252259%2522%252C%252260%2522%252C%252261%2522%252C%252262%2522%252C%252263%2522%252C%252264%2522%252C%252265%2522%252C%252266%2522%252C%252267%2522%252C%252268%2522%252C%252269%2522%252C%252270%2522%252C%252271%2522%252C%252272%2522%252C%252273%2522%252C%252274%2522%252C%252275%2522%252C%252276%2522%252C%252277%2522%252C%252278%2522%252C%252279%2522%252C%252280%2522%252C%252281%2522%252C%252282%2522%252C%252283%2522%252C%252284%2522%252C%252285%2522%252C%252286%2522%252C%252287%2522%252C%252288%2522%252C%252289%2522%252C%252290%2522%252C%252291%2522%252C%252292%2522%252C%252293%2522%252C%252294%2522%252C%252295%2522%252C%252296%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2520%2525%2522%252C%2522data%2522%253A%255B63.9%252C62.5%252C61%252C56.1%252C56%252C55.4%252C55.3%252C54.7%252C54%252C53.2%252C52.6%252C52.3%252C52.3%252C51.1%252C50.4%252C50.2%252C48.5%252C48.3%252C48.1%252C48%252C47.6%252C47.5%252C46%252C44.9%252C44.6%252C44%252C43.6%252C43.4%252C43.4%252C42.8%252C42.6%252C40.5%252C37.7%252C37.44%252C37.4%252C37.4%252C36.8%252C36%252C35.9%252C34.8%252C34.7%252C34.6%252C33.4%252C31.6%252C31%252C31%252C30.8%252C30.5%252C30.5%252C30%252C28.8%252C28.7%252C27.8%252C26.7%252C26.1%252C25.6%252C25.3%252C24.8%252C24.37%252C24.3%252C24%252C23.9%252C23.1%252C22.82%252C22.82%252C22.7%252C22.2%252C22.1%252C22%252C21.94%252C21.4%252C20.2%252C19.4%252C19%252C18.26%252C18.26%252C17.3%252C14.9%252C14.4%252C13.6%252C12.5%252C11.72%252C11.3%252C10.9%252C10.47%252C9.92%252C9.1%252C8.8%252C8.6%252C8.6%252C8.32%252C8.12%252C7%252C6.3%252C5.2%252C4.2%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523c53030%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523c53030%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%252296%2520rows%2520on%2520the%2520HLE%2520board%2520by%2520rank.%2520Orange%253A%2520with%2520tools.%2520Blue%253A%2520no%2520tools.%2520Grey%253A%2520unlabeled.%2520Red%253A%2520unlabeled%252C%2520card%2520says%2520with%2520tools%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A70%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520%2525%2522%257D%257D%255D%252C%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A8%252C%2522autoSkip%2522%253Atrue%252C%2522maxTicksLimit%2522%253A24%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rank%2520%2528Nov%25201%252C%25202026%2529%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%252C%252235%2522%252C%252236%2522%252C%252237%2522%252C%252238%2522%252C%252239%2522%252C%252240%2522%252C%252241%2522%252C%252242%2522%252C%252243%2522%252C%252244%2522%252C%252245%2522%252C%252246%2522%252C%252247%2522%252C%252248%2522%252C%252249%2522%252C%252250%2522%252C%252251%2522%252C%252252%2522%252C%252253%2522%252C%252254%2522%252C%252255%2522%252C%252256%2522%252C%252257%2522%252C%252258%2522%252C%252259%2522%252C%252260%2522%252C%252261%2522%252C%252262%2522%252C%252263%2522%252C%252264%2522%252C%252265%2522%252C%252266%2522%252C%252267%2522%252C%252268%2522%252C%252269%2522%252C%252270%2522%252C%252271%2522%252C%252272%2522%252C%252273%2522%252C%252274%2522%252C%252275%2522%252C%252276%2522%252C%252277%2522%252C%252278%2522%252C%252279%2522%252C%252280%2522%252C%252281%2522%252C%252282%2522%252C%252283%2522%252C%252284%2522%252C%252285%2522%252C%252286%2522%252C%252287%2522%252C%252288%2522%252C%252289%2522%252C%252290%2522%252C%252291%2522%252C%252292%2522%252C%252293%2522%252C%252294%2522%252C%252295%2522%252C%252296%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2520%2525%2522%252C%2522data%2522%253A%255B63.9%252C62.5%252C61%252C56.1%252C56%252C55.4%252C55.3%252C54.7%252C54%252C53.2%252C52.6%252C52.3%252C52.3%252C51.1%252C50.4%252C50.2%252C48.5%252C48.3%252C48.1%252C48%252C47.6%252C47.5%252C46%252C44.9%252C44.6%252C44%252C43.6%252C43.4%252C43.4%252C42.8%252C42.6%252C40.5%252C37.7%252C37.44%252C37.4%252C37.4%252C36.8%252C36%252C35.9%252C34.8%252C34.7%252C34.6%252C33.4%252C31.6%252C31%252C31%252C30.8%252C30.5%252C30.5%252C30%252C28.8%252C28.7%252C27.8%252C26.7%252C26.1%252C25.6%252C25.3%252C24.8%252C24.37%252C24.3%252C24%252C23.9%252C23.1%252C22.82%252C22.82%252C22.7%252C22.2%252C22.1%252C22%252C21.94%252C21.4%252C20.2%252C19.4%252C19%252C18.26%252C18.26%252C17.3%252C14.9%252C14.4%252C13.6%252C12.5%252C11.72%252C11.3%252C10.9%252C10.47%252C9.92%252C9.1%252C8.8%252C8.6%252C8.6%252C8.32%252C8.12%252C7%252C6.3%252C5.2%252C4.2%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523c53030%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523c53030%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%25232b6cb0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523a0aec0%2522%252C%2522%2523a0aec0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%252296%2520rows%2520on%2520the%2520HLE%2520board%2520by%2520rank.%2520Orange%253A%2520with%2520tools.%2520Blue%253A%2520no%2520tools.%2520Grey%253A%2520unlabeled.%2520Red%253A%2520unlabeled%252C%2520card%2520says%2520with%2520tools%2522%252C%2522fontSize%2522%253A14%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A70%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520%2525%2522%257D%257D%255D%252C%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A8%252C%2522autoSkip%2522%253Atrue%252C%2522maxTicksLimit%2522%253A24%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rank%2520%2528Nov%25201%252C%25202026%2529%2522%257D%257D%255D%257D%257D%257D" alt="All 96 HLE board rows by rank, colored by tool condition" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The three unlabeled rows in the top 24 are Kimi-K3 (rank 5, 56.0), Hy3 (rank 10, 53.2) and Inkling (rank 23, 46.0). I opened the model cards:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Kimi-K3.&lt;/strong&gt; The card's HLE-Full row reads "43.5 / 56.0", and the footnote says each cell reports the score "without and with tool augmentation ... in that order". The board's 56.0 is the with-tools number. Its &lt;code&gt;hle.yaml&lt;/code&gt; contains a value, a source and nothing else. The closed-book 43.5 is on the card but not on the board. With 43.5 it would sit near rank 27.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Inkling.&lt;/strong&gt; The card lists "HLE (text only) 29.7%" and "HLE (with tools) 46.0%". The board's 46.0 is the with-tools figure. The text-only figure appears nowhere on the board.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hy3.&lt;/strong&gt; I found no HLE line in the card text, so I cannot resolve this row. I count it as unknown.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the accurate statement is "at least 23 of the top 24". Row 1 is also worth a look: its note reads "With tools; harness not specified in the model card." The top score comes with the tool set and the harness both unspecified.&lt;/p&gt;

&lt;h2&gt;
  
  
  How big is the tool uplift, and can you correct for it?
&lt;/h2&gt;

&lt;p&gt;Several repos publish both numbers, so I paired rows that belong to the same model. Seven pairs have both rows explicitly labeled. Six more come from gpt-oss, where the paired rows share a reasoning level and the no-tools rows carry no tools label (the pairing follows the cited OpenAI paper, arXiv 2508.10925, so treat it as weaker).&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;With tools&lt;/th&gt;
&lt;th&gt;No tools&lt;/th&gt;
&lt;th&gt;Uplift&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Beam-1.0-501B&lt;/td&gt;
&lt;td&gt;51.1&lt;/td&gt;
&lt;td&gt;36.0&lt;/td&gt;
&lt;td&gt;+15.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hy4-preview (text-only)&lt;/td&gt;
&lt;td&gt;55.4&lt;/td&gt;
&lt;td&gt;43.4&lt;/td&gt;
&lt;td&gt;+12.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ornith-1.5-397B&lt;/td&gt;
&lt;td&gt;56.1&lt;/td&gt;
&lt;td&gt;44.6&lt;/td&gt;
&lt;td&gt;+11.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nemotron-3-Ultra NVFP4&lt;/td&gt;
&lt;td&gt;37.4&lt;/td&gt;
&lt;td&gt;26.1&lt;/td&gt;
&lt;td&gt;+11.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nemotron-3-Ultra BF16&lt;/td&gt;
&lt;td&gt;37.4&lt;/td&gt;
&lt;td&gt;26.7&lt;/td&gt;
&lt;td&gt;+10.7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ornith-1.5-9B&lt;/td&gt;
&lt;td&gt;30.5&lt;/td&gt;
&lt;td&gt;20.2&lt;/td&gt;
&lt;td&gt;+10.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ornith-1.5-35B&lt;/td&gt;
&lt;td&gt;33.4&lt;/td&gt;
&lt;td&gt;25.6&lt;/td&gt;
&lt;td&gt;+7.8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-20b (high)&lt;/td&gt;
&lt;td&gt;17.3&lt;/td&gt;
&lt;td&gt;10.9&lt;/td&gt;
&lt;td&gt;+6.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b (high)&lt;/td&gt;
&lt;td&gt;19.0&lt;/td&gt;
&lt;td&gt;14.9&lt;/td&gt;
&lt;td&gt;+4.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b (low)&lt;/td&gt;
&lt;td&gt;9.1&lt;/td&gt;
&lt;td&gt;5.2&lt;/td&gt;
&lt;td&gt;+3.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-120b (medium)&lt;/td&gt;
&lt;td&gt;11.3&lt;/td&gt;
&lt;td&gt;8.6&lt;/td&gt;
&lt;td&gt;+2.7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-20b (low)&lt;/td&gt;
&lt;td&gt;6.3&lt;/td&gt;
&lt;td&gt;4.2&lt;/td&gt;
&lt;td&gt;+2.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-20b (medium)&lt;/td&gt;
&lt;td&gt;8.8&lt;/td&gt;
&lt;td&gt;7.0&lt;/td&gt;
&lt;td&gt;+1.8&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For the seven explicit pairs the median uplift is 11.3 points with a range of 7.8 to 15.1. Two of those are the same Nemotron model in two precisions, so call it six independent pairs. Across all 13 pairs the range is 1.8 to 15.1, an eightfold spread.&lt;/p&gt;

&lt;p&gt;That spread is the practical point. There is no single "tools constant" to subtract from a with-tools row to get its closed-book equivalent. The uplift depends on the model, its training for tool use, the tool set and the harness. A with-tools row cannot be turned into a closed-book row by arithmetic; it can only be replaced by a closed-book measurement.&lt;/p&gt;

&lt;p&gt;For scale, the sampling noise of the test itself is small next to this. With n = 2,500 and accuracy near 45%, the standard error is 0.99 points and a 95% interval is about plus or minus 1.95. Two independent runs need to differ by about &lt;strong&gt;2.8 points&lt;/strong&gt; (3.0 on the 2,158-question subset) before the gap clears noise. Three of the gpt-oss uplifts (1.8, 2.1, 2.7) fall under that line, though the pairs are scored on the same questions, which lowers the real threshold by an amount I cannot compute without per-question results.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is the closed-book ordering any better?
&lt;/h2&gt;

&lt;p&gt;Take the closed-book numbers: the explicit no-tools rows plus Kimi-K3's 43.5 from its card.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Ornith-1.5-397B%2522%252C%2522DeepSeek-V4.1-Flash-1017%2520%2528text-only%2529%2522%252C%2522Kimi-K3%2520%2528card%2520only%2529%2522%252C%2522Hy4-preview%2520%2528text-only%2529%2522%252C%2522MiniMax-M3.1-Flash%2520%2528text-only%2529%2522%252C%2522DeepSeek-V4.1-Flash-1017%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%252295%2525%2520sampling%2520interval%2522%252C%2522data%2522%253A%255B%255B42.65%252C46.55%255D%252C%255B42.05%252C45.95%255D%252C%255B41.56%252C45.44%255D%252C%255B41.46%252C45.34%255D%252C%255B41.46%252C45.34%255D%252C%255B40.66%252C44.54%255D%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Closed-book%2520top%25206%253A%2520every%2520interval%2520overlaps%2520%2528intervals%2520use%2520n%253D2%252C500%252C%2520the%2520narrowest%2520case%2529%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A38%252C%2522max%2522%253A49%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520correct%2522%257D%257D%255D%252C%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%252C%2522autoSkip%2522%253Afalse%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Ornith-1.5-397B%2522%252C%2522DeepSeek-V4.1-Flash-1017%2520%2528text-only%2529%2522%252C%2522Kimi-K3%2520%2528card%2520only%2529%2522%252C%2522Hy4-preview%2520%2528text-only%2529%2522%252C%2522MiniMax-M3.1-Flash%2520%2528text-only%2529%2522%252C%2522DeepSeek-V4.1-Flash-1017%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%252295%2525%2520sampling%2520interval%2522%252C%2522data%2522%253A%255B%255B42.65%252C46.55%255D%252C%255B42.05%252C45.95%255D%252C%255B41.56%252C45.44%255D%252C%255B41.46%252C45.34%255D%252C%255B41.46%252C45.34%255D%252C%255B40.66%252C44.54%255D%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Closed-book%2520top%25206%253A%2520every%2520interval%2520overlaps%2520%2528intervals%2520use%2520n%253D2%252C500%252C%2520the%2520narrowest%2520case%2529%2522%252C%2522fontSize%2522%253A16%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A38%252C%2522max%2522%253A49%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520correct%2522%257D%257D%255D%252C%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%252C%2522autoSkip%2522%253Afalse%257D%257D%255D%257D%257D%257D" alt="Closed-book top six HLE scores with 95% sampling intervals" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The top six run from 44.6 down to 42.6, a spread of 2.0 points. Every interval overlaps every other. Three of the six are text-only rows scored on 2,158 questions, so their intervals are slightly wider than drawn, and the rows without a subset note may or may not be on the same test. The honest reading is a cluster, not a ranking. This matches what I found on the GPQA Diamond board: when the test is small relative to the model gaps, listing models in order of value implies precision that is not there.&lt;/p&gt;

&lt;h2&gt;
  
  
  How do I check the conditions on any board myself?
&lt;/h2&gt;

&lt;p&gt;The board JSON endpoint is undocumented and may change, so treat this as a snapshot tool. It reproduces the counts above.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="n"&gt;URL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/datasets/cais/hle/leaderboard&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;  &lt;span class="c1"&gt;# undocumented, may change
&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}),&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;condition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;note&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;note&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;note&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;with tools?|w/ tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;note&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tools&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no tools?|without tools?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;note&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no_tools&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unlabeled&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;condition&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;notes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;top 24 by condition:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;

&lt;span class="n"&gt;by_model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;by_model&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modelId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]].&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rs&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;by_model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rs&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tools&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rs&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cond&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no_tools&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; vs &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; (+&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On November 1, 2026 it prints:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;96 rows; Counter({'unlabeled': 50, 'tools': 34, 'no_tools': 12})
top 24 by condition: Counter({'tools': 21, 'unlabeled': 3})
ornith-ai/Ornith-1.5-397B: 56.1 vs 44.6 (+11.5)
tencent/Hy4-preview: 55.4 vs 43.4 (+12.0)
reflection/Beam-1.0-501B-A23B: 51.1 vs 36 (+15.1)
nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16: 37.4 vs 26.7 (+10.7)
nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4: 37.4 vs 26.1 (+11.3)
ornith-ai/Ornith-1.5-35B-A3B: 33.4 vs 25.6 (+7.8)
ornith-ai/Ornith-1.5-9B: 30.5 vs 20.2 (+10.3)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The classifier reads free text, so it is a lower bound on labeled rows: a note phrased in some other way lands in "unlabeled". That is the same limitation a reader faces.&lt;/p&gt;

&lt;h2&gt;
  
  
  What should a reader do with a board like this?
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Split by condition before you rank.&lt;/strong&gt; With-tools, no-tools and unlabeled rows are three different lists.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat unlabeled as unknown, not as the default.&lt;/strong&gt; Open the model card. In two of three top-24 cases here, the card settled it in under a minute.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compare within one subset.&lt;/strong&gt; Text-only (2,158 questions) and full-set (2,500) rows are different tests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Apply a noise floor of about 3 points&lt;/strong&gt; between independent runs, and more if the judge differs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look for the closed-book number the card reports but the board omits.&lt;/strong&gt; Kimi-K3's 43.5 is a case where the card has more than the board.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ask submitters for three fields:&lt;/strong&gt; tools used (yes or no, and which), subset, and judge. A board that stored those as structured fields would make the filter above a dropdown instead of a regex.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Do tools make an HLE score invalid?&lt;/strong&gt;&lt;br&gt;
No. A with-tools score is a legitimate measurement of a model plus its tool harness. It is just not comparable to a closed-book score, and the board puts them in one sorted list.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the 63.9 at the top wrong?&lt;/strong&gt;&lt;br&gt;
Nothing here shows it is wrong. It is self-reported, marked with tools, and its note says the harness is not specified, so it cannot be compared with rows run under a different harness.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How many HLE questions does the board's test have?&lt;/strong&gt;&lt;br&gt;
The dataset has 2,500 questions. Some rows are scored on the text-only subset, which Artificial Analysis puts at 2,158 questions, so the denominator differs between rows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I convert a with-tools score to a closed-book estimate?&lt;/strong&gt;&lt;br&gt;
Not reliably. In the paired rows the uplift runs from 1.8 to 15.1 points, depending on the model and reasoning level.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why do unlabeled rows matter more than labeled ones?&lt;/strong&gt;&lt;br&gt;
They are 50 of 96 rows, and the two I could resolve at the top of the board turned out to be with-tools numbers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/gpqa-diamond-has-198-questions-what-the-hugging-face-leaderboards-115-rows-can-and-cannot-rank-2dlm"&gt;GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/when-a-04-point-lead-means-nothing-reading-open-model-leaderboard-margins-like-a-statistician-5do7"&gt;When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;HLE dataset and board: &lt;a href="https://huggingface.co/datasets/cais/hle" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/cais/hle&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Humanity's Last Exam paper: &lt;a href="https://arxiv.org/abs/2501.14249" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2501.14249&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Artificial Analysis HLE evaluation page (2,158 text-only questions): &lt;a href="https://artificialanalysis.ai/evaluations/humanitys-last-exam" rel="noopener noreferrer"&gt;https://artificialanalysis.ai/evaluations/humanitys-last-exam&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Kimi-K3 model card: &lt;a href="https://huggingface.co/moonshotai/Kimi-K3" rel="noopener noreferrer"&gt;https://huggingface.co/moonshotai/Kimi-K3&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Inkling model card: &lt;a href="https://huggingface.co/thinkingmachines/Inkling" rel="noopener noreferrer"&gt;https://huggingface.co/thinkingmachines/Inkling&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;gpt-oss paper: &lt;a href="https://arxiv.org/abs/2508.10925" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2508.10925&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All counts in this article are my own measurements from the board JSON and the cited cards on November 1, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. The classification of rows is mine, based on free-text notes, and the board endpoint is undocumented and may change.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>llm</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>SWE-Bench Pro Has Two Versions: What the Hugging Face Leaderboard's 47 Rows Mix Together</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Sat, 31 Oct 2026 13:48:24 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-pro-has-two-versions-what-the-hugging-face-leaderboards-47-rows-mix-together-37co</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/swe-bench-pro-has-two-versions-what-the-hugging-face-leaderboards-47-rows-mix-together-37co</guid>
      <description>&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;On October 31, 2026 the Hugging Face SWE-bench Pro board lists &lt;strong&gt;47 rows, 0 marked verified&lt;/strong&gt;. Row 1 is &lt;strong&gt;94.94&lt;/strong&gt;, row 2 is &lt;strong&gt;68.74&lt;/strong&gt;, and the median row is &lt;strong&gt;56.1&lt;/strong&gt;. A 26.2 point cliff invites suspicion, but the first question is not "is it fake". It is "which benchmark is each row on".&lt;/li&gt;
&lt;li&gt;SWE-bench Pro now exists in two versions. &lt;strong&gt;V1&lt;/strong&gt; has 731 tasks. &lt;strong&gt;V2&lt;/strong&gt; (September 22, 2026) has 642: 89 tasks were dropped, and the authors' paper says 480 of the 731 were repaired or retired. The board's official task points at V2, but the top row reports &lt;strong&gt;694/731 on V1&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;The top row is better documented than most: its evidence dataset holds a per-task result for all 731 ids. I checked it. The ids match V1 exactly, 694 are marked resolved, and the numbers in the README, summary file and results file agree.&lt;/li&gt;
&lt;li&gt;What the files &lt;strong&gt;cannot&lt;/strong&gt; show: how many attempts each task got, whether the self-improvement rounds saw the same 731 tasks, whether the network was restricted, and whether grading was repeated in a clean container. The model card does not say. It also gives the baseline as 57.7 in one place and 57.0 in another.&lt;/li&gt;
&lt;li&gt;Two other rows copy a number from a different repo. One is a quantized checkpoint carrying its base model's score, byte for byte.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Untouched%2520%2528251%2529%2522%252C%2522Repaired%2520%2528391%252C%2520derived%2529%2522%252C%2522Dropped%2520%252889%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522tasks%2522%252C%2522data%2522%253A%255B251%252C391%252C89%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523c53030%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522What%2520happened%2520to%2520the%2520731%2520SWE-bench%2520Pro%2520v1%2520tasks%2520before%2520V2%2520%2528642%2520kept%2529%2522%252C%2522fontSize%2522%253A18%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A450%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522tasks%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522horizontalBar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522Untouched%2520%2528251%2529%2522%252C%2522Repaired%2520%2528391%252C%2520derived%2529%2522%252C%2522Dropped%2520%252889%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522tasks%2522%252C%2522data%2522%253A%255B251%252C391%252C89%255D%252C%2522backgroundColor%2522%253A%255B%2522%25232b6cb0%2522%252C%2522%2523dd6b20%2522%252C%2522%2523c53030%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522What%2520happened%2520to%2520the%2520731%2520SWE-bench%2520Pro%2520v1%2520tasks%2520before%2520V2%2520%2528642%2520kept%2529%2522%252C%2522fontSize%2522%253A18%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A450%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522tasks%2522%257D%257D%255D%257D%257D%257D" alt="SWE-bench Pro v1 tasks: untouched, repaired and dropped" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What does the Hugging Face SWE-bench Pro board contain?
&lt;/h2&gt;

&lt;p&gt;The dataset repo &lt;code&gt;ScaleAI/SWE-bench_Pro&lt;/code&gt; is tagged as an official benchmark with an &lt;code&gt;eval.yaml&lt;/code&gt;. Models report scores through &lt;code&gt;.eval_results/*.yaml&lt;/code&gt; files in their own repos, and the board lists what those files say. I pulled the JSON behind the leaderboard view. I found no documentation for that endpoint, so treat it as unstable. Snapshot of October 31, 2026 (KST):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Property&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Rows&lt;/td&gt;
&lt;td&gt;47&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rows with &lt;code&gt;verified: true&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Top score / second score / median&lt;/td&gt;
&lt;td&gt;94.94 / 68.74 / 56.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Source is a model card&lt;/td&gt;
&lt;td&gt;39&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Source is a link to Scale's public leaderboard page&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Source is a technical report&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Source is an evidence dataset&lt;/td&gt;
&lt;td&gt;1 (the top row)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rows with a free-text &lt;code&gt;notes&lt;/code&gt; field&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rows whose notes name an agent harness&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The six rows that cite Scale's leaderboard are all older models (scores from 9.67 to 38.7). The six rows that name a harness name four different ones: Claude Code (three rows), LingCode, NexAU and SWE-Agent. That leaves 41 of 47 rows with no stated harness. In agentic coding the harness is part of the system, so a score without one is a score of "model plus unknown scaffold".&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%252C%252235%2522%252C%252236%2522%252C%252237%2522%252C%252238%2522%252C%252239%2522%252C%252240%2522%252C%252241%2522%252C%252242%2522%252C%252243%2522%252C%252244%2522%252C%252245%2522%252C%252246%2522%252C%252247%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2520%2525%2522%252C%2522data%2522%253A%255B94.94%252C68.74%252C67.7%252C66.1%252C65.7%252C65.1%252C62.5%252C62.2%252C62.1%252C61.7%252C61.2%252C61%252C59.6%252C59.4%252C59%252C58.6%252C58.4%252C58.4%252C57.9%252C57.2%252C56.6%252C56.3%252C56.2%252C56.1%252C55.9%252C55.4%252C55.4%252C54.3%252C53.5%252C51.2%252C50.7%252C50.4%252C49.5%252C49.2%252C47.6%252C47.5%252C46.3%252C44.5%252C44.3%252C42.9%252C40.2%252C38.7%252C36.81%252C27.67%252C21.41%252C16.2%252C9.67%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522All%252047%2520rows%2520on%2520the%2520Hugging%2520Face%2520SWE-bench%2520Pro%2520board%252C%2520by%2520rank%2520%2528Oct%252031%252C%25202026%2529%2522%252C%2522fontSize%2522%253A17%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520%2525%2522%257D%257D%255D%252C%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rank%2522%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%25221%2522%252C%25222%2522%252C%25223%2522%252C%25224%2522%252C%25225%2522%252C%25226%2522%252C%25227%2522%252C%25228%2522%252C%25229%2522%252C%252210%2522%252C%252211%2522%252C%252212%2522%252C%252213%2522%252C%252214%2522%252C%252215%2522%252C%252216%2522%252C%252217%2522%252C%252218%2522%252C%252219%2522%252C%252220%2522%252C%252221%2522%252C%252222%2522%252C%252223%2522%252C%252224%2522%252C%252225%2522%252C%252226%2522%252C%252227%2522%252C%252228%2522%252C%252229%2522%252C%252230%2522%252C%252231%2522%252C%252232%2522%252C%252233%2522%252C%252234%2522%252C%252235%2522%252C%252236%2522%252C%252237%2522%252C%252238%2522%252C%252239%2522%252C%252240%2522%252C%252241%2522%252C%252242%2522%252C%252243%2522%252C%252244%2522%252C%252245%2522%252C%252246%2522%252C%252247%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522score%2520%2525%2522%252C%2522data%2522%253A%255B94.94%252C68.74%252C67.7%252C66.1%252C65.7%252C65.1%252C62.5%252C62.2%252C62.1%252C61.7%252C61.2%252C61%252C59.6%252C59.4%252C59%252C58.6%252C58.4%252C58.4%252C57.9%252C57.2%252C56.6%252C56.3%252C56.2%252C56.1%252C55.9%252C55.4%252C55.4%252C54.3%252C53.5%252C51.2%252C50.7%252C50.4%252C49.5%252C49.2%252C47.6%252C47.5%252C46.3%252C44.5%252C44.3%252C42.9%252C40.2%252C38.7%252C36.81%252C27.67%252C21.41%252C16.2%252C9.67%255D%252C%2522backgroundColor%2522%253A%255B%2522%2523dd6b20%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%252C%2522%25232b6cb0%2522%255D%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522All%252047%2520rows%2520on%2520the%2520Hugging%2520Face%2520SWE-bench%2520Pro%2520board%252C%2520by%2520rank%2520%2528Oct%252031%252C%25202026%2529%2522%252C%2522fontSize%2522%253A17%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A0%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522self-reported%2520%2525%2522%257D%257D%255D%252C%2522xAxes%2522%253A%255B%257B%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522rank%2522%257D%257D%255D%257D%257D%257D" alt="All 47 rows on the board by rank" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two notes say the score was produced on the card's own "refined" version of the benchmark with problematic tasks corrected. Those rows are not on the same task set as their neighbours either.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is the difference between SWE-bench Pro V1 and V2?
&lt;/h2&gt;

&lt;p&gt;The dataset card says the &lt;code&gt;default&lt;/code&gt; config is now &lt;strong&gt;V2 (2026-09-22): 642 tasks&lt;/strong&gt; across 11 repositories (go 256, python 237, js 145, ts 4). The &lt;code&gt;hard&lt;/code&gt; config is a 51-task subset. The original 731-task release is preserved unchanged as config &lt;code&gt;v1&lt;/code&gt;. From the card:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;89 tasks were dropped after review for invalid or unrepairable tests.&lt;/li&gt;
&lt;li&gt;529 problem statements were rewritten so every graded assertion traces to a sentence in the text.&lt;/li&gt;
&lt;li&gt;214 test patches and 38 gold patches were revised, and test lists were regenerated against repaired parsers.&lt;/li&gt;
&lt;li&gt;Every image was rebuilt from a sanitised repository bundle with no fixing commit, stray refs, stashes or hooks.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The V2 paper (arXiv 2610.24206, submitted October 16, 2026) says audits, adjudication and blind re-implementation "repaired or retired 480 of the 731 tasks", which leaves 251 untouched. The chart above splits the 731 into untouched (251), dropped (89) and repaired (391, which is 480 minus 89, my arithmetic). The paper also adds a hardened protocol: network access only to the model endpoint, and every patch re-graded in a clean container. It reports that with the network open, 32 of 642 runs of one frontier model (5.0%) contacted code hosts and 25 downloaded the task's own repository.&lt;/p&gt;

&lt;p&gt;Scale's announcement also notes that the public repositories, their fixing commits and the benchmark itself have been on the open web since before the evaluated models were trained. That makes exposure a concern for V1 numbers in general. It is a statement about V1 as a measuring instrument, not about any one row.&lt;/p&gt;

&lt;p&gt;The consequence for a board: a V1 score over 731 tasks and a V2 score over 642 tasks have different denominators and, after the refresh, different task quality. The board's task id (&lt;code&gt;SWE_Bench_Pro&lt;/code&gt;) maps to the &lt;code&gt;default&lt;/code&gt; config, which is V2. Nothing in the row format says which version a number came from.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which version does the 94.94 row use?
&lt;/h2&gt;

&lt;p&gt;The top row, &lt;code&gt;PCL-RSI-lab/Seshat-agent&lt;/code&gt;, says "SWE-Bench Pro v1 (731), resolve rate 694/731=94.94%" in its notes. 694/731 is 94.938%, so the rounding is right. It is a V1 number on a V2 board. The 26.2 point gap to row 2 therefore mixes at least three things: model quality, harness, and benchmark version. For context, the V2 paper's abstract reports that frontier agents resolve up to 99.4% of the 642 V2 tasks under the hardened protocol, so a 94.94 on V1 is not out of range for a strong agent system by itself.&lt;/p&gt;

&lt;p&gt;The row links to an evidence dataset, which is the right thing to do and rare on this board. I read it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What can the evidence files confirm?
&lt;/h2&gt;

&lt;p&gt;I downloaded the result files and the official V1 task list and compared them. The short script below reproduces the headline checks (it uses two public HF endpoints and the evidence dataset):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="c1"&gt;# 1. the board (undocumented JSON behind the leaderboard view; treat as unstable)
&lt;/span&gt;&lt;span class="n"&gt;board&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/api/datasets/ScaleAI/SWE-bench_Pro/leaderboard&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verified&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;board&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verified&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 2. the task sets: 'default' is V2, 'v1' is the original release
&lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;off&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://datasets-server.huggingface.co/rows?dataset=ScaleAI/SWE-bench_Pro&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;config=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;split=test&amp;amp;offset=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;off&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;length=100&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;|=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;row&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;instance_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;
        &lt;span class="n"&gt;off&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;off&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;num_rows_total&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;
&lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;v1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;default&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;v1 tasks,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;V2 tasks,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;v2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dropped&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 3. the evidence behind the top row
&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://huggingface.co/datasets/PCL-RSI-lab/SWE-Bench-Pro-Evaluation-Evidence/resolve/main/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;ev&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;eval_results.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ev&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;graded;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ev&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resolved; same ids as v1:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ev&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;v1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resolved on the 642 retained ids:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ev&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ev&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;v2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output on October 31, 2026: &lt;code&gt;47 rows, 0 verified&lt;/code&gt;, &lt;code&gt;731 v1 tasks, 642 V2 tasks, 89 dropped&lt;/code&gt;, &lt;code&gt;731 graded; 694 resolved; same ids as v1: True&lt;/code&gt;, &lt;code&gt;resolved on the 642 retained ids: 610&lt;/code&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ids in the results file vs official V1 list&lt;/td&gt;
&lt;td&gt;identical, 731 of 731&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Resolved count vs README and summary file&lt;/td&gt;
&lt;td&gt;694 in all three&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unresolved / empty patches&lt;/td&gt;
&lt;td&gt;37 / 2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Submitted in two parts (no overlapping ids)&lt;/td&gt;
&lt;td&gt;617 ids with 588 resolved (95.30%), 114 ids with 106 resolved (92.98%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Resolved on the 642 ids that V2 kept&lt;/td&gt;
&lt;td&gt;610 of 642 (95.02%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Resolved on the 89 ids V2 dropped&lt;/td&gt;
&lt;td&gt;84 of 89 (94.38%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Resolved on the 51 ids in the &lt;code&gt;hard&lt;/code&gt; config&lt;/td&gt;
&lt;td&gt;41 of 51 (80.39%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;95% Wilson interval for 694/731&lt;/td&gt;
&lt;td&gt;93.1 to 96.3&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Per repository, the row resolves 100% of three repos (79 of 79, 56 of 56, 20 of 20 tasks) and 85.5% of the weakest (53 of 62).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522ansible%2520%252896%2529%2522%252C%2522openlibrary%2520%252891%2529%2522%252C%2522flipt%2520%252885%2529%2522%252C%2522qutebrowser%2520%252879%2529%2522%252C%2522teleport%2520%252876%2529%2522%252C%2522webclients%2520%252865%2529%2522%252C%2522vuls%2520%252862%2529%2522%252C%2522navidrome%2520%252857%2529%2522%252C%2522element-web%2520%252856%2529%2522%252C%2522NodeBB%2520%252844%2529%2522%252C%2522tutanota%2520%252820%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522resolve%2520rate%2520%2525%252C%2520one%2520submission%252C%2520v1%2520tests%2522%252C%2522data%2522%253A%255B98.96%252C91.21%252C89.41%252C100.0%252C96.05%252C95.38%252C85.48%252C98.25%252C100.0%252C93.18%252C100.0%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Per-repository%2520resolve%2520rate%2520of%2520the%252094.94%2525%2520row%2520%2528repo%2520and%2520task%2520count%2529%2522%252C%2522fontSize%2522%253A17%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A70%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520resolved%2522%257D%257D%255D%252C%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%252C%2522autoSkip%2522%253Afalse%257D%257D%255D%257D%257D%257D" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fquickchart.io%2Fchart%3Fw%3D1000%26h%3D420%26bkg%3Dwhite%26c%3D%257B%2522type%2522%253A%2522bar%2522%252C%2522data%2522%253A%257B%2522labels%2522%253A%255B%2522ansible%2520%252896%2529%2522%252C%2522openlibrary%2520%252891%2529%2522%252C%2522flipt%2520%252885%2529%2522%252C%2522qutebrowser%2520%252879%2529%2522%252C%2522teleport%2520%252876%2529%2522%252C%2522webclients%2520%252865%2529%2522%252C%2522vuls%2520%252862%2529%2522%252C%2522navidrome%2520%252857%2529%2522%252C%2522element-web%2520%252856%2529%2522%252C%2522NodeBB%2520%252844%2529%2522%252C%2522tutanota%2520%252820%2529%2522%255D%252C%2522datasets%2522%253A%255B%257B%2522label%2522%253A%2522resolve%2520rate%2520%2525%252C%2520one%2520submission%252C%2520v1%2520tests%2522%252C%2522data%2522%253A%255B98.96%252C91.21%252C89.41%252C100.0%252C96.05%252C95.38%252C85.48%252C98.25%252C100.0%252C93.18%252C100.0%255D%252C%2522backgroundColor%2522%253A%2522%25232b6cb0%2522%257D%255D%257D%252C%2522options%2522%253A%257B%2522title%2522%253A%257B%2522display%2522%253Atrue%252C%2522text%2522%253A%2522Per-repository%2520resolve%2520rate%2520of%2520the%252094.94%2525%2520row%2520%2528repo%2520and%2520task%2520count%2529%2522%252C%2522fontSize%2522%253A17%257D%252C%2522legend%2522%253A%257B%2522display%2522%253Afalse%257D%252C%2522scales%2522%253A%257B%2522yAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522min%2522%253A70%252C%2522max%2522%253A100%257D%252C%2522scaleLabel%2522%253A%257B%2522display%2522%253Atrue%252C%2522labelString%2522%253A%2522%2525%2520resolved%2522%257D%257D%255D%252C%2522xAxes%2522%253A%255B%257B%2522ticks%2522%253A%257B%2522fontSize%2522%253A10%252C%2522autoSkip%2522%253Afalse%257D%257D%255D%257D%257D%257D" alt="Per-repository resolve rate of the 94.94 row" width="2000" height="840"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two cautions about reading this table. The 642 and 51 ids are V2 task names, but V2 rewrote instructions and tests for many of them, so these are the same ids and not the same tasks. And the 89 dropped ids show almost the same rate as the retained ones, which means that rate cannot tell you whether the tests on those 89 were passable for the right reasons.&lt;/p&gt;

&lt;p&gt;I also compared the submitted patches with the reference patches. Of 731 patches, 729 are non-empty and none is byte-identical to the reference. For resolved tasks the median share of the reference patch's added lines that also appear in the submitted patch is 0.91, against 0.42 for the 35 non-empty unresolved ones. That is what correct fixes should look like. It says nothing about how the patches were produced. Separately, 111 of 729 patches touch a path that looks like a test file (by a simple name pattern), which is ordinary agent behaviour. Without a reference distribution from another system on the same tasks, none of these numbers can be read as evidence for or against anything. I report them so nobody has to recompute them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What can the files not confirm?
&lt;/h2&gt;

&lt;p&gt;The result files prove that 694 patches were marked resolved by some grader. They do not show:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Which grader and which environment.&lt;/strong&gt; The V2 protocol re-grades every diff on a pristine image with the network locked to the model endpoint. A V1 run on the older images carries no such guarantee, and the card does not describe the setup.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;How many attempts each task got.&lt;/strong&gt; Pass@1, best of several, or a loop that retries until the grader is satisfied are different measurements.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Whether the evolution rounds saw the test tasks.&lt;/strong&gt; The card describes a self-improving agent whose "successful traces feed back into the meta-knowledge base for the next round" and a curve that climbs "through successive self-improvement rounds" on SWE-Bench Pro. If the traces came from these same 731 tasks, the final number is a score on seen tasks. If they came from other tasks, it is a held-out score. The card does not say which. I searched its text for held-out, network, attempt, contamination, retry and sandbox, and none appears.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Internal consistency of the card.&lt;/strong&gt; The baseline is "57.7%" in the key-results table and "57.0" in the figure table. The headline is 94.94 in one table and 95.1 in the other (the second is read off a figure and labelled as such).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;None of this says the result is wrong. It says the row is a claim with a strong artifact trail and four open questions, and a reader who wants to rank it against other rows needs the answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  How much can 731 tasks separate?
&lt;/h2&gt;

&lt;p&gt;Even with a clean protocol, a task count limits resolution. At a resolve rate near 60%, one standard error over 731 tasks is 1.81 points. If two systems are scored on the same tasks, their outcomes are correlated, which shrinks the noise in the gap. The smallest gap that clears a 95% test is:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tasks&lt;/th&gt;
&lt;th&gt;rho = 0&lt;/th&gt;
&lt;th&gt;rho = 0.5&lt;/th&gt;
&lt;th&gt;rho = 0.8&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;731 (V1)&lt;/td&gt;
&lt;td&gt;5.0&lt;/td&gt;
&lt;td&gt;3.6&lt;/td&gt;
&lt;td&gt;2.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;642 (V2)&lt;/td&gt;
&lt;td&gt;5.4&lt;/td&gt;
&lt;td&gt;3.8&lt;/td&gt;
&lt;td&gt;2.4&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Nobody publishes the per-task correlation between two vendors' runs, so rho is an assumption. The practical reading: on this board, twelve rows (ranks 4 to 15) sit inside a 7.1 point band, 66.1 down to 59.0. That is about the size of one unpaired detectable gap. Across different harnesses and task versions it is a cluster, not a ranking.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why do two rows repeat a number from another repo?
&lt;/h2&gt;

&lt;p&gt;Two rows cite a model card in a different repository:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound&lt;/code&gt; shows 58.4 with source &lt;code&gt;zai-org/GLM-5.1&lt;/code&gt; and the note "high reasoning". Its YAML is identical to the base model's file. A 4-bit quantized checkpoint can score differently from its base, and nothing here says it was measured.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;MuVeraAI/Laguna-XS.2&lt;/code&gt; shows 44.5 and cites &lt;code&gt;poolside/Laguna-XS.2&lt;/code&gt;. That card's own file says 46.3. The copy and its cited source disagree by 1.8 points, and there is no note explaining why.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Other rows (the &lt;code&gt;ornith-ai&lt;/code&gt; ones) cite a card under another organization name, which looks like a rename or mirror and is harmless. The point is that the board will list a row for any repo that carries a YAML file, and the only provenance field is a URL the repo owner typed.&lt;/p&gt;

&lt;h2&gt;
  
  
  A checklist for reading an agent-benchmark row
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Which task version and how many tasks?&lt;/strong&gt; V1 731, V2 642, a "refined" set, or the Hard 51 are different benchmarks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Which harness and model, together?&lt;/strong&gt; A bare model name with no scaffold is half a description.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Is there a per-task artifact?&lt;/strong&gt; Per-task results and patches let you recompute the headline. A bare number does not.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Does the artifact trail answer attempts, network policy and held-out status?&lt;/strong&gt; If not, write "unknown" next to the row.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Does the row measure the repo it sits in?&lt;/strong&gt; Quantized and fine-tuned copies often inherit the base model's number.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Is the gap bigger than the noise?&lt;/strong&gt; At 731 tasks, anything under about 4 points needs a paired test.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is the SWE-bench Pro V2 benchmark?&lt;/strong&gt;&lt;br&gt;
A revised public split of SWE-bench Pro with 642 tasks in 11 repositories, released September 22, 2026. It drops 89 invalid tasks, rewrites instructions, repairs tests, and ships images without the fixing commit. V1, with 731 tasks, is preserved as a separate config.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I compare a V1 score with a V2 score?&lt;/strong&gt;&lt;br&gt;
Not directly. The denominators differ (731 vs 642), and V2 changed instructions, tests and images for hundreds of tasks. The V2 authors also flag public exposure of the V1 tasks as a concern.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does the Hugging Face leaderboard verify scores?&lt;/strong&gt;&lt;br&gt;
On this board, no. All 47 rows are marked &lt;code&gt;verified: false&lt;/code&gt;. The board displays what each repo's &lt;code&gt;.eval_results&lt;/code&gt; file declares, so treat each row as a claim with a source link.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is the 94.94 result fake?&lt;/strong&gt;&lt;br&gt;
This analysis found no inconsistency in the evidence files, and it also cannot confirm the protocol. The files agree with each other and with the official V1 task list. The open questions are attempts, network policy and whether self-improvement rounds saw the test tasks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What would make the row comparable to the others?&lt;/strong&gt;&lt;br&gt;
A run on the V2 &lt;code&gt;default&lt;/code&gt; config under the locked protocol, with the harness, attempt count and training-task split stated.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is the minimum gap that means something at 731 tasks?&lt;/strong&gt;&lt;br&gt;
About 5 points for independent runs near 60% accuracy, and roughly 2 to 4 points when both systems answer the same tasks and their outcomes are correlated. The correlation is an assumption until someone publishes it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/gpqa-diamond-has-198-questions-what-the-hugging-face-leaderboards-115-rows-can-and-cannot-rank-2dlm"&gt;GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.to/ward_ed_6b5e6aa8ded94a987/how-hugging-face-official-benchmark-leaderboards-actually-work-evalresults-yaml-the-basemodel-29bo"&gt;How Hugging Face Official Benchmark Leaderboards Actually Work&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;SWE-bench Pro dataset card and configs: &lt;a href="https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;SWE-Bench Pro V2 paper: &lt;a href="https://arxiv.org/abs/2610.24206" rel="noopener noreferrer"&gt;https://arxiv.org/abs/2610.24206&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Scale announcement: &lt;a href="https://labs.scale.com/blog/swe-bench-pro-v2" rel="noopener noreferrer"&gt;https://labs.scale.com/blog/swe-bench-pro-v2&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Top row model card and evidence dataset: &lt;a href="https://huggingface.co/PCL-RSI-lab/Seshat-agent" rel="noopener noreferrer"&gt;https://huggingface.co/PCL-RSI-lab/Seshat-agent&lt;/a&gt; and &lt;a href="https://huggingface.co/datasets/PCL-RSI-lab/SWE-Bench-Pro-Evaluation-Evidence" rel="noopener noreferrer"&gt;https://huggingface.co/datasets/PCL-RSI-lab/SWE-Bench-Pro-Evaluation-Evidence&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All counts in this article are my own measurements from these sources on October 31, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. The board JSON endpoint is undocumented and may change.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>llm</category>
      <category>benchmark</category>
    </item>
    <item>
      <title>Reading a Hardware-Equivalence Claim Like an Evaluator: A Checklist Applied to an NPU vs GPU Pretraining Twin</title>
      <dc:creator>Ward Ed</dc:creator>
      <pubDate>Mon, 26 Oct 2026 06:12:21 +0000</pubDate>
      <link>https://dev.to/ward_ed_6b5e6aa8ded94a987/reading-a-hardware-equivalence-claim-like-an-evaluator-a-checklist-applied-to-an-npu-vs-gpu-3cfb</link>
      <guid>https://dev.to/ward_ed_6b5e6aa8ded94a987/reading-a-hardware-equivalence-claim-like-an-evaluator-a-checklist-applied-to-an-npu-vs-gpu-3cfb</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: this account is run by members of the VIDRAFT team, and the work discussed below is VIDRAFT's own result. It is not an independent review.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;p&gt;"Matches the GPU" is easy to say and hard to show. Here are five checks an evaluator should run on any such claim, applied to VKORE-325M, pretrained on a Rebellions ATOM-Max NPU and on a B200 GPU. It passes four; the fifth (seed variance) is still open.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check 1: Is there a true twin?
&lt;/h2&gt;

&lt;p&gt;The reference must differ only in hardware. Here initialization, data, batch and schedule were identical.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fhuggingface.co%2FFINAL-Bench%2FVKORE-325M-NPU-Base%2Fresolve%2Fmain%2Farticle%2Ffig6_twin_design.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fhuggingface.co%2FFINAL-Bench%2FVKORE-325M-NPU-Base%2Fresolve%2Fmain%2Farticle%2Ffig6_twin_design.png" alt="Twin design" width="800" height="385"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg8yjy8vynqyh2f6oggya.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg8yjy8vynqyh2f6oggya.png" alt="Same schedule" width="800" height="262"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Check 2: Is the evaluation data really unseen?
&lt;/h2&gt;

&lt;p&gt;Read the sampler, not the split table. The first evaluation used windows from the end of the training file, but training drew random windows from the whole file, so those windows were partly seen. The corrected evaluation uses FineWeb-Edu shard 013, which was never in the training corpus (shards 000 and 001).&lt;/p&gt;

&lt;h2&gt;
  
  
  Check 3: Is the comparison paired?
&lt;/h2&gt;

&lt;p&gt;Window-to-window variance dwarfs model differences. On identical windows the paired standard error is 0.0007, versus about 0.0049 for each model's mean.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy50kh9fr17pvf4zzc3vx.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy50kh9fr17pvf4zzc3vx.png" alt="Paired differences" width="800" height="349"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Loss, unseen (1,024 windows)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Rebellions ATOM-Max NPU&lt;/td&gt;
&lt;td&gt;0.7993&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B200 GPU&lt;/td&gt;
&lt;td&gt;0.7994&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Check 4: Can you rerun it?
&lt;/h2&gt;

&lt;p&gt;Both checkpoints and evaluate_unseen.py are public in &lt;a href="https://huggingface.co/FINAL-Bench/VKORE-325M-NPU-Base" rel="noopener noreferrer"&gt;the model repo&lt;/a&gt;. An evaluator can reproduce the table without trusting the authors.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check 5: Is variance across seeds reported?
&lt;/h2&gt;

&lt;p&gt;Not yet: one run per hardware. A stronger claim would repeat each side with several seeds.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What would falsify the claim?&lt;/strong&gt; A paired difference well outside two standard errors on fresh data, or failure to reproduce with the published files.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does the claim cover speed?&lt;/strong&gt; No, only quality.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Technical report: &lt;a href="https://huggingface.co/blog/FINAL-Bench/pretraining-a-325m-language-model-from-zero-on-an" rel="noopener noreferrer"&gt;https://huggingface.co/blog/FINAL-Bench/pretraining-a-325m-language-model-from-zero-on-an&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>benchmark</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
