DEV Community

Cover image for WildClawBench on Hugging Face: Rank 1 Is a Runtime, Because Scores, Minutes and Dollars Share One Ranked Column
Ward Ed
Ward Ed

Posted on

WildClawBench on Hugging Face: Rank 1 Is a Runtime, Because Scores, Minutes and Dollars Share One Ranked Column

TL;DR

  • On November 11, 2026 the Hugging Face board for the WildClawBench agent benchmark lists 52 rows from 23 open-weight models, and the rows are three different measurements in one column: 24 are overall scores in percent, 16 are total minutes and 12 are total dollars. The board ranks them together, high to low, and all 52 rows carry lower_is_better: false.
  • Ranks 1 to 16 are all minutes. Rank 1 is 708, the slowest model's runtime for the 60 tasks. The 24 score rows sit at ranks 17 to 42 mixed with 2 cost rows, and the 10 cheapest-looking rows fill ranks 43 to 52, so the cheapest run is ranked last.
  • The values themselves are right. For 14 of the 15 models that also appear in the benchmark's own README table, every value on the board matches it. The problem is the ranking and the units, not the data. The one exception is a model-card score for Hy3 that is 3.9 points above the README's.
  • Read one metric at a time, the picture is useful. Among the 12 models that have all three rows, GLM-5.2 scores 54.2 against Kimi-K3's 54.5 at 43% of the total cost ($17.10 against $40.08). Score tracks cost (rank correlation 0.82) and barely tracks time (0.16).
  • Score gaps are small for a 60-task suite: 13 of the 23 steps between neighbouring score rows are 1.0 point or less, and the README shows a model moving by up to 18 points when only the agent harness changes.

WildClawBench board rows in board order, colored by what they measure: total minutes, score, and total cost

What does the WildClawBench board on Hugging Face rank?

WildClawBench is a benchmark of 60 hand-built agent tasks in six categories (productivity, code, social interaction, search, creative synthesis and safety). Each task runs in its own Docker container inside an agent harness, and each metric is scored from 0.00 to 1.00. The README says judge-based metrics use an LLM judge, defaulting to openai/gpt-5.4. The reported "overall" score follows a weighted multimodal and pure-text breakdown, so it is not a count of passed tasks.

The dataset page carries a leaderboard built the usual way: model repos publish a YAML file under .eval_results/ and the board collects them. The board JSON is public at /api/datasets/internlm/WildClawBench/leaderboard. This dataset's eval.yaml declares three tasks, not one:

task_id Name in eval.yaml Unit
overall Overall score, percent
avg_time Avg Time (min) minutes
avg_cost Avg Cost ($) US dollars

Each model repo's YAML has up to three entries, one per task_id. The board JSON has no task_id field. Each row carries a value, a rank and a lower_is_better flag, and the rows are sorted by value, descending.

Why is rank 1 a runtime and not a model?

Because the sort does not know the units. Minutes for a 60-task suite run into the hundreds and dollars into the tens, while scores top out near 60, so a descending sort sorts by unit first. The first 16 ranks are minutes, then the 24 scores arrive with two dollar rows (ranks 34 and 37) interleaved, and the rest is dollars.

The same model shows up three times. Kimi-K3 is at rank 9 (488 minutes), rank 19 (54.5 percent) and rank 34 ($40.08). Qwen3.8-2.4T-A95B is rank 1 for 708 minutes and rank 18 for its 56.2 score. A reader who scans for the top of the column sees a list of the slowest runs.

There is also a direction problem. Time and cost are lower-is-better, and every row says lower_is_better: false. The cheapest runs on the board, Kimi-K2.5 and Step-3.5-Flash at $6.60 each, are ranks 51 and 52, the bottom of the column, for being cheap.

I also tried the obvious workaround. Adding ?task_id=avg_time or ?task=overall to the board endpoint returns the same 52 rows, with 708 first. The dataset page embeds the same list in the props of its leaderboard component, also starting with 708, and I found no task selector in those props. I did not render the page in a browser, so I cannot say what a visitor sees beyond that.

How can you check which unit each row is in?

Read each row's source file and match the row's value to the entry whose task_id it came from. All 52 values resolved to exactly one entry. This reproduces every number in this post:

import collections, json, urllib.request, yaml

def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    return urllib.request.urlopen(req, timeout=120).read().decode("utf-8")

DS = "internlm/WildClawBench"
board = json.loads(get(f"https://huggingface.co/api/datasets/{DS}/leaderboard"))
print(len(board), "rows;", sum(r["lower_is_better"] for r in board), "marked lower-is-better")

declared = yaml.safe_load(get(f"https://huggingface.co/datasets/{DS}/raw/main/eval.yaml"))["tasks"]
print("declared tasks:", {t["id"]: t["name"] for t in declared})

cache, rows = {}, []
for r in board:
    key = (r["modelId"], r.get("pullRequest"), r["filename"])
    if key not in cache:
        ref = f"refs%2Fpr%2F{key[1]}" if key[1] else "main"
        cache[key] = yaml.safe_load(get(f"https://huggingface.co/{key[0]}/raw/{ref}/{key[2]}"))
    hit = {e["dataset"]["task_id"] for e in cache[key] if float(e["value"]) == float(r["value"])}
    rows.append((r["rank"], r["modelId"], r["value"], hit.pop() if len(hit) == 1 else "ambiguous"))

print("rows per task_id:", dict(collections.Counter(t for *_, t in rows)))
print("ranks 1-16 by task_id:", dict(collections.Counter(t for *_, t in rows[:16])))
print("rank 1:", rows[0])
print("rank 52:", rows[-1])
by_model = collections.defaultdict(dict)
for rank, model, value, task in rows:
    by_model[model].setdefault(task, []).append(rank)
print("Kimi-K3 ranks by task_id:", by_model["moonshotai/Kimi-K3"])

for model, per_task_min, per_task_usd in (("zai-org/GLM-5", 6.22, 0.19), ("MiniMaxAI/MiniMax-M2.7", 9.18, 0.12)):
    t = next(v for _, m, v, k in rows if m == model and k == "avg_time")
    c = next(v for _, m, v, k in rows if m == model and k == "avg_cost")
    print(model, "board:", t, "min,", c, "USD | README per-task x 60:", round(per_task_min * 60), "min,", round(per_task_usd * 60, 2), "USD")
Enter fullscreen mode Exit fullscreen mode

On November 11 it prints:

52 rows; 0 marked lower-is-better
declared tasks: {'overall': 'Overall', 'avg_time': 'Avg Time (min)', 'avg_cost': 'Avg Cost ($)'}
rows per task_id: {'avg_time': 16, 'overall': 24, 'avg_cost': 12}
ranks 1-16 by task_id: {'avg_time': 16}
rank 1: (1, 'Qwen/Qwen3.8-2.4T-A95B', 708, 'avg_time')
rank 52: (52, 'stepfun-ai/Step-3.5-Flash', 6.6, 'avg_cost')
Kimi-K3 ranks by task_id: {'avg_time': [9], 'overall': [19], 'avg_cost': [34]}
zai-org/GLM-5 board: 373 min, 11.4 USD | README per-task x 60: 373 min, 11.4 USD
MiniMaxAI/MiniMax-M2.7 board: 551 min, 7.2 USD | README per-task x 60: 551 min, 7.2 USD
Enter fullscreen mode Exit fullscreen mode

(The last two lines use per-task averages from the harness table in the dataset README. It needs pip install pyyaml.)

One model, three board ranks for the 12 models that have a score, a minutes and a cost row

Are the numbers on the board correct?

Yes, as far as I can check, and that is what makes this a presentation problem rather than a data problem. The dataset README has its own model table for the OpenClaw harness. Fifteen of its 31 models also have rows on the board (the other 16 are mostly closed models, plus a few open ones without a row). For 14 of those 15, every value the board carries equals the README's score, minutes or dollars. Two small catches:

  • The label says average, the values are totals. eval.yaml calls the tasks "Avg Time (min)" and "Avg Cost ($)", but the board's 373 minutes for GLM-5 is the README's 6.22 minutes per task multiplied by 60, and its $11.40 is $0.19 multiplied by 60. The README states this ("multiplied by 60 for the full 60-task suite"). A reader who trusts the label reads 708 as an average of 11.8 hours per task. It is the total, 11.8 hours across the suite, or about 11.8 minutes per task.
  • One score does not match. Hy3 is listed at 53.6 from its model card, while the README's OpenClaw table has 49.7. That is 3.9 points between two figures for the same model, and the board does not say what harness the card used.

Two other README values are absent from the board: Kimi-K2.7-Code has a total cost of $72.31 there but no cost row on the board, and four models (Qwen3.8-2.4T-A95B, Kimi-K2.7-Code, Intern-S2-Preview-397B and Qwen3.8-27B) have a score and minutes but no dollar row. Only 12 of the 23 models have all three rows.

What does each column look like when read on its own?

Split by task_id, each metric has a sensible ordering. The score column, highest first, starts with dots3-note-prev at 61.7, then Qwen3.8-2.4T-A95B at 56.2, Kimi-K3 at 54.5, GLM-5.2 at 54.2 and Hy3 at 53.6.

Where the scores come from matters. 18 of the 24 score rows come from a source named "WildClawBench", the benchmark team's own site, and the other 6 from model cards (dots3-note-prev, Hy3, Moderato-V1-Pro, Muse-Glimmer-30B and two openPangu-2.0-Flash rows). The top of the score column, 61.7, is a model-card figure. The highest row with the benchmark team as its source is 56.2. Two of the 24 score rows are the same model: openPangu-2.0-Flash appears at 41.5 and 35, and only the notes field ("Avg@3, Thinking" and "Avg@3, Non-Thinking") tells them apart. Those two are the only rows with notes on the whole board.

Provenance is also thin. All 52 rows are verified: false. 50 of them come from 21 pull requests that are still open on the model repos, and 2 (Nex-N2-Pro and Moderato-V1-Pro) from a main branch. The board JSON has no date, but the YAML files do: 29 rows are dated May 22, 2026, 21 are dated between July 1 and August 16, and 2 have none.

How big is a gap on a 60-task suite?

Smaller than the ranking suggests. Scores are weighted rubric means, not pass counts, so I cannot convert them to tasks. As a scale, if the 60 tasks carried equal weight, one task would be 1.67 points, and the README describes the overall as a weighted multimodal and pure-text breakdown, so treat that as a rough unit rather than an exact one. Among the 24 score rows, 13 of the 23 steps between neighbours are 1.0 point or less: Kimi-K3 to GLM-5.2 is 0.3, Hy3 to Nex-N2-Pro is 0.1, and GLM-5.1 (48.2) to Qwen3.8-27B (48.0152) is 0.18. The board lists no run-to-run spread and no per-task data, so nothing on it shows whether a 0.3 point gap is real. The dataset has a separate trajectories repo, which is where per-task evidence would have to come from.

Qwen3.8-27B's 48.0152 is also the only value on the board with more than two decimals.

What does cost add?

Cost and time answer different questions, and the board's column cannot show it. Across the 12 models with all three rows, the rank correlation between score and total cost is 0.82, and between score and total time only 0.16. A slow run is not a better run, and a better run is usually a pricier one, with exceptions.

Score against total cost for the 12 models that have all three rows

GLM-5.2 is the clearest exception: 54.2 against Kimi-K3's 54.5, at $17.10 against $40.08 and 442 against 488 minutes. A 0.3 point score gap is inside any plausible noise on 60 tasks, while the cost gap is 2.3 times. Cost here is tokens multiplied by prices at evaluation time, so it mixes how many tokens a model burns with what its provider charges. The README documents the rates it used for only two models, Claude Opus 4.8 and Kimi-K2.7-Code, so I would treat the dollar column as indicative.

Does the harness change the order?

It does, and the board does not say which harness a row used. The README says its model leaderboard runs on the OpenClaw harness, so the rows sourced from the benchmark team's site are presumably OpenClaw. The 6 model-card rows state none. The README's harness table runs four models on four scaffolds (OpenClaw, Claude Code, Codex and Hermes Agent) with the same 60 tasks and the same grading:

Model Lowest score Highest score Spread Best harness
GPT-5.4 48.4 56.8 8.4 Codex
GLM 5 31.0 46.4 15.4 Hermes Agent
MiMo V2 Pro 29.9 48.1 18.2 Hermes Agent
MiniMax M2.7 32.0 37.1 5.1 Hermes Agent

On OpenClaw, GLM 5 (42.6) is ahead of MiMo V2 Pro (40.2). On Hermes Agent the order flips: 48.1 against 46.4. A spread of 5 to 18 points from the scaffold alone is bigger than almost every gap on the score column, so a model-card figure and a benchmark-team figure are not the same experiment unless the harness is given.

What else on the board is worth knowing?

  • Ties get different ranks. Kimi-K2.5 and Step-3.5-Flash are both $6.60 and hold ranks 51 and 52.
  • Closed models are absent. That is expected for a board built from model-repo YAML files, but it means the README's top 7 (all closed) are missing, and the best score on the board (61.7) is not the best on the benchmark (67.2 in the README).
  • No row has a task field. The page's embedded data has the same keys as the API (rank, value, source, notes, lower_is_better and a few more), so the unit has to be inferred from the source file.
  • Some rows are old. 29 of the 52 rows are dated May 22, 2026, and the newest is dated August 16, so the board is a snapshot of two or three batches, not a live view.

What can a reader do with this board?

  1. Check the unit before the rank. If a "score" column has values above 100, it is not a score.
  2. Split by task_id. Fetch the YAML, or use the code above, and sort each metric in its own direction.
  3. Treat time and cost as lower-is-better. The board's flag says otherwise.
  4. Look for the harness. Without one, a card figure and a table figure can differ by 5 to 18 points for reasons unrelated to the model.
  5. Do not over-read sub-1-point gaps on a 60-task suite with a judge-graded score.

For the board, three changes would remove most of the confusion: group rows by task_id, carry the direction (lower_is_better) per task, and show the harness. The YAML already carries the task_id, so the information exists in the files and is dropped when the board is assembled.

FAQ

What is the WildClawBench leaderboard on Hugging Face?
It is the board on the dataset page for internlm/WildClawBench, a suite of 60 agent tasks. Model repos publish YAML result files and the board collects them. As of November 11, 2026 it has 52 rows from 23 models.

Why is rank 1 on the WildClawBench board a time of 708?
Because the board mixes three tasks (overall score, total minutes and total cost) in one descending column. 708 is the total minutes of the slowest run, not a score, and all 16 time rows rank above all 24 score rows.

Are the WildClawBench numbers on Hugging Face wrong?
No. For 14 of the 15 models that also appear in the benchmark's own README table, every value matches. The ordering and the missing direction flag are the problem, and so is one model-card score (Hy3) that is 3.9 points above the README's.

Is "Avg Time" on the board an average per task?
The label says average, but the values are totals for the 60-task suite. The README multiplies the per-task averages by 60, and the board's 373 minutes for GLM-5 equals 6.22 minutes per task times 60.

How should I compare models on WildClawBench?
Compare scores only within one harness, ignore gaps under about 1 point, and read cost and time as lower-is-better in their own columns. The benchmark's README table gives the harness and the per-task averages, and for the open-weight models it matches the board.

Further reading

Top comments (0)