DEV Community

Cover image for Codex is a top-4 harness for GLM-5.3-Flash. On real repos it's 10th of 11.
Ian
Ian

Posted on

Codex is a top-4 harness for GLM-5.3-Flash. On real repos it's 10th of 11.

Two public leaderboards published rankings this week for OpenAI's Codex CLI wrapped around the same open model, GLM-5.3-Flash.

One ranks it 4th of 22 harnesses by pass rate. The other ranks it 10th of 11.

Neither board is wrong. They asked different questions, and the gap between them is the most useful thing I found in either one.

Two boards, one question

The first is Runta's FrontierHarness Eval v1.1, which hit the Hacker News front page on Tuesday (344 points). It ran 22 harness configurations around 5 open models on Baseten, 62 tasks, 4 attempts each: 27,280 runs. For each model it names a best-value harness, the lowest cost per pass unless another one passes at least 5 points more often for at most 30% more money. The live leaderboard lets you pick your model and read the answer.

The second is Joel Niklaus's Harness Leaderboard on Hugging Face, trending there the same day. It runs 11 harnesses around 4 open models, 3 attempts each (two of the models at both low and high reasoning effort), and it puts harnesses it can't separate from the leader into one tied group.

Two models appear on both: GLM-5.3-Flash and DeepSeek-V4.1-Flash. So I pulled both boards' numbers and lined them up. Runta's per-model figures ship as data inside the post's own page; the Hugging Face board has an open JSON endpoint. No account needed for either.

Where they agree

Pi, the small four-tool harness, is the cheapest harness per pass on GLM-5.3-Flash on both boards: $0.223 per pass on Runta's, $0.132 at low effort and $0.184 at high on the Hugging Face one. The dollars don't compare (different providers, different prices), but the order does. Pi's pass rate is middle of the pack on Runta's board (8th of 22) and inside the tied first group on the other.

With DeepSeek-V4.1-Flash, both boards say the same thing about Pi from the other side: cheap, but behind on passes. It's 15th of 22 on Runta's board and sits in the second group at both efforts on Hugging Face's.

That's the kind of answer I'd act on. Two teams, two setups, two task lists, one conclusion.

Where they split

Now Codex with GLM-5.3-Flash. On Runta's board it passes 59.3% of attempts, 4th of 22, and it's the second cheapest per pass. On the Hugging Face board it's 10th of 11 at both efforts, ahead of only OpenClaw.

The Hugging Face board scores two benchmarks, so I split the score:

GLM-5.3-Flash, Hugging Face board v0.1.8 (Nov 17)
                 Terminal-Bench 2.1   DeepSWE v1.1
codex   low         60.2%  (8th)      26.7%  (10th)
codex   high        67.4%  (5th)      35.1%  (10th)
top     low         64.7%             55.9%
top     high        71.3%             63.4%
Enter fullscreen mode Exit fullscreen mode

On terminal tasks Codex is fine: middle of the pack, close to the top. On DeepSWE, long feature and bug-fix work in real Go, Python, TypeScript and Rust repos, it falls 28 to 29 points behind the leader.

Then I counted Runta's task list. All 62 tasks are Terminal-Bench: 45 from version 2.1 and 17 from version 4. There is no repo work in it at all. Runta's board never asked Codex the question it fails on the other board.

It isn't Codex everywhere, either. With DeepSeek-V4.1-Flash, Codex scores 61.3% and 64.0% on DeepSWE, 6th of 11, and sits in the tied top group. The weak spot is this harness, this model, this kind of task.

Same name, different harness

I can't tell you why from what's published. But the two boards didn't run quite the same Codex.

Runta's suite file pins Codex 0.164.0, tells it the model's window is 1,048,576 tokens, and caps tool output at 10,000 tokens. The Hugging Face board ran 0.162.1, and its settings doc says the trials now on the board used Codex's fallback for a model it doesn't recognize: a 272,000-token window, tool output cut at 10,000 bytes, and no built-in patch tool (the prompt tells the model to run apply_patch through the shell). Codex speaks only OpenAI's Responses format, so each board puts a translator in front of the model: a LiteLLM proxy on Runta's, a responses.js bridge on Hugging Face's.

Whether any of that explains a 28-point gap on repo work is my guess, not a finding. It does show that "Codex" in a leaderboard row is Codex plus somebody's adapter. The Hugging Face changelog makes the same point in public: when OpenClaw moved to a new version and started receiving each model's context window, reply limit and image input, its DeepSeek scores rose 16 to 21 points in one release.

What I'd actually do with these boards

Runta's headline advice is to rethink your harness when you change models. I'd go one step further. A row on a harness leaderboard ranks a model, a harness, a kind of task and an adapter, all at once. The best harness for your model is the one two boards agree on for the kind of work you do.

Runta's own numbers back the caution. When they redrew the 62 tasks 4,000 times, only Codex with GLM-5.3 (the larger GLM) stayed the best-value pick nearly every time, 99.6%. The other four winners held in 25% to 53% of the draws, and the post puts a 95% interval of about plus or minus 10 points on each pass rate. Majromax made the same point in the HN thread: a second sample might not keep these conclusions.

On HN, andai wrote that benchmarks crossing models with harnesses find models doing far better in their "own" harness. Runta's page carries the data for that test too, though the post doesn't chart it. Z.ai's ZCode with GLM-5.3 is the one clear case: 4.6 points above the other harnesses' average (their p = 0.03). Moonshot's KimiCode with Kimi-K3 ranks 14th of 22 by pass rate. DeepSeek's four harness profiles land 7th to 14th with their own model.

Where it gets shaky

Runta builds the runtime these trials ran on, and its founder said so at the top of the HN thread. You can read the data and the task list, but rerunning the eval needs a Runta login and a Baseten key. Every model is open, served by one provider, so closed models and other caches could reorder everything.

The Hugging Face board moves under you. It's at v0.1.8, and its changelog is a long run of cells re-run after plumbing fixes. That's honest, and it means a rank from last week may not be today's.

And neither board is your codebase.

Try it

The Hugging Face board answers from the terminal, no key:

curl -s https://joelniklaus-harness-leaderboard.hf.space/api/index | jq -r '
  .leaderboard["zai-org/GLM-5.3-Flash"].stages.full.efforts.high.harnesses
  | to_entries | sort_by(-.value.macro)[]
  | [.key, .value.benchmarks["terminal-bench-2.1"].rate, .value.benchmarks["deep-swe"].rate, .value.cost_per_pass]
  | @tsv'
Enter fullscreen mode Exit fullscreen mode

Swap in deepseek-ai/DeepSeek-V4.1-Flash or low effort. Then open Runta's leaderboard, pick the same model, and see which names show up near the top of both.

Which harness do you run your open model in, and did you pick it for the model or for the kind of work you give it?

Top comments (1)

Collapse
 
gautamtalksdev profile image
Gautam K •

Interesting that leaderboard rank and real repo rank differ that much. Benchmarks reward the tasks they contain, so testing on your own repos is the only number that matters for your team.