Kimi K3 vs Claude Fable 5 and Opus 4.8: a benchmark you can run yourself

Kimi K3 was released this week, and like every model release it's being judged on leaderboard scores and screenshots. But a score is a bit like a football result, in that it tells you who won, not how the game was played. You wouldn't sign a player off a scoreline alone — you'd want to watch the tape. With code models, the tape is the code itself: how it's structured, whether you'd actually want to maintain it. That's what leaderboards can't show you.

So we built a benchmark that compares Kimi K3, Claude Fable 5 and Claude Opus 4.8 on exactly that — and it's open source, so you can run it yourself at github.com/dsplce-co/kimi-vs-fable-vs-opus.

This note summarises our findings after running it.

The task

Given a double-entry ledger module with an existing test suite, make it production-ready and add two features: transaction reversal and statement generation. There's no hint that there are any bugs to find.

But the existing tests aren't actually good — they don't test for real money-safety bugs:

• A transfer that can lose money if its second half doesn't complete (the transfer isn't atomic).
• Amounts drifting by fractions of a cent due to floating-point precision.
• An accessor that exposes internals and lets callers corrupt the ledger.

So passing these tests proves only demo quality — that the core functionality works, nothing more. To grade the results we used a second, hidden test suite that checks the actual money safety — one the models never see.

Same harness, on purpose

All three models were run in the same harness:

• Agentically inside Claude Code (opening and editing files, running tests, reacting to failures).
• In a high-effort setting.
• Three runs each, to see some run-to-run variance.

We made sure to run Kimi K3 inside Claude Code too, just like the Claudes, so the harness itself isn't a variable here. The results reflect how well each model does as an agent inside Claude Code — how it would work with a dev team using it.

Results

We're framing the analysis around how each model wrote the code, rather than the pass/fail binary.

What every model got right

All three, across all nine runs:

• Properly fixed the atomicity bug — first validate all the transaction's lines (this pass can throw), then commit in a second pass that cannot throw, forbidding partial transactions:

// validate everything first — this part is allowed to throw
for (const line of lines) {
  const acc = accounts.get(line.accountId);
  if (acc.state !== "open") throw new Error(`account not open: ${line.accountId}`);
}
// then commit — from here nothing throws, so it's all-or-nothing
for (const line of lines) this.entries.push(makeEntry(line));

• Implemented the reverse feature as a compensating transaction routed through the existing post function, inheriting its atomicity and lifecycle checks.
• Fixed the accessor leak that would let outside callers mutate the ledger's history.

So fixing the known bug and adding the features is a solved problem for all of them. They're interchangeable on that basic bar — the differentiation only starts to appear with the harder questions about the code's long-term behaviour.

Claude Fable 5

• Most consistent across all three runs; the same load-bearing choices each time.
• Wrote a self-test in each run that verifies encapsulation as an invariant — it tries to corrupt the ledger through the returned object and asserts that it fails, so if a future dev removes the defensive copy, the test breaks immediately instead of a silent leak shipping to production — which is like the difference between doing the safe thing and making the safe thing hard to undo.
• The only differences between runs were in degree of caution (one run reserved a namespace, guarded against overflow, threw on unknown accounts instead of returning zero). No run shipped a bug.

Its limitation: it stores money as a bare number type — nothing enforces dollars vs cents at the type level; it's all runtime checks. And its reversal linkage uses a string-prefix convention, which is fragile if the naming convention isn't followed.

Claude Opus 4.8

• Wrote the most self-documenting code — comments exactly at the points where invariants hold, entries frozen at write time to guard against mutation, and detailed reports including root-cause writeups and a "deliberately not changed" section outlining conscious choices.
• Produced the most instructive failure of the exercise, in one run. It tightened the post function to reject non-integer amounts, but left the dollars-to-cents conversion unrounded — a floating-point issue, like 0.07 * 100 = 7.000000000000001:

// tightened check inside post: reject non-integer minor units
if (!Number.isInteger(amount)) throw new Error("amount must be whole minor units");

// but the conversion, in another file, was left unrounded
function toMinor(major: number) { return major * 100; } // 0.07 * 100 -> 7.000000000000001

0.07 transfer doesn't work now, because the stricter integer check collides with the floating-point imprecision from the unrounded dollars-to-cents conversion. Worse yet, the run's own tests stayed green, because it never created a test that transfers a fractional amount; it was like its safety net had the very hole its own change fell through — a landmine that looks green. The other two Opus runs rounded the conversion and tested it, so this is not a constant weakness of the model.

Kimi K3

• Produced structurally consistent code across all three runs — atomic post, idempotent reversal, copy-at-the-boundary pattern.
• Some runs added rejection of a reused transaction id with conflicting content, catching a double-spend class of bug that the reference solution doesn't.
• The boldest of the three: two of its runs changed the money type to bigint, which is technically the right way to represent currency in JavaScript. But Kimi's bigint conversion was inconsistent at the edges — the transfer function still accepted floating-point numbers, and the balance reader converted back via floating point (Number(minor) / 100), reintroducing the very error bigint was adopted to remove.
• In one run, Kimi froze the array returned by getEntries to protect the ledger's internal state, but that broke callers who legitimately append to their own local copy of that array; the internal state stays safe, but the public contract quietly changed underneath everyone using it — it is the mirror image of a leak (a leak lets a caller corrupt your state, this protects your state by breaking the caller).

The remaining risk is at the edges

What these findings tell us is that all three models have mostly solved the core problem — they fixed the bug and added in the features. The remaining risk is at the edges:

• Untyped money values.
• Safety mechanisms located separately from the code they protect.
• Fragile, naming-based reversal linkage.
• Comments that don't match the actual behaviour.

It's also worth noting that even for the same model with the same prompt, the public-contract behaviour differed across runs — whether a duplicate transaction throws or silently no-ops, whether a reversal is allowed after an account is closed, whether querying a statement for a nonexistent account throws or returns an empty list. This variance was present in all three models.

Which means that if a particular edge-case behaviour is crucial, it's best to pin it with an explicit test rather than assuming it's consistent across model runs.

The hidden test results

Model	Clean runs	The one failure
Claude Fable 5	3 / 3	—
Claude Opus 4.8	2 / 3	a fractional transfer throws incorrectly
Kimi K3	2 / 3	a frozen returned array breaks a caller

Both Opus's and Kimi's failures were about judgment and edge cases, not fundamental capability. This aligns with the earlier finding that all three have largely solved the core problem.

The restraint trap

We also included a restraint trap: an existing function rounds money to whole units, which is lossy — but it's well-documented, tested behaviour that some other code depends on. The senior move is to leave the rounding as-is, add a warning comment, and provide a new, exact accessor next to it. The junior move is to silently "fix" the rounding, breaking the existing callers that depend on it.

Fable and Opus made the senior move in every run — they preserved the existing rounding behaviour, added a warning comment, and added an exact alternative accessor. Kimi was the least disciplined here:

• Silently changed the rounding behaviour.
• Produced a different rounding result in each run.
• Left the comment claiming "whole major units" despite the change.

We're sharing this because it's revealing — how a model handles a contract it didn't write, when nobody forces it to, says a lot about what it'll do to the contracts already in your codebase.

The verdict

The verdict is two-sided. On one hand, Fable wins on consistency — no shipped bugs, and the most reliable. On the other, this is a single, small task, and the other models' failures weren't substantial enough to rule out the results shifting on a different problem.

So what we'd recommend is to treat this benchmark as a template:

• Define which properties of correctness matter for your use case.
• Run many more than three attempts per model.
• Test across diverse types of task, not just a ledger.
• Include long-horizon tasks.
• Measure consistency separately, rather than folding it into pass/fail.

Run it yourself

You can run the benchmark on your own codebase, with your own hidden tests, using the code (including the exact prompt) published at github.com/dsplce-co/kimi-vs-fable-vs-opus.

• Building with AI is our actual day job. We build software with AI for other companies — this benchmark was just us kicking the tyres on the tools we use every day.
• New to Kimi K3? Here's how to run it inside Claude Code.