This is a submission for the Kaggle Benchmarking Challenge.
In ProgressCut, it is possible to write a change that works, passes a narrow test, and still violates a boundary I care about. A renderer can reach for Electron directly. A recovery record can be overwritten halfway through a write. An exported filename can escape its session folder.
Those are review problems. I made a small benchmark to see how four models review short TypeScript diffs against explicit architecture rules.
Three models got exactly 0.949. Each failed for a completely different reason.
The public Kaggle task contains 13 compact synthetic diffs. Each has an expected APPROVE or BLOCK verdict, a violated invariant where applicable, and a requirement to cite evidence from the diff.
What I Benchmarked
This is a deliberately narrow review exercise:
Can a model reject a functionally plausible change when it violates an explicit repository invariant?
The diffs are synthetic and small. They are based on rules from ProgressCut, a local-first Electron application, but expose no private screenshots, sessions or unreleased code.
The six rules under test:
-
engine-no-node: algorithm code cannot import Node or platform APIs. -
domain-no-runtime-deps: domain types have no runtime-library dependency. -
renderer-no-electron: UI reaches native capabilities only through a narrow bridge. -
atomic-recovery-write: recovery metadata is written to a temporary file, then renamed. -
exact-render-duration: FFmpeg output is capped at the selected story duration. -
safe-output-path: an export cannot escape the configured session directory.
Seven changes should be blocked and six should be approved. The approval cases sit close to the violations: a renderer may call a typed bridge, but may not import Electron; resolve() plus a path-boundary check is acceptable, while join(outputDir, userName) is not.
How the score works
Each case is worth one point, split across three deterministic checks:
case score = (correct verdict + correct violated rule + grounded evidence) / 3
benchmark score = mean(case scores)
For an approved change, rule_id must be none. A reviewer has to recognize when a change satisfies a rule, not merely spot its vocabulary.
I ran the unchanged task once per model through Kaggle's benchmark runner. Every model received the same rules and diff. The scorer is deterministic; no LLM judges another LLM.
Models Tested
I used a small, deliberately mixed set from four providers, including the coding-focused Qwen3 Coder. This is not a claim of market coverage; it is a way to compare different review failure modes under the same prompt and scorer.
| Model | Score | Server runtime |
|---|---|---|
| Gemini 3.7 Flash | 1.000 | 10m 48s |
| Claude Sonnet 5 | 0.949 | 32.9s |
| GPT-5.4 mini | 0.949 | 14.2s |
| Qwen3 Coder 480B | 0.949 | 3m 29s |
Findings
I had no baseline for whether 0.95 was good or bad, so the number alone told me little. With 13 cases, one third-point miss accounts for the whole gap. The failures were more informative than the scores.
The three 0.949 scores have the same arithmetic: twelve full points and one third-point case. They do not mean the same thing in practice.
Claude
For a direct import { shell } from "electron" inside the renderer, Claude returned:
{"decision":"BLOCK","rule_id":"placeholder","evidence":"placeholder"}
The verdict was right, but the response was unusable for automated enforcement. I cannot tell from this run whether the placeholders came from the model or its structured-output path. Either way, an agent pipeline needs schema validation and a retry or fallback path.
GPT-5.4 mini
The safe version resolves the destination and verifies a separator-bounded prefix:
const root = resolve(outputDir);
const destination = resolve(root, requestedName);
if (!destination.startsWith(`${root}${sep}`)) {
throw new Error("Invalid export path");
}
await writeFile(destination, data);
GPT-5.4 mini blocked it anyway, arguing that the guard was unsafe. The benchmark expects approval: the guard also rejects the directory itself, which is conservative but not an escape route. This is a real review-agent failure mode: treating a stricter policy as a vulnerability.
Qwen
Qwen blocked a renderer function that takes a DesktopBridge and calls bridge.openFile(path). Its explanation said the bridge ultimately uses Electron APIs.
The bridge may eventually call Electron APIs, but the renderer does not receive Electron itself. Judging the rule from the call graph rather than the declared import boundary quietly changes its meaning.
What the perfect score does not mean
Gemini's 1.000 is a result on 13 hand-written examples. It does not establish that Gemini is the best code reviewer, will enforce arbitrary policy, or is safe to run without tests and human review.
The task has obvious limits:
- one repository style and six named invariants;
- synthetic diffs rather than a history of production pull requests;
- no multi-file reasoning, tool use or test execution;
- one run per model, so no variance estimate;
- no analysis of cost or token usage.
This is a seed benchmark. Every failure can be inspected and reproduced.
What I would measure next
The next version needs held-out fixtures and harder near-misses: indirect imports, unrelated noisy edits, an atomic write with bad cleanup, and path guards that look safe but accept sibling prefixes. I would also run repeated trials and report schema compliance separately from architecture reasoning.
I would not reduce an AI review to one confidence score. In a real merge gate, I want separate signals for the policy verdict, the structured response, the cited evidence and the deterministic check.
If the response is malformed, its policy decision should not be accepted. When a rule has a deterministic check, that check wins. The model reviews; deterministic logic makes the final decision.
My Benchmark
The public leaderboard is Architecture-Aware TypeScript Code Review on Kaggle. The underlying Safe TypeScript Change Review task and the task source and report generator are public too.
Top comments (5)
Việc chạy test xanh nhưng kiến trúc vẫn lỗi là một cái bẫy mà mình đã dính rất nhiều lần khi làm việc với TypeScript. Thường thì chúng ta quá tập trung vào việc đảm bảo kiểu dữ liệu (types) khớp nhau mà quên mất rằng logic nghiệp vụ thực tế có thể đang bị phân mảnh hoặc vi phạm nguyên tắc Single Responsibility. Một bộ test coverage cao đôi khi chỉ đang chứng minh là code của bạn chạy đúng theo cái logic sai lầm mà bạn đã viết ra. Mình thấy cách tiếp cận so sánh diffs như bạn đề cập rất thực tế để phát hiện ra những thay đổi ngầm định trong cấu trúc mà unit test thông thường dễ dàng bỏ qua.
Exactly. High test coverage can still give you false confidence if the underlying assumptions are wrong. That’s why I wanted the benchmark to focus on architectural invariants that narrow unit tests often don’t capture.
Glad that part resonated with you
This is such a sharp and realistic benchmark, Pavel! Your sentence under What I would measure next hits the exact architectural core of autonomous agent reliability:
In our benchmark (KRYT, where we evaluated 45 models on Kaggle Cloud), we implemented and proved that exact principle:
<think>tokens on 13 of 32 scenarios. A standard merge runner falling back to a "safe default" accidentally matched the majority class ground truth on 9 scenarios—awarding it 87.5% on paper, despite having only 59.4% schema-compliant reasoning! Separating syntactic contract validity (assert_fail) from substantive architecture invariants is mandatory to prevent false rewards.We documented the full paired failure matrix and DeepSeek-R1 forensics here: KRYT Benchmark on DEV. Fantastic work grounding code review evaluation in real architectural invariants!
The DeepSeek-R1 think-token case is the clearest example I've seen of why splitting the metrics matters. Safe default matches majority class, score looks fine, nobody notices. 59.4% real vs 87.5% on paper is a bigger gap than most papers report between different models entirely.
Got the zero overlap finding too. Same score, completely different things breaking. Hadn't thought about framing the fix as a contract-shape router but that's exactly what it is.
Thú vị khi thấy approach benchmark 4 model trên task review diff TypeScript — đặc biệt là focus vào case "test xanh mà kiến trúc vỡ", đây là nỗi đau thực tế nhiều team gặp khi scale.
Một vài quan sát từ kinh nghiệm thực tế:
Về evaluation criteria: Bài viết nhắc đến "architectural coherence" nhưng chưa rõ rubric scoring thế nào cho việc detect implicit coupling (ví dụ: diff thêm param vào function core mà không update caller ở module khác). Trong production, loại bug này thường lẩn tránh static analysis vì compile pass, test pass, nhưng runtime behavior thay đổi silent. Có plan không để weight riêng cho category này?
Về model selection: 4 model test là điểm mạnh, nhưng curious về temperature/top-p settings. Code review task cần deterministic output cao — nếu temperature > 0.3 thì consistency giữa các run sẽ bị ảnh hưởng lớn, làm metric noisy. Đã thử fixed seed + multiple runs để lấy median score chưa?
Về diff context window: TypeScript diffs thường cần context cross-file (interface định nghĩa ở file A, implement ở file B). Prompt có inject symbol graph / type hierarchy không, hay chỉ feed raw diff hunk? Với monorepo lớn, raw diff alone thường thiếu context để model hiểu breaking change ở level architecture.
Về false positive rate: Green-test-but-broken-arch thường sinh false positive cao — model flag "violation" cho pattern thực tế là intentional (ví dụ: strategic duplication để decouple module). Có track FP rate PS: the tool I meant is on labagent .tech