CodeVetter's public v1 benchmark is a reproducible recognition benchmark for agent-written bugs. It publishes 27 synthetic cases, 29 labeled findings, reviewer outputs, scoring rules, downloads, and explicit limitations.
That answers a narrow question: does the tested review pipeline recognize these known issues under this fixture set? It does not prove that CodeVetter verifies arbitrary production pull requests.
Read the result in layers
I separate the evidence on the benchmark page into three layers:
- Published now: the 27-case recognition benchmark and its exact results.
- Implemented qualification infrastructure: task packages, executable checks, receipts, and evaluation contracts described by the repository.
- Not yet proven publicly: repeated provider runs on a broad realistic corpus with reliability, cost, and latency comparisons.
Blurring those layers would turn a useful result into a marketing claim it cannot support.
Choose the benchmark for the question
Recognition fixtures help isolate issue-type coverage and scoring behavior.
Repository-task benchmarks such as SWE-bench ask whether a generated patch resolves a real issue inside a reproducible environment.
A product-specific agent benchmark should go further by retaining the requested task, exact patch, environment, checks, evidence, and uncertainty. No single score answers all three questions.
Inspect the artifacts
A benchmark should expose its dataset, per-case pages, reviewer outputs, scoring method, and reproduction path. It should also put the limitations before the call to action.
For this benchmark, those limits include synthetic cases, narrow languages and categories, no timing or cost dimension, and precision as a material constraint.
The next credible proof
The next step is not a larger slogan. It is qualified, repeated agent-task evidence with immutable receipt linkage and a clear failure taxonomy.
You can inspect the cases, downloads, scorer, and current limitations at https://codevetter.com/benchmark.
Top comments (0)