Public coding-agent benchmarks answer an important question: which system performs best across a shared task set?
Engineering teams usually need a different answer:
Which agent, model, prompt, and tool configuration can we trust on our own codebase?
RepoTrials is an Apache-2.0, local-first tool that turns fixes already present in Git history into repeatable regression evaluations for coding agents.
The core idea
A repository already contains examples of real engineering work:
- code existed in a broken state;
- a test exposed the behavior;
- a human fix made that test pass; and
- the rest of the test suite guarded against regressions.
RepoTrials reconstructs that transition and packages it as a sealed task:
real fix commit
→ historical base revision
→ hidden test patch
→ equal agent trials
→ separate verifier
→ comparable evidence
The agent receives the historical workspace without the future fix, hidden tests, or later Git objects. Grading is behavioral: the relevant failing tests must pass, while protected regression tests must remain green.
Why this is useful
A public leaderboard can be directionally helpful, but it cannot model every private architecture, convention, dependency, or failure mode.
RepoTrials lets a team build a benchmark from the code it already owns. The engine can remain public while the task corpus, hidden tests, reference patches, and raw run artifacts remain local under .repotrials/.
That makes the result useful for questions such as:
- Did a new model actually improve on our repository?
- Is a cheaper configuration good enough for our routine fixes?
- Does a new prompt help consistently, or did it only win one lucky run?
- Can we reproduce the comparison later from machine-readable artifacts?
Try the complete demo
The v0.1.0 demo creates a two-commit repository, mines one historical task, validates it, runs a no-op agent and a fixing agent, compares the results, writes an HTML report, and exports the task for Harbor.
git clone --depth 1 --branch v0.1.0 https://github.com/PozziTiv4ik/Repo-Trials.git
cd Repo-Trials
python -m pip install -e ".[dev]"
python scripts/demo.py
No model API key is required for the demo.
Expected comparison:
noop-agent 0/1 resolved
fix-agent 1/1 resolved
delta +100 percentage points
What happens under the hood
For a base revision B, hidden test patch T, and historical human solution S, validation checks:
BASE B + original tests → pass
RED B + T → relevant failure
GOLD B + S + T → pass
NOOP B + T → not resolved
Accepted tasks can be run against any command-based coding agent. Results are stored as JSON and can be rendered as HTML reports. Tasks can also be exported to a Harbor-compatible layout with a separate verifier.
Honest v0.1 boundaries
This is the first public release, not a security or procurement certification.
The current version is strongest on Python repositories with reproducible historical tests. Human task review still matters. Validation and agent commands execute arbitrary repository code, so untrusted workloads belong in an isolated environment. The project also documents unsupported patch shapes and historical checkout limitations rather than hiding them.
Those constraints are exactly why the project is open: the evaluation contract, schemas, threat model, and artifacts can be inspected and challenged.
Feedback wanted
If you evaluate coding agents on a real repository, try the demo and tell us where the workflow breaks for you.
Disclosure: this launch post was drafted with AI assistance and checked against the v0.1.0 project documentation.
Top comments (0)