I have been building Chaser Agent and other AI systems across the Chase ecosystem. As those systems became more capable, I needed a better way to inspect what an agent actually did—not just look at its final answer.
So I built Agent Review Studio.
It is an open-source, local-first system for evaluating and refining AI agents and agent harnesses.
Why I built it
An agent run can produce a final answer, source files, extracted claims, proposed actions, memory candidates and a complete execution trace. Looking only at the final response hides most of the engineering evidence.
I wanted one review flow that could answer practical questions:
- Which claim came from which source?
- Was enough surrounding context preserved?
- Are proposed actions supported by the evidence?
- Did the agent handle uncertainty properly?
- Is a candidate harness actually better than its baseline?
What the Studio does
Agent Review Studio lets an engineer:
- Create a workspace for any agent or harness.
- Run or import a versioned evaluation.
- Inspect claims beside their source evidence and execution traces.
- Label failures and write the correction.
- Score the complete run across five quality dimensions.
- Preserve immutable review history.
- Compare baseline and candidate runs.
- Export reviewed improvement data.
The important part is what happens next. A reviewed failure can become a golden evaluation case or regression test. We can update a prompt, retrieval system, tool policy, memory rule or workflow, run the same task again and compare the result with the original baseline.
How we use it in the Chase ecosystem
The launch demo uses a real Chaser Agent research run built around a public Cloudflare source. Agent Review Studio opens the run's files, places extracted claims beside the retained evidence and lets the operator classify what was correct, weak, missing or irrelevant.
The example labels shown in the video are an unsaved demonstration. The +17 result is an automated comparison, and human review is still pending. I kept those boundaries visible because an automated score should not be mistaken for human approval.
Chaser Agent is only one workspace. Fresh installations of the Studio contain no preloaded agent. Other engineers can name the system they are building, describe its objective and import their own datasets and runs.
Evaluation is not automatic fine-tuning
This is agent evaluation, human labelling, evidence curation and harness refinement.
The Studio does not automatically change model weights. It creates the evaluation infrastructure and trusted improvement data needed to refine prompts, tools, retrieval, memory and orchestration. Reviewed examples can later become candidates for a separate, governed model fine-tuning pipeline.
Try it
Version 1.5.0 is open source under Apache-2.0.
I would value feedback from other people building agents: what evidence do you require before deciding that a new harness version is genuinely better?
Disclosure: I used AI assistance while preparing and editing this launch article. I reviewed the claims, product boundaries and final publication myself.
Top comments (0)