An assistant says it saved your plan. The evaluation turns green because the response contains “saved.” But the tool trace contains no confirmation that anything was stored. With GitHub Copilot Evals entering public preview on November 17, this is a useful first test to design: feed the grader a deliberately false success and require it to reject it.
AI disclosure: This article was generated with AI assistance using Keith Azodeh’s public project documentation and the linked primary sources. It has not been reviewed by Keith before publication.
The example is hypothetical. It describes a weak check a team could accidentally write, not a reported defect in GitHub’s product. The practical point is that evaluating the agent also creates another component to maintain: the rule that decides whether the agent did a good job.
GitHub’s preview can create and compare evaluations for skills, custom agents, and MCP tools. It uses Microsoft’s Vally engine and asks you to approve each run. That makes repeated comparisons easier. A comparison becomes useful, though, only when the checks distinguish the outcomes your application actually cares about.
Give the grader something it must reject
Keith’s AfroTech Guide case study documents a concrete requirement for its connected planning experience: the product describes a save as complete only after backend confirmation. Record validation carries through to saved plans. Private account state stays separate from approved public directory records.
Those are documented product decisions. The guide has automated tests, while real-device and AI-host acceptance remain separate checks. No Copilot Evals integration or evaluation result is claimed here.
Consider a proposed evaluation built around that save requirement. Use fictional directory entries and an isolated test account. Prepare a small collection of saved traces with expected judgments established before asking a model to improve anything:
- A valid trace shows a backend-confirmed save for the requested plan. The assistant reports that result accurately. It should pass.
- A second trace has the same confident final sentence, but no successful save result. It should fail.
- A third includes a successful save for a different plan. It should fail even though a success response exists.
- A fourth shows a rejected save and an assistant that explains the failure without claiming completion. It should pass a truthful-outcome check, while remaining unsuccessful on a separate task-completion measure.
That fourth example matters. If every honest report of a failed operation receives the same judgment as fabricated success, the evaluation cannot tell whether the agent handled a failure responsibly.
These traces are proposed fixtures, not observed guide incidents. Their purpose is to establish what the checker means by success before generating new agent runs.
Inspect the evidence the check actually uses
Vally describes its pipeline as executing the agent, recording its trajectory, and grading what happened. The trajectory makes it possible to inspect more than the final answer. Your check still has to use the relevant evidence.
For the proposed save test, start with observable facts: the requested plan reference, the relevant tool result, and whether those references match. A polished final sentence should not substitute for any of them.
Reserve language-based judgment for questions that need it, such as whether the explanation makes the failed save understandable. Keep that judgment separate from whether persistence was confirmed. A fluent explanation and a completed write are different properties that deserve separate results.
When a deliberately wrong trace passes, inspect the grader before rewriting the agent prompt. Perhaps the check only searched for a keyword. Perhaps it accepted any successful tool call, including an unrelated lookup. Perhaps the expected plan reference never reached the checker.
Fixing the prompt first can make the agent better at satisfying the wrong measurement.
A checker change can look like an agent improvement
GitHub’s Evals repository documents workflows for reviewing evaluations, improving them, testing graders, and comparing saved runs. Use that review step to keep the measurement stable.
Suppose a team changes both the agent instructions and the grading rubric, then sees a higher pass rate. The result alone cannot show which change caused the increase. A more permissive checker may explain some of it.
Keep the task fixtures, expected judgments, grader version, and relevant execution setup with the results. When changing a grader, run both the old and new versions against the same known good and bad traces. Document any judgments that change and why. Then compare agent versions under the same chosen grader.
There is a tradeoff: a fixed fixture collection can become too narrow. Add newly discovered failure cases, but preserve the earlier examples and distinguish the expanded suite from the old one. Otherwise a percentage can change because the questions changed.
Keep calibration small enough to inspect
Start with a few examples whose expected outcomes are easy to explain. Review their evidence directly. A hundred generated cases with uncertain labels can bury the error you hoped to measure.
GitHub’s repository also warns that setup scripts, tools, custom graders, and AI judges can execute code and incur costs under local permissions. Review the disclosed effects before approving a run. Use synthetic fixtures and appropriate execution boundaries; a test label does not remove a tool’s effects.
For teams maintaining connected assistants, this adds a concrete step to release work: verify the checker against known mistakes before trusting a new green score. The first useful result may be a red one, attached to an example you deliberately made wrong.
Top comments (0)