Updated October 29, 2026. The original article’s recommendation to abandon manual labeling was too broad. This revision explains when prompt-derived criteria help and when reviewed references remain necessary.
Your system prompt can define behavioral requirements for an agent. Turning those requirements into evaluation criteria is useful. Treating the prompt as a substitute for all ground truth is not.
The right approach depends on the question you are trying to answer.
Convert instructions into explicit criteria
Consider this instruction:
Answer from the supplied policy. Be concise. If the policy is missing, say so.
It suggests three checks: support in the supplied policy, an application-specific conciseness rubric and appropriate handling of missing information. Each check needs its own evidence and a definition of acceptable behavior.
A keyword scan can help detect a narrow prohibited string. It does not reliably establish politeness, policy compliance or whether a factual claim is correct. A model judge can evaluate a richer rubric, but its judgments also need validation.
Preserve reviewed examples
Manual annotation can be costly and inconsistent. Clear guidelines, representative cases and reviewer agreement make it more useful. A changed prompt does not automatically make every existing example obsolete; review which expectations the change actually affects.
Keep examples for important facts, boundaries and previously observed failures. For a policy agent, that might include a current return window, an unavailable policy and a question outside the product’s scope.
Reference-free checks can reduce the need for expected answers on suitable tasks. They still need the fields specified by the evaluator. For reference-based accuracy, retain an appropriate expected output or source.
Validate the evaluation before scaling it
Run a small reviewed set first. Inspect failures and a sample of passes. Identify false positives, missed errors and ambiguous criteria. Preserve the rubric and configuration with the results so later comparisons are interpretable.
When a prompt changes, rerun relevant cases and inspect regressions. Judge a candidate against the application’s requirements rather than assuming that a higher aggregate score establishes a better release.
What the product workflow supports
I’m Shashank Agarwal, founder and CEO of Noveum AI and creator of API.market. Noveum connects instrumented traces, prepared evaluation items, selected checks, investigation and candidate testing.
As of October 29, 2026, the published scorer library lists 112 evaluators across 15 categories. Teams select compatible checks; the count does not mean every evaluator runs on every trace. Outputs may be numeric, boolean or labels, so preserve native results and saved pass decisions.
NovaPilot findings and recommendations help organize an investigation. They are candidates for engineering review, not guaranteed root causes or production fixes. Test a reviewed candidate before deciding whether to release it.
The eval-driven development guide expands this workflow. I share my engineering and founder work on LinkedIn.
This refresh was drafted with AI assistance.

Top comments (0)