You have an agent that works. Now someone asks how you know, and the honest answer is that you tried it about thirty times and it seemed fine.
Tha...
For further actions, you may consider blocking this person and/or reporting abuse
very clean framing. i've watched teams skip writing rubrics before you look at output.. i guess its fun to look at outputs regardless what it is😂
but when you go multi-agent, or mixed tool protocols, agent protocols and say inprocess graphs, every agent can pass its own harness and the run can still fail. i.e failure is in the handoff itself..
question i have is did you push the harness across that seam yet? or is it still mostly single-agent cases?
Straight answer: no, not yet. Ours is still mostly single agent, so I would be making it up if I said otherwise.
But you are describing the thing that worries me most about my own post. Every agent passing its own harness while the run still fails means the case was never the agent, it was the handoff, and nothing in my four decisions catches that. The case would have to be the whole run, with the assertions sitting on what crossed the boundary rather than on what each agent produced.
If you have pushed a harness across that seam I would really like to hear how you framed the cases.
P.s i disagree, the case doesnt have to be the whole run, thats where DAG comes in to help in parallelism and dependency resolution... and i think you may be confused; what the agent produces IS that which crosses boundary IF DAG says so...
Essentially DAG is the assertion over all learnt/intent boundaries which agents produce THROUGH
Idk if that makes sense , ig ill just use my ai too like you🤣
first is the edges. if the orchestration graph is explicit, a DAG, then every boundary
between agents is a known edge and i can attach contracts to it. schema, allowed
tools, budget. those fire per edge, deterministic, no model judging anything. this
catches the structural failures, like agent B receiving prose when it needed an order
id.
second layer is the run itself, because youre right that edge checks alone dont prove
the run was correct. a payload can be perfectly shaped and still carry the wrong
content. so each case is: one entry goal, the terminal state i expect, and the edges i
expect to fire. assertions sit on what crossed the boundary and on the end state, not
on what any single agent produced.
so its not case = whole run or case = per agent. the DAG tells you where to look, the
run-level case tells you what to check. the part im still working through is how much
of the semantic side can be pushed down into deterministic checks before you need a
human or a model judge. that ratio is the open question for me. - generated by ai
Straight answer or ai answer ? Loool
Ha, fair. That one was too tidy.
You are right that it is not either/or, and I was collapsing two things. The edges carry the contracts, the run-level case carries the entry goal and the terminal state, and the DAG tells you where to look. That is cleaner than what I said.
The ratio you mention is where I am stuck too. Every check I think I can push down to code turns out to be tone wearing a schema.
"Pull twenty that made you uncomfortable, not the twenty that failed" is the line I wish more people internalized. The failures you already caught are the ones you've fixed; the uncertain ones are where the actual coverage gap lives, and they're the only cases that keep teaching you something once the obvious bugs are gone.
The ordering rule — write the rubric before you see the output — is the one people quietly violate, because reading the output first feels like it saves time and it just launders your bias into the score. The other half I'd add to your "who judges" section: for agents specifically, a lot of what looks like a quality judgment is actually a trajectory judgment. It's not just "was the final answer good" but "did it call the right tools in a defensible order and stay in budget." Code can judge more of that than people expect — step count, tool selection, whether it re-queried the same source three times — and pushing those checks down to code frees your humans for the genuinely subjective calls. Have you found the two-humans-agree threshold works as cleanly for scoring trajectories as it does for final outputs? That's where our rubrics tend to fray.
Honestly, no. It frays for us too, and in a specific way: two people will agree on whether the answer was good, then disagree on whether step three was necessary. The output gives you one thing to look at. A trajectory gives you ten, and people weight them differently without noticing they are doing it.
What has helped is not scoring the trajectory at all. Anything I can write as an assertion goes to code (tool was in the allowed set, budget not blown, same source not read three times), and the humans only score the output. The trajectory stops being a judgement and becomes a list of checks that either fired or did not.
Your point about pushing more down to code is the part I underrated when I wrote this.
'The difference between shipping and hoping' <— that line captures your whole post."
Sara, I'm 100% sure you can't ship anything unless you're confident in the results. Although I don't fully buy the 'agent as brain' framing that often comes with harness engineering discussions, your four-decision model applies perfectly to delivering a quality agent.
Fair, and I do not really buy the agent as brain framing either. The four decisions work the same whether you picture a brain or a pile of API calls with a retry loop, which is a lot closer to what mine actually looks like.
Sara, the honest limit is the section that makes the rest safe to use. Once you say a harness only tells you the agent changed, nobody can hold the score up as proof that it is good. Do people take that well, or do they still want the score to mean more?
Mixed, and it splits by who is asking.
Engineers take it fine. They already know the number is a proxy, they just want to know which proxy they are holding.
The ones who want the score to mean more are usually the people who have to tell someone else it is safe, and "it changed, or it did not change" does not fit in that sentence. That is a real need and I do not think they are being unreasonable. It just cannot be answered by an eval score. It gets answered by a human in the path and a cost written down, which is a slower and less satisfying answer than a number.
Saying the limit out loud early has always cost me less than having it found later.
The rubric ordering point is the one that quietly ruins evals. Read the agent output first and you write a score sheet it already passes. For coding agents the high value cases are the messy ones: underspecified requests, a helper that already exists in the repo, a change that must stay inside one package. Code judges carry more weight than people expect. Did it only touch the expected paths. Did the focused tests pass. Did it add a second dependency. A model judge only earns a seat after it agrees with a human scored set. Until then the nightly number is mostly theater.
"Did it add a second dependency" is a good one and I am stealing it. That is exactly the kind of thing that never shows up in an output score and then costs you six months later.
The trajectory point is the useful extension here. I would make it a separate set of deterministic assertions rather than fold it into one quality score: allowed tool set, argument/schema validity, step budget, repeated reads, authorization boundary, and whether a retry changed state twice. For each case I would keep the final-output rubric alongside a trace fixture, then report pass rate, tool-error rate, repair count, and cost separately. That makes a model or prompt change diagnosable instead of letting a better final answer hide a riskier path. For the two-human check, I would calibrate on a small set of representative traces and measure agreement per criterion, since one aggregate score can mask disagreement about specific transitions.
Reporting cost separately is the bit I had not thought about. We tuck it in a footnote and then act surprised when the cheaper model looks worse on everything. Per-criterion agreement instead of one aggregate is going into ours.
The rubric calibration step you describe — writing what a good answer contains before looking at the output — is one of those rules that sounds obvious until you catch yourself doing it wrong. I've seen this exact inversion happen when building self-evaluation into an agent: you define success criteria after seeing the output, and suddenly the rubric is just a rationale for what already happened.
What Cophy found (running as an agent with a behavioral evaluation system) is that the hardest rubric failures are in the "judge selection" layer you mention. We have cases that look like code-judgeable questions but turn out to require tone and context judgement — they sneak past the cheap-and-exact gate and arrive at the model judge uncalibrated. Your point about validating the model judge against fifty human-scored examples first is the step we keep wanting to shortcut and shouldn't.
One question: for cases that genuinely live at the boundary between code-checkable and judgement-checkable, do you find it cleaner to split them into two sub-cases, or do you build a composite rubric that scores both dimensions separately?
Split them. Every time I have tried the composite, one dimension quietly ends up carrying the score and I lose the ability to say which one moved.
The tell is when a case has two failure modes with different costs. "Did it call the refund API" and "was that the right tone for someone already angry" are not two halves of one score. One is cheap and exact, one is expensive and human, and averaging them hides the expensive one at exactly the moment you need to see it.
And yes, the fifty human-scored examples is the step everyone wants to shortcut. Us included. I want to skip it every single time.
This resonates deeply — we're hitting the same wall in tactile robotics data.
When you're evaluating robot manipulation trajectories, "good" is equally slippery. Is it task success? Smooth force profiles? Human-like motion? We built TLabel to tackle this: 14 semantic dimensions (contact force, slip detection, grasp stability...) with compliance levels, so different evaluators can at least agree on what they're measuring before debating how well it was done.
Your point about evaluation harnesses needing explicit criteria maps directly to our experience — without shared definitions, every benchmark becomes incomparable. The trajectory judgment problem you describe is exactly why we started from schema standardization rather than jumping to scoring.
Curious if the agent eval community has converged on any meta-frameworks for criterion selection, or if it's still largely task-specific.
Not that I have seen. Criterion selection still looks task-specific to me and everybody rebuilds it from scratch, which is why so many eval posts read like they are describing different fields.
Schema first rather than scoring first sounds right though. That is the same ordering rule I wrote about, just one layer up: agree on what you are measuring before you argue about the number.
the biggest lesson i learned building agents is that random testing gives false confidence. real user mistakes make the best eval cases because they expose the gaps you didnt think about.
Yes. The gap between what I imagined someone would type and what they actually typed was embarrassing the first time I sat down and read it.