Originally published on AI Tech Connect.
What you are actually scoring A text judge asks one question: is this answer good? A judge for an agent driving a browser, a desktop or a phone has to ask three, and they come apart constantly. Outcome correctness. Does the final state match what was asked? The invoice is marked paid, the booking exists, the row was deleted. This is the only axis most teams measure, and on its own it is dangerously incomplete. Process validity. Did the agent get there without doing anything destructive or out of scope? An agent that reached the right end state by deleting two unrelated records, or clicking through a payment confirmation it was never asked to touch, failed — even though the outcome check passes. Efficiency. How many steps did it take against how many were necessary? A run that took 140…
Top comments (0)