The refusal result is the most useful signal here. For the eval, I'd separate four outcomes instead of a single accuracy number: correct tool and args, correct refusal, correct clarification, and unsafe/plausible action. Their costs are very different, so a server that turns ambiguity into a clarifying question should not be scored like one that silently chooses a near-twin tool.
To keep synthetic tasks from flattering the schemas that generated them, use held-out authoring: derive the intent set from real support/integration failures, have a separate process paraphrase it without seeing tool names, and freeze a test split before changing descriptions. Report confidence intervals and slices for negative tasks, overlapping names, optional-vs-required fields, enums, long catalogs, and multiple valid tools.
I'd also pin server contract, model, client prompt/tool serializer, temperature, and catalog selection policy in every result. A score is otherwise hard to compare across releases. The production correlation I would watch is repair turns per successful operation plus false-positive tool calls per 1,000 out-of-scope requests; those capture both usability cost and the dangerous “do something” failure mode.
One direct ask: held-out authoring is the piece I'd most value a second
brain on. If you're at all interested in sketching the design (in the issue
thread, zero commitment), the door is open.
This is the most useful comment I've gotten anywhere on this project — thank you.
You're right on all three counts. The current eval collapses "clarifying
question" into refusal, and their costs are obviously different; a four-outcome
taxonomy (correct call / correct refusal / correct clarification / unsafe
plausible action) is strictly better and I'm stealing it. The self-flattering
risk is real too — synthesis currently sees tool names, so held-out authoring
with a paraphrase step that never sees the catalog is the right fix. And result
pinning (model, temperature, serializer, catalog policy) is only partial today:
model + temp are recorded, the rest isn't.
"Repair turns per successful operation + false-positive calls per 1k
out-of-scope requests" is a better production north star than anything I had
written down.
Would you mind if I turn this into tracked issues on the repo with credit? If
you'd rather file them yourself I'll tag them as roadmap directly.
The refusal result is the most useful signal here. For the eval, I'd separate four outcomes instead of a single accuracy number: correct tool and args, correct refusal, correct clarification, and unsafe/plausible action. Their costs are very different, so a server that turns ambiguity into a clarifying question should not be scored like one that silently chooses a near-twin tool.
To keep synthetic tasks from flattering the schemas that generated them, use held-out authoring: derive the intent set from real support/integration failures, have a separate process paraphrase it without seeing tool names, and freeze a test split before changing descriptions. Report confidence intervals and slices for negative tasks, overlapping names, optional-vs-required fields, enums, long catalogs, and multiple valid tools.
I'd also pin server contract, model, client prompt/tool serializer, temperature, and catalog selection policy in every result. A score is otherwise hard to compare across releases. The production correlation I would watch is repair turns per successful operation plus false-positive tool calls per 1,000 out-of-scope requests; those capture both usability cost and the dangerous “do something” failure mode.
Follow-up: your points are now tracked issues on the repo, credited to this
comment — github.com/TengByte/mcpgrade/issues
One direct ask: held-out authoring is the piece I'd most value a second
brain on. If you're at all interested in sketching the design (in the issue
thread, zero commitment), the door is open.
This is the most useful comment I've gotten anywhere on this project — thank you.
You're right on all three counts. The current eval collapses "clarifying
question" into refusal, and their costs are obviously different; a four-outcome
taxonomy (correct call / correct refusal / correct clarification / unsafe
plausible action) is strictly better and I'm stealing it. The self-flattering
risk is real too — synthesis currently sees tool names, so held-out authoring
with a paraphrase step that never sees the catalog is the right fix. And result
pinning (model, temperature, serializer, catalog policy) is only partial today:
model + temp are recorded, the rest isn't.
"Repair turns per successful operation + false-positive calls per 1k
out-of-scope requests" is a better production north star than anything I had
written down.
Would you mind if I turn this into tracked issues on the repo with credit? If
you'd rather file them yourself I'll tag them as roadmap directly.
tracked, shipped, thanks