This post is my submission for DEV Education Track: Build Multi-Agent Systems with ADK.
What I Built
Most lead scoring demos share a fa...
For further actions, you may consider blocking this person and/or reporting abuse
The corroboration is only as strong as how independent the two readers actually are, and right now both are Gemini (flash and flash-lite). Same training lineage tends to fail the same way. The hedged inputs that make one reader misread "possibly renewing" are the ones most likely to fool the other identically, so both fire the same rule and agree. On exactly the inputs where you want a second opinion, correlated readers turn agreement into false comfort instead of evidence. Your A2A setup is what makes real independence reachable, a genuinely different vendor or framework behind the agent card, so the design already points the right way. It just isn't cashed in with two same-family models yet. Separate thing worth watching: score-space corroboration catches "they read the notes differently" but not "the shared rubric is generous the same way for both," since both readers run the same rules, so a rubric that over-scores shows low disagreement and a healthy-looking EXCLUDED rate while the CONFIRMED bucket goes unaudited.
Both of these are right, and the first one I will take on the chin: flash and flash-lite are nominal independence, not real independence. Same lineage fails the same way, and it fails hardest on exactly the hedged inputs where you wanted the second opinion, so agreement there is closer to an echo than a check. The A2A card exists precisely so the second reader can become a different vendor or architecture without touching the pipeline, which is the fix, but you are right that it is not cashed in yet. The part I would add: independence should not be assumed from 'two services,' it should be measured. Run both readers on a labeled ambiguous set and watch how often they agree when they should split. If two readers almost never disagree on inputs built to be disagreeable, that is not corroboration, that is one model wearing two coats, and the agreement rate tells you so.
Your second point is the sharper one, and it is more dangerous because it is invisible. Score-space corroboration compares reader against reader, which is orthogonal to rubric against reality. Both readers run the same rubric, so a rubric that over-scores produces agreement, a low EXCLUDED rate, and a CONFIRMED bucket nobody audits, and the whole thing looks healthy. Here is the uncomfortable part for me: that is my own thesis biting the project one floor up. A low EXCLUDED rate is a silent zero. It could mean the readers agree, or it could mean the rubric is too generous to ever let them disagree, and from the outside those are indistinguishable. So EXCLUDED rate is a reader-disagreement metric, not a health metric, and I should not let it read as one.
And corroboration structurally cannot fix that, because the rubric sits upstream of both readers, so no second reader audits it. That needs a different instrument: calibration against labeled outcomes, is a CONFIRMED lead actually a good lead, not another reader agreeing it cleared the same bar. Two axes, and the project has one. Agreement is evidence exactly to the degree the readers are independent and the rubric is calibrated, and you put your finger on the two places it currently buys less than it looks. Good catch on both.
Update: both of these went into the known-limitations in the repo, credited to you. Independence as "measure it, do not assume it," and the EXCLUDED-rate one as its own bullet, since a low abstention rate can mean the readers agree or the rubric is too generous to ever fire, and corroboration cannot tell those apart. Thanks for the sharpening.
The labeled-outcome calibration is the right instrument, and it has one blind spot worth building around from the start. You only ever get a conversion label for leads you acted on, which is the CONFIRMED population the rubric already blessed. The ones it EXCLUDED never get pursued, so they never get a label, so a rubric that's too strict and drops good leads leaves no trace in the outcome data. Calibration against outcomes can tell you a CONFIRMED lead was actually bad. It structurally can't tell you an EXCLUDED lead was actually good, because that arm of the experiment never runs. If you want the rubric honestly calibrated in both directions, you have to spend a little: periodically pursue a random sample of low-score or EXCLUDED leads anyway, just to buy the counterfactual labels. Otherwise the calibration quietly optimizes precision and goes blind on recall.
That is the exact blind spot, and it has a name from credit scoring: reject inference. You only observe outcomes on the population the rubric accepted, so calibrating on it measures precision on the CONFIRMED and is structurally blind to recall on the EXCLUDED. A too-strict rubric that drops good leads leaves no evidence it did, because the counterfactual arm never runs. Your fix is the only one that works, and two things make or break it: the sample has to be random within the rejected pool, not the most promising rejects, or you have just rebuilt the bias one layer in, and it has to be a permanent budgeted line, not a one-time study, because recall drifts and last quarter's estimate rots like any other stale fact.
And this is the same bug the whole thread keeps finding, one more floor down. An unlabeled EXCLUDED bucket is a silent zero: no bad outcomes from rejected leads reads as "the rubric is correctly strict" when it actually means "we never looked." The random pursuit is the only thing that makes the rejected side declare its own recall instead of resting on absence of evidence. And it gets cut first for the reason we already hit, pursuing a probably-bad lead is a certain cost now against a diffuse benefit later, so under pressure it is the first line killed and the rubric quietly optimizes precision and goes blind on recall exactly as you say. Which means the exploration sample cannot be a good intention, it has to be a pre-committed budget owned by someone, or it does not survive contact with a deadline.
The part that gets me is that "no observation" quietly becomes "negative outcome" once you train on accept-only data. Nobody writes that rule down, it just falls out of the sampling. So the model ends up most confident exactly where it has the least evidence, on the population it never got to watch repay.
Turning each rejection into a recorded event with its reason, instead of an absence in the table, is what lets you go back later and test whether the cutoff was drawing the boundary in the right place. The permanent exploration budget is that same idea in dollars: you're paying to keep the excluded set from going dark.
Agreed, and I would fold your two into one: recording the reason logs where you drew the line, but only the budget can tell you the line was wrong, because falsifying a cutoff needs outcomes from the other side of it, and the reason is generated by the same boundary you are auditing, so it can never supply them. A perfect audit trail of a cutoff you still cannot test.
The part I would add is that the budget has to be aimed. Spend it by uncertainty sampling and you fund the region the model is already unsure about, which is not the dark one. Your own line is the guide: confidence is highest where evidence is thinnest, so the dollars belong on the confident-and-rejecting set, the population it is sure about and never watched repay. Explore where the model is uncertain and you keep the lights off in exactly the room you were paying to light.
The "refuses to guess" part is the right product decision. Multi-agent systems get more useful when they can return uncertainty cleanly: missing evidence, conflicting signals, or not enough confidence to qualify the lead yet. That is better than a confident fake score.
Agreed, and I would push it one step further: those three are not one bucket. Collapsing them into a single "uncertain" throws away the most useful thing the system knows.
Missing evidence says go enrich, it knows what it lacks. Conflicting signals says escalate to a human, since more data can make it worse. Low confidence on complete, consistent evidence says defer, the lead is just borderline and no extra data fixes it. So refusing to guess is necessary but not sufficient: the abstention has to be typed, because the type is the lever, enrich, escalate, or defer. An untyped uncertainty flag is still a small silent failure, moved up from the score to the abstention.
You built it. After all those comments about abstention rates and silent zeros — here's a system that actually runs the thesis.
The line I keep coming back to: "A defensible abstention beats a fake-precise number built on a contradiction." That's the whole thread in one sentence.
And the exchange with ANP2 Network in your comments — especially you calling out your own EXCLUDED rate as a potential silent zero — that's the kind of honesty that makes the engineering worth trusting. Most people ship and stop. You shipped and kept digging.
"The production failures it kills by construction" — stealing that framing for a future story 😄
Thank you, that lands harder coming from someone who was in the thread while it was still just an argument.
Take the framing, but take the asterisk with it, because that is the part your story actually needs. "Kills by construction" is only true for one of the two failure modes. It kills the disagreement failure, two readers flip a rule and one of them is wrong, because the system just refuses to score that. It does nothing for the agreement failure, both readers wrong the same way, which is exactly the case that correlates hardest on the hedged inputs where you wanted the second opinion most. Construction buys you the contradiction, not the shared blind spot.
That is also why the EXCLUDED rate is a silent zero and not a health bar. A low rate can mean the readers agree because the input was clean, or agree because they share a bias and the rubric over-scores, and corroboration cannot separate those two, because the rubric sits upstream of both readers. Reader against reader is orthogonal to reader against truth. That one needs calibration against labeled outcomes, a different instrument.
So the honest version of the line you are stealing: it kills the failures that show up as disagreement, and it makes the failures that show up as agreement louder by contrast. That second half is the one nobody writes down. Looking forward to the story.
Appreciate the asterisk — already mapped it into the series outline. There's a story coming that turns on exactly that shared-blind-spot problem. Can't say which one yet (gotta keep some surprises 😄), but I've got the calibration-vs-corroboration distinction earmarked for the post-mortem box. Looking forward to sharing it. ⚔️
Perfect home for it. One gift for the post-mortem beat if you want it: the shared blind spot shows up as the healthiest state on the dashboard, everyone agreed, EXCLUDED rate near zero, so the failure walks in wearing the green light and nothing looks wrong until the outcomes come back. Looking forward to it.