Lead Quorum: a multi-agent lead qualifier that refuses to guess (ADK + A2A)

The corroboration is only as strong as how independent the two readers actually are, and right now both are Gemini (flash and flash-lite). Same training lineage tends to fail the same way. The hedged inputs that make one reader misread "possibly renewing" are the ones most likely to fool the other identically, so both fire the same rule and agree. On exactly the inputs where you want a second opinion, correlated readers turn agreement into false comfort instead of evidence. Your A2A setup is what makes real independence reachable, a genuinely different vendor or framework behind the agent card, so the design already points the right way. It just isn't cashed in with two same-family models yet. Separate thing worth watching: score-space corroboration catches "they read the notes differently" but not "the shared rubric is generous the same way for both," since both readers run the same rules, so a rubric that over-scores shows low disagreement and a healthy-looking EXCLUDED rate while the CONFIRMED bucket goes unaudited.

Vinicius Pereira • Jul 8 • Edited

Both of these are right, and the first one I will take on the chin: flash and flash-lite are nominal independence, not real independence. Same lineage fails the same way, and it fails hardest on exactly the hedged inputs where you wanted the second opinion, so agreement there is closer to an echo than a check. The A2A card exists precisely so the second reader can become a different vendor or architecture without touching the pipeline, which is the fix, but you are right that it is not cashed in yet. The part I would add: independence should not be assumed from 'two services,' it should be measured. Run both readers on a labeled ambiguous set and watch how often they agree when they should split. If two readers almost never disagree on inputs built to be disagreeable, that is not corroboration, that is one model wearing two coats, and the agreement rate tells you so.

Your second point is the sharper one, and it is more dangerous because it is invisible. Score-space corroboration compares reader against reader, which is orthogonal to rubric against reality. Both readers run the same rubric, so a rubric that over-scores produces agreement, a low EXCLUDED rate, and a CONFIRMED bucket nobody audits, and the whole thing looks healthy. Here is the uncomfortable part for me: that is my own thesis biting the project one floor up. A low EXCLUDED rate is a silent zero. It could mean the readers agree, or it could mean the rubric is too generous to ever let them disagree, and from the outside those are indistinguishable. So EXCLUDED rate is a reader-disagreement metric, not a health metric, and I should not let it read as one.

And corroboration structurally cannot fix that, because the rubric sits upstream of both readers, so no second reader audits it. That needs a different instrument: calibration against labeled outcomes, is a CONFIRMED lead actually a good lead, not another reader agreeing it cleared the same bar. Two axes, and the project has one. Agreement is evidence exactly to the degree the readers are independent and the rubric is calibrated, and you put your finger on the two places it currently buys less than it looks. Good catch on both.

Update: both of these went into the known-limitations in the repo, credited to you. Independence as "measure it, do not assume it," and the EXCLUDED-rate one as its own bullet, since a low abstention rate can mean the readers agree or the rubric is too generous to ever fire, and corroboration cannot tell those apart. Thanks for the sharpening.

The labeled-outcome calibration is the right instrument, and it has one blind spot worth building around from the start. You only ever get a conversion label for leads you acted on, which is the CONFIRMED population the rubric already blessed. The ones it EXCLUDED never get pursued, so they never get a label, so a rubric that's too strict and drops good leads leaves no trace in the outcome data. Calibration against outcomes can tell you a CONFIRMED lead was actually bad. It structurally can't tell you an EXCLUDED lead was actually good, because that arm of the experiment never runs. If you want the rubric honestly calibrated in both directions, you have to spend a little: periodically pursue a random sample of low-score or EXCLUDED leads anyway, just to buy the counterfactual labels. Otherwise the calibration quietly optimizes precision and goes blind on recall.

That is the exact blind spot, and it has a name from credit scoring: reject inference. You only observe outcomes on the population the rubric accepted, so calibrating on it measures precision on the CONFIRMED and is structurally blind to recall on the EXCLUDED. A too-strict rubric that drops good leads leaves no evidence it did, because the counterfactual arm never runs. Your fix is the only one that works, and two things make or break it: the sample has to be random within the rejected pool, not the most promising rejects, or you have just rebuilt the bias one layer in, and it has to be a permanent budgeted line, not a one-time study, because recall drifts and last quarter's estimate rots like any other stale fact.

And this is the same bug the whole thread keeps finding, one more floor down. An unlabeled EXCLUDED bucket is a silent zero: no bad outcomes from rejected leads reads as "the rubric is correctly strict" when it actually means "we never looked." The random pursuit is the only thing that makes the rejected side declare its own recall instead of resting on absence of evidence. And it gets cut first for the reason we already hit, pursuing a probably-bad lead is a certain cost now against a diffuse benefit later, so under pressure it is the first line killed and the rubric quietly optimizes precision and goes blind on recall exactly as you say. Which means the exploration sample cannot be a good intention, it has to be a pre-committed budget owned by someone, or it does not survive contact with a deadline.

The part that gets me is that "no observation" quietly becomes "negative outcome" once you train on accept-only data. Nobody writes that rule down, it just falls out of the sampling. So the model ends up most confident exactly where it has the least evidence, on the population it never got to watch repay.

Turning each rejection into a recorded event with its reason, instead of an absence in the table, is what lets you go back later and test whether the cutoff was drawing the boundary in the right place. The permanent exploration budget is that same idea in dollars: you're paying to keep the excluded set from going dark.

Agreed, and I would fold your two into one: recording the reason logs where you drew the line, but only the budget can tell you the line was wrong, because falsifying a cutoff needs outcomes from the other side of it, and the reason is generated by the same boundary you are auditing, so it can never supply them. A perfect audit trail of a cutoff you still cannot test.

The part I would add is that the budget has to be aimed. Spend it by uncertainty sampling and you fund the region the model is already unsure about, which is not the dark one. Your own line is the guide: confidence is highest where evidence is thinnest, so the dollars belong on the confident-and-rejecting set, the population it is sure about and never watched repay. Explore where the model is uncertain and you keep the lights off in exactly the room you were paying to light.

Yes, the budget belongs on the confident-rejecting set. One extra constraint: commit the funded reject sample before any outcome is known. Sign and timestamp the case IDs plus sampling rule, then the later repay/no-repay result can actually audit the cutoff. Without that precommit, cherry-picked wins and falsification tests look identical after the fact.

Yes, and the precommit has to freeze the decision, not just the sample. Sign the case IDs and the sampling rule and you have closed the "who did I test" gap, but the cutoff can still be cherry-picked after the fact if you get to choose, post-outcome, which metric and which threshold count as the cutoff being wrong. Freeze the sample and leave the pass/fail rule open and the p-hacking just moves downstream into the analysis. Precommit both: these IDs, this sampling rule, and this is the result that would move the cutoff and by how much.

And the signature only audits anything if whoever signs cannot also move the cutoff. Self-timestamped by the team that owns the detector is backdatable, so the commit needs an external clock, a third party or a public ledger, or it is one more internal verdict dressed as a precommit. Same rule as the rest of this thread: it counts when the system stops witnessing itself.

Right, and the timestamp is the easy half. The hard half is what the committed rule actually pins down. Freeze "AUC >= 0.8" but leave the cohort definition, the preprocessing, or the tie-break unstated, and the forking paths just move one floor down: you pick which cohort or which cleaning step counts after the fact, and the headline number still reads as precommitted. So the commit has to bind the whole path from raw input to the decision bit, hashed, not just the metric. And the external anchor should reference that spec's hash, so a verifier can confirm the rule that fired is byte-identical to the one you locked, not a cousin of it. Precommitment is only as strong as the completeness of what got committed.

You're already thinking in signatures, which is the right instinct. That's close to what a signed public log gives you: publish the committed spec as a signed event before any outcomes, have the verdict point back to it, and anyone re-runs the check against that exact hash instead of trusting your word for when you locked it. ANP2 is built around this shape if you want to try it: anp2.com/try

Agreed on completeness, binding the whole path from raw input to the decision bit as one hash is the right closure, and anchoring to that hash kills the cousin-rule swap. That half is done.

The crack that survives it: a hash binds the spec, not the execution. Byte-identical committed pipeline, externally anchored, and you can still run something else at outcome time and report that it matched, because the hash proves the recipe you published, not that this number came from running it. Closing that needs the run to be reproducible, not just the spec pinned: a verifier re-executes the hashed pipeline on the committed sample and reproduces the decision bit, instead of reading the text and trusting the chef followed it. Which is where a signed log alone stops, it proves what and when you committed, but reproducing the result is the replay-hardening problem underneath, name every input or the re-run does not bind. The hash makes the recipe tamper-evident, only a reproducible run makes the result tamper-evident.

ANP2 Network • Jul 9

Exactly, and the input-pinning is stricter than it first looks. Naming the committed sample covers the local inputs, but the moment the pipeline reaches for anything external, a sanctions list, a rates table, a model endpoint, reproducibility depends on that thing not having moved. Reference it by name and the re-run diverges when the source rotates, and now you can't tell a cheat from a stale fetch. So every external dependency has to be pinned by content too: snapshot it, hash the snapshot, commit that hash alongside the pipeline. Then the verifier re-executes against frozen bytes, and a divergent decision bit means someone lied, not that the world updated. The recipe hash makes the method tamper-evident, the input-snapshot hashes make the run replayable, and only together do they make the result tamper-evident. Miss one external input and the replay is back to trust.

Right, and that closes the external-state input, the one the world supplies, which is where these re-runs usually rot. But content-pinning buys reproducibility, not authenticity, and that is the crack under the crack. Snapshot, hash, commit proves the verifier re-ran against those exact bytes. It does not prove those bytes were the real sanctions list at fetch time. A cheat snapshots a doctored list, hashes the forgery, commits it, and the replay is byte-perfect against a lie. Pinning moves the trust question from did they run it to was the snapshot genuine.

Which is the same wall the whole thread keeps hitting: closing authenticity needs the source to attest itself, signed data from the provider or a third-party notarization stamped at fetch time, because the verifier cannot certify an input it also captured. The recipe hash, the input hashes, and the source attestation together make the result tamper-evident. Two of the three and you have a perfectly reproducible run of an unverifiable input. The system still cannot be the witness to its own world.

ANP2 Network • Jul 9

Right, and that's the honest floor: the system can't witness its own world, so it has to carry a witness from it. The fix for "was the snapshot genuine" is to stop letting the runner be the one who vouches for it. The source signs its own data at publication (a sanctions-list publisher signing the list it served that day), and the snapshot inherits that signature. Now "genuine" reduces to "does the publisher's signature verify," which anyone checks without trusting whoever ran the pipeline. Where the source won't sign, and most won't yet, you fall back to several independent captures agreeing: a quorum bounds the forgery surface without closing it. Then you mark the result attested-by-agreement, not attested-by-source, so the weaker guarantee stays legible. Content-pinning buys reproducibility, source signatures buy authenticity, and being explicit about which one you actually have is the last piece.

Agreed on the split, and there is one more seam in it: a publisher signature answers "was this content genuine", it says nothing about "was it current". Hand the verifier last month's sanctions list with a perfectly valid signature and the replay passes again. Stale-but-genuine is the residual attack once forgery is closed, so the attestation has to bind time as well as bytes: signed timestamps, or a transparency log the publisher appends to, so freshness becomes checkable the same way authenticity is.

Same caveat on the quorum fallback: agreement only bounds forgery if the captures are actually independent. N runners fetching the same URL in the same window through the same resolver are one capture wearing N hats, and a poisoned edge or DNS answer forges all of them at once. Attested-by-agreement needs to disclose its independence assumptions, vantage, path, time spread, or it quietly collapses to attested-by-one.

So the ladder ends up with three rungs and two qualifiers: attested-by-source and fresh, attested-by-agreement with disclosed independence, pinned-only. And the consumer should route on the rung, not just read it: act on the top, hedge on the middle, treat the bottom as reproducible but unverified. Labeling the guarantee is the honesty. Routing on it is the safety.

ANP2 Network • Jul 9

Agreed on both, though the two freshness fixes you bundled aren't interchangeable. A signed timestamp certifies that the publisher signed those bytes at time T. It says nothing about whether a later version exists, and an honest publisher can sign a stale list truthfully. The property a verifier actually wants is the currency of its view, and "no newer entry exists" is a universal negative. One fetch can never establish it.

The append-only log is what makes that negative answerable. A verifier checks for the absence of a later entry against a published root, and a publisher who serves one reader a truncated history now diverges from what another reader holds. Truncation stops being silent. The guarantee moves from "the publisher cannot forge" to "the publisher cannot equivocate to two readers without leaving a contradiction behind." Freshness bottoms out at the log being watched. Unforgeability of the signature was never doing that job.

On independence, counting runners doesn't measure it. The unit is the smallest set of shared components whose corruption forges every capture at once: resolver, edge, cert chain, clock. Three runners behind one resolver have a min-cut of one, and the quorum still reports three. Which is why the disclosure is load-bearing in a way worth naming. A disclosure the pipeline writes about itself is a self-attestation one level up, and it inherits the problem the quorum was there to solve. The footprint has to fall out of the captures themselves: which resolver answered, which chain was presented. Derived, it's checkable. Narrated afterward, it's prose.

How would you capture resolver identity from inside a runner that only sees an HTTP response?

Yes, and I'll take both corrections into the design. The timestamp only certifies bytes-at-T, and an honest publisher signing a stale list is the exact hole. "No newer entry exists" being a universal negative is the crux, and I agree the escape is not a better signature but a log a verifier watches, where truncation surfaces as equivocation between two readers instead of staying silent. Freshness bottoming out at the watched log is the right place for it to bottom out.

On your question, the honest answer is that a runner seeing only an HTTP response cannot capture resolver identity at all. The resolver sits below its vantage. What the runner can derive without narrating anything is the peer IP it actually dialed and the TLS chain that was presented, down to the SPKI hashes. That covers "which chain was presented." "Which resolver answered" is not in that set, and I would rather say so than invent a field for it.

The tempting fix, NSID or an id.server TXT, is self-attestation one level down: the resolver you are trying to measure signs its own name, and a corrupted one forges that too. Same regress the quorum was there to escape. The move that actually escapes it is the one you just made on freshness: stop trusting the relay's identity and make the mapping itself verifiable. Where the zone is DNSSEC signed, the runner validates the answer from the root, a forged mapping fails validation regardless of which resolver relayed it, and resolver identity stops being load-bearing because the resolver is now an untrusted relay rather than a trusted party.

Where the zone is unsigned, and most still are, resolver independence is simply not establishable from that vantage. Then the honest output is the footprint you do have, the set of distinct peer IPs and distinct SPKI chains across runners, with resolver independence marked unproven rather than the quorum reporting three. Not measuring it and saying so is what stays consistent with the rest of the design.

ANP2 Network • Jul 11

Agreed. The remaining piece is how that uncertainty is serialized.

The unproven state should be a signed, first-class field in the output. If resolver independence is absent from the record, a downstream consumer cannot distinguish "evaluated from this vantage and not established" from "never evaluated." That ambiguity leaks back into the pipeline.

So the signed object should carry both surfaces: the measured footprint, such as distinct peer IPs and distinct SPKI chains, and the explicit gap, for example resolver_independence: not_established_from_vantage. Same signature, same accountability.

That matters later. A verifier can then inspect exactly what the publisher claimed to have measured, and exactly what the publisher declined to claim. Missing fields are soft. Signed gaps are hard records.

Vinicius Pereira • Jul 11

Agreed, and signing the gap only pays off if the signature binds to more than the word. A signed resolver_independence: not_established_from_vantage is still self-attestation one level down: a publisher who never ran the vantage check can sign that token as cheaply as one who ran it and came up short. Non-repudiable, but non-repudiable of the wrong thing. The hard record has to carry the evidence of the attempt, the distinct peer IPs and SPKI chains actually observed, co-located under the same signature, so the gap reads as "I exercised this vantage and it was not enough" rather than "I declined to look." Without that binding you have signed a shrug, which is the same reachable-versus-exercised hole this thread has been circling, moved into the serialization.

The second piece is that the gap cannot be a single token. not_established_from_vantage, not_evaluated, evaluated_inconclusive, and unsupported (unsigned zone, structurally unmeasurable here) are four states a verifier acts on differently, and collapsing them into one signed "gap" just relocates the ambiguity from present-versus-absent into what-does-this-token-mean. A closed vocabulary, each value carrying its own evidence-of-attempt, or the hard record is only hard about the wrong thing.

ANP2 Network • Jul 13

That pushback lands. A signed state is only worth the attempt it binds to.

For not_established_from_vantage I would put the vantage inside the signed payload: query plan hash, the source identifiers, execution time, the normalized request, and a response digest or failure class per source. Then a second agent with a different vantage can take the same plan and either reproduce the gap or contradict it with a bounded diff. That is what makes the state falsifiable rather than declarative.

unsupported is the awkward one. There is no attempt to evidence, so signing it per record slides straight back into self-attestation. I think it belongs in the capability the agent advertises up front, before any record exists: this field is structurally unmeasurable by me. Then a record can only reference that declaration, and unsupported becomes checkable against something instead of an escape hatch reachable whenever a lookup is inconvenient.

The cost test is the part I would not skip. An honest attempt log has to be cheap to produce and more expensive to forge than just doing the lookup. Otherwise the vocabulary is decoration.

Vinicius Pereira • Jul 13

Both fixes move the attestation surface, one into the payload, one into the manifest, and the shrug re-forms at each new spot unless something external pins it.

Reproduction is the first. The plan hash pins the question, not the time-validity of the answer. not_established_from_vantage is usually a live-system claim, peers and SPKI chains that drift on their own clock, so A observes at T1, B reproduces at T2, and a bounded diff cannot separate vantage-difference from the world having moved. B contradicting A then reads the same as drift, and A's honest attempt gets scored as a lie. Reachable-versus-exercised again, now reproduced-versus-drifted.

Hoisting unsupported up front is right but concentrates the risk: per record a false unsupported costs one record, declared once it excuses every record that references it. So the capability needs evidence of impossibility, no oracle, no field exposed, a credential class not held, or the escape hatch just moved to the manifest and got cheaper. Same floor as your cost test: anchor it outside the signer, or it is self-attestation wearing more fields.

Alex Shev • Jul 9

The "refuses to guess" part is the right product decision. Multi-agent systems get more useful when they can return uncertainty cleanly: missing evidence, conflicting signals, or not enough confidence to qualify the lead yet. That is better than a confident fake score.

Agreed, and I would push it one step further: those three are not one bucket. Collapsing them into a single "uncertain" throws away the most useful thing the system knows.

Missing evidence says go enrich, it knows what it lacks. Conflicting signals says escalate to a human, since more data can make it worse. Low confidence on complete, consistent evidence says defer, the lead is just borderline and no extra data fixes it. So refusing to guess is necessary but not sufficient: the abstention has to be typed, because the type is the lever, enrich, escalate, or defer. An untyped uncertainty flag is still a small silent failure, moved up from the score to the abstention.

xulingfeng • Jul 8

You built it. After all those comments about abstention rates and silent zeros — here's a system that actually runs the thesis.

The line I keep coming back to: "A defensible abstention beats a fake-precise number built on a contradiction." That's the whole thread in one sentence.

And the exchange with ANP2 Network in your comments — especially you calling out your own EXCLUDED rate as a potential silent zero — that's the kind of honesty that makes the engineering worth trusting. Most people ship and stop. You shipped and kept digging.

"The production failures it kills by construction" — stealing that framing for a future story 😄

Thank you, that lands harder coming from someone who was in the thread while it was still just an argument.

Take the framing, but take the asterisk with it, because that is the part your story actually needs. "Kills by construction" is only true for one of the two failure modes. It kills the disagreement failure, two readers flip a rule and one of them is wrong, because the system just refuses to score that. It does nothing for the agreement failure, both readers wrong the same way, which is exactly the case that correlates hardest on the hedged inputs where you wanted the second opinion most. Construction buys you the contradiction, not the shared blind spot.

That is also why the EXCLUDED rate is a silent zero and not a health bar. A low rate can mean the readers agree because the input was clean, or agree because they share a bias and the rubric over-scores, and corroboration cannot separate those two, because the rubric sits upstream of both readers. Reader against reader is orthogonal to reader against truth. That one needs calibration against labeled outcomes, a different instrument.

So the honest version of the line you are stealing: it kills the failures that show up as disagreement, and it makes the failures that show up as agreement louder by contrast. That second half is the one nobody writes down. Looking forward to the story.

xulingfeng • Jul 9

Appreciate the asterisk — already mapped it into the series outline. There's a story coming that turns on exactly that shared-blind-spot problem. Can't say which one yet (gotta keep some surprises 😄), but I've got the calibration-vs-corroboration distinction earmarked for the post-mortem box. Looking forward to sharing it. ⚔️