DEV Community: Vinicius Pereira

Python's casefold() merged two of my customers into one tenant

Vinicius Pereira — Wed, 12 Aug 2026 18:17:44 +0000

I spent a few days building a repo about multi-tenant retrieval. One knowledge base, many customers, and one promise: a customer can never retrieve another customer's documents. The filter runs before the candidate set is scored, the guard re-checks every chunk on the way out, every query lands in an audit log.

Then I ran an adversarial pass whose only job was to break the promise rather than confirm it. It broke it in four lines of setup, and the hole was in the function I had written specifically to prevent identity confusion.

The line that was supposed to be boring

Entitlement is compared on tenant identifiers, so identifiers get canonicalised once, at construction, and compared exactly afterwards:

def normalise_id(value: str) -> str:
    return value.strip().casefold()

The reasoning felt obvious. " ACME " and acme are the same customer, somebody will paste one with a trailing space eventually, and a fence that says "access denied" because of a space is a support ticket. casefold() rather than lower() because casefold is the Unicode-correct one, the one you are told to use for caseless comparison.

That is exactly why it is wrong here.

The tenant that was two tenants

The adversary created a customer called Straße-Werke and gave a completely unrelated principal a grant on strasse-werke. The unrelated principal read the documents.

>>> "Straße-Werke".casefold() == "strasse-werke".casefold()
True

casefold() is designed for caseless matching, and caseless matching is deliberately many-to-one. It is not a case conversion, it is a mapping onto a common form, and several of those mappings collapse characters that are not case variants of each other at all:

>>> "ß".casefold()          # one character becomes two
'ss'
>>> "K".casefold()     # KELVIN SIGN
'k'
>>> "ﬁ".casefold()     # LATIN SMALL LIGATURE FI
'fi'
>>> "ς".casefold() == "σ".casefold()   # final sigma
True

Every one of those is a merge. Feed casefold() two identifiers that an upstream registry considers distinct, and you get one entitlement key. Two customers become one customer.

Why this is worse than a normal access bug

Nothing failed. That is the part worth sitting with.

The filter ran correctly, against a key that was wrong. The candidate set was built correctly, from a key that was wrong. My guard, the deliberate second check that re-validates every chunk before it leaves, calls the same entitlement function, so it agreed. The audit log recorded a completely ordinary query:

QUERY principal=svc-partner allow=[strasse-werke/*/*] deny=[] returned=3

There is no exception, no anomaly, no unusual pattern. A reviewer reading that line sees a principal reading documents it is entitled to read, because by the time anything is logged, the two tenants are already the same tenant. Every layer of defence in depth inherits the same wrong key, so depth buys you nothing. The mistake happened before the first layer ran.

Then the same bug again, wearing different clothes

Having fixed the fold, I still had strip(). str.strip() with no argument removes all Unicode whitespace, not just the ASCII kind:

>>> "kronos\u3000".strip() == "kronos"   # IDEOGRAPHIC SPACE
True

The same applies to U+00A0, U+2007, U+1680, U+205F, U+0085 and U+2028. So a registry that enforces uniqueness on the raw string happily accepts kronos and kronos plus an ideographic space as two different customers, and my fence folds them into one.

The detail that makes this a good trap: zero-width space and the other invisible characters that people usually test for, U+200B, the soft hyphen, the byte order mark, are not stripped. So a test suite that covers "invisible characters" in the obvious way passes while the actual merge path stays open.

Two fixes I tried and threw away

Use lower() instead of casefold(). It reads like the conservative choice. It does not help:

>>> "K".lower()
'k'

The Kelvin sign folds under lower() too, because that mapping is simple case conversion, not full folding. Swapping the function narrows the hole without closing it, which is the worst outcome available: it feels fixed.

Reject non-ASCII identifiers at construction. This closes it, and it is wrong for a different reason. It refuses legitimate identifiers from most of the world in order to fix a bug in my normalisation, and it hides the actual rule behind a character-set restriction that has nothing to do with the problem.

The fix

Canonicalise as little as possible, and refuse anything that would need canonicalising to be safe:

ASCII_FOLD = str.maketrans(ascii_uppercase, ascii_lowercase)
ASCII_WHITESPACE = " \t\n\r\v\f"

def normalise_id(value: str) -> str:
    """Trim ASCII whitespace, fold ASCII case, and nothing else."""
    return value.strip(ASCII_WHITESPACE).translate(ASCII_FOLD)

Straße-Werke and strasse-werke now stay two tenants. A tenant named with non-ASCII characters keeps working, and keeps its identity.

Then the second half, which matters more than the first: anything still carrying whitespace or a non-printable character after the trim is refused at construction, naming the offending code point in the error. Not folded onto its neighbour, not silently accepted. A tenant id with an ideographic space in the middle of it is a data problem upstream, and the honest thing a fence can do is say so out loud instead of guessing which neighbour it meant.

The general shape of it

The lesson generalises past Unicode, and it is the reason I am writing this down rather than quietly pushing the patch.

When a normalised value becomes a security key, normalisation is part of the attack surface. Any many-to-one transform applied to an identity is a merge operation, and a merge between two identities is a privilege grant. Case folding, accent stripping, whitespace collapsing, punctuation removal, homoglyph mapping, lowercasing an email local part: all of them are helpful in a search box and all of them are dangerous the moment the output is compared for authorisation.

Two questions worth asking of any such function:

Can two inputs that the system elsewhere considers distinct produce the same output?
If they can, which layer is supposed to notice? If the answer is "the layer that compares the outputs", there is no such layer.

The reason I found this at all is that the adversarial pass was told to breach the fence, not to test it. A test written by the person who wrote the code asks "does it do what I meant". A test written to break it asks "what did I mean that was wrong". Those find different bugs, and the second kind is the kind that reaches production.

Why the repo concentrates risk there on purpose

The repo's actual argument is older and simpler than the Unicode story. Most multi-tenant retrieval filters after the ranking, which means a leak needs one caller, one cache, one new endpoint to forget. Filtering before the candidate set is scored moves all of that risk into a single function.

Which is precisely how I ended up here. Concentrating the risk is the right trade, and the bill it comes with is that the one function you concentrated it into is now the only thing worth attacking. That is why it carries 141 adversarial tests, and why four of them are about a character that is not a case variant of anything.

The repo is at github.com/vinimabreu/tenant-fence: the fence, the deliberately wrong version kept next to it so the suite can demonstrate the leak rather than describe it, and the tests that would have caught this on day one.

My exactly-once refund paid twice, and 373 passing tests could never have caught it

Vinicius Pereira — Tue, 04 Aug 2026 21:24:54 +0000

I spent a week building a repo about making LangGraph agents trustworthy in production. Routing measured against labelled fixtures instead of vibes. A human gate before anything irreversible. Durable resume, so a process that dies mid-refund comes back and does not pay the customer twice.

373 tests, all green. CI green on four Python versions. Then I ran an adversarial pass over it, whose only job was to break my claims rather than confirm them, and the first thing it broke was the one I had written the repo to prove.

The claim

The centre of the whole thing is one sentence from my README:

INSERT OR IGNORE then read. The claim and the check are one statement, so two attempts cannot both believe they are first.

Standard idempotency. Every effectful action gets a key, the key is inserted once, and whoever loses the race replays the stored result instead of running the handler again. My tests hammered it: eight concurrent processes on one key, one handler execution, seven replays. Crash injected at every boundary in the run, resumed from disk, effect count unchanged.

The bug

The key was built like this:

action_id = digest({
    "ticket": ticket_id,
    "tool": call.name,
    "args": call.args,      # <- what the model typed
})

call.args is raw model output. A language model asked for a $45.00 refund can write that amount in more than one way, and all of them are valid JSON that my schema happily accepts:

    4500  ->  action_id 1270c45d1612d5d1
  "4500"  ->  action_id 8f0f6363fea8adcf
  4500.0  ->  action_id b78a82d722dde95d

distinct idempotency keys for one identical $45.00 refund: 3

Three keys. Three refunds. Same customer, same invoice, same amount.

If a queue retries a ticket, or a customer submits twice, or the same conversation is driven again for any of the boring reasons production does that, and the model spells the number differently on the second pass, my exactly-once guarantee quietly becomes at-least-once. On money.

Why my tests were structurally incapable of finding it

This is the part I keep thinking about.

The test suite runs offline against a deterministic stand-in: a keyword classifier that, given the same ticket, emits byte-identical JSON every single time. That is a deliberate design choice and mostly a good one. It makes the suite fast, free, and reproducible, with no API key and no network.

It also means the second delivery of a ticket always produced exactly the same string as the first. The suite could never observe the failure, because the only component capable of producing the failure had been replaced with one that cannot.

My tests were not weak. They were blind by construction, in a way that green output cannot show you. Every assertion I had written was true. The thing I never asserted was that two equivalent calls collapse to one key, because with a deterministic generator, equivalent and identical are the same word.

The fix is one line of intent: derive the key from the arguments after the tool's own schema has parsed them, not from whatever the model typed.

parsed = spec.validate_args(call.args).model_dump(mode="json")
action_id = digest({
    "ticket": ticket_id,
    "tool": call.name,
    "args": parsed,          # parsed, not typed
})

Now 4500, "4500", 4500.0, " 4500 " and "+4500" all produce one key, while seven genuinely different refunds still produce seven. And there is a test that drives the same ticket twice with different spellings and asserts a single handler execution, so it stays fixed.

The second thing it broke: a metric that was measuring the wrong noun

While it was in there, the same pass killed a number I had been quoting proudly.

My report said tool choice 21/22 correct (95.5%). It compared the name of the tool the model picked against the expected name. Nothing anywhere compared the arguments.

So this ticket:

We were charged $90.00 on INV-10032 but only $45.00 of that was valid. Refund the difference.

with the model proposing issue_refund(invoice_ref="INV-10032", amount_cents=9000) scored as perfectly correct. Right tool, double the money. A version proposing $9,000 instead of $45 also scored perfectly correct.

"Chose the right action" and "chose the right tool name" are not the same claim, and I had been publishing the second one under the first one's label.

The fix added expected arguments to every fixture and a separate metric that counts them. Which is why my headline numbers got worse:

	before	after
routing accuracy	92.0%	90.2%
tool metric	21/22 name only	21/23 counting arguments
in-doubt stops	8	24
crash boundaries swept	176	208

Nothing regressed. The measurements stopped flattering me. The in-doubt count tripled because the audit also found two windows inside the effect where a crash could land and my crash-point list did not know they existed, which meant a line in my report was true by accident rather than by design.

What the repo does now

The whole thing in one pass. What follows is the same three acts, one at a time.

Routing is scored per branch with a confusion matrix, and the parser never raises: output it cannot trust becomes an escalation carrying one of nine named reasons, recorded in state.

Irreversible tools stop for a human, and the interrupt is measured in both directions. Missing a required stop is scored at threshold zero, because there is no acceptable rate for shipping a refund nobody approved. Stopping when you did not need to is scored separately with a real budget, because approval fatigue is how a gate stops meaning anything: interrupt people often enough for nothing and they start clicking approve without reading.

And the part that took the longest to get honest. The process can be killed at any of 208 points in a run and resumed from disk. 112 of the reachable boundaries come back byte-identical with the effect count unchanged. The other 24 land in a window that genuinely cannot be closed, between claiming the effect and recording its result, and there the system stops and says it does not know rather than guessing. A human reconciles it with the ledger in front of them.

Zero duplicates. Zero mismatches. A CI gate fails the build when any of those numbers moves.

vinimabreu / langgraph-production

A LangGraph support agent with the reliability layer around it: routing measured against labelled fixtures, a human gate before irreversible tools, durable resume with exactly-once effects, and a CI gate that fails when a number moves.

langgraph-production

A LangGraph agent with the reliability layer that decides whether it can ship.

Building the graph is the easy half. The half that decides whether it goes live is everything around it: what happens when the process dies between two nodes whether the refund gets paid twice on the way back up, whether an approval somebody gave on Tuesday can authorise a different action on Thursday, and whether anyone can say out loud how often the router is right.

This repo answers those four questions with code and with numbers, on a support desk for a fictional product. All data is synthetic.

The report this repo exists to produce

Real output from python -m langgraph_production.evaluation on this repository, not an illustration:

========================================================================
SUPPORT GRAPH EVALUATION
========================================================================
  fixtures      answer 13, escalate 15, tool 23  (total 51)
ROUTING
  accuracy      46/51 = 90.2%
  macro F1      0.907
  tool choice   22/23 correct (95.7%) on

…

View on GitHub

The takeaway I did not expect

I went in expecting the audit to find sloppy corners. It found the opposite: my careful parts were fine, and the failure was hiding in the seam between two decisions that were each individually correct.

Using a deterministic stand-in for tests: correct.
Keying idempotency off the tool call: correct.

Together they produce a system that is provably exactly-once against a generator that cannot vary, and at-least-once against the one you actually ship with.

So the question I now ask about any test suite, including yours: what can this suite not observe, by construction? Not what did I forget to assert. What has been designed out of the room. If you replace the non-deterministic component with a deterministic one for testing, every bug that only exists because of non-determinism is invisible to you, and it will stay invisible while the dashboard stays green.

Green tests are evidence about the world you built for them. It is worth being explicit about how much of the real one you left out.

Code, numbers, and the harness that produces them: github.com/vinimabreu/langgraph-production

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

The ChocoDEV Signature Bar: one chocolate bar, zero images

Vinicius Pereira — Sun, 02 Aug 2026 22:08:15 +0000

This is a submission for Frontend Challenge: Comfort Food Edition, CSS Art.

Inspiration

Yesterday I submitted a landing page for ChocoDEV, a chocolate brand that exists only in CSS. A brand needs a product shot, and this brand has a rule: no images, ever. So the product shot had to be drawn in the only medium the brand allows.

It is one chocolate bar with a bite taken out of it, because a perfect untouched bar is a lie. Nobody photographs their comfort food before tasting it.

Demo

Live: vinimabreu.dev/chocodev/bar

The fallen chunk reacts if you click it. That is the entire JavaScript budget, three lines, and the piece is complete without them.

Journey

Every chunk is a molded facet: four trapezoid faces built from angular gradients, so the light lands differently on each slope, the way a real mold presses chocolate. The letters are not printed on top, they are pressed in, one dark shadow above and one thin highlight below, and that two-shadow trick is the whole illusion of depth.

The bite was the hard part. A clean clip-path zigzag looked like scissors had done it, so the exposed edge got fracture facets in lighter matte browns and a few crumbs, because chocolate never breaks politely.

There is exactly one animation, a slow sheen that crosses the bar like it is turning under a studio light, and it sits behind a prefers-reduced-motion guard. When your system asks for stillness, the bar just stands there looking edible.

Everything is gradients, box-shadow, border-radius and clip-path. Open devtools, it is chocolate all the way down.

ChocoDEV: a chocolate bar you can eat, in pure CSS

Vinicius Pereira — Sun, 02 Aug 2026 21:22:19 +0000

This is a submission for Frontend Challenge: Comfort Food Edition, Perfect Landing.

What I Built

ChocoDEV is a landing page for a chocolate brand that does not exist, and I am fine with that. Chocolate is my comfort food, and it turns out it is also South America's: the oldest traces of cacao were found in the upper Amazon, which as a Brazilian I consider a home win.

The page has one rule: zero images. No PNGs, no SVGs, no background URLs. Every square of chocolate, every drip, every bevel is gradients, box-shadow and keyframes. The footer makes the claim and devtools can check it, which is my favorite kind of claim.

The centerpiece is a chocolate bar you can actually eat. Each chunk is a button, each bite updates a counter, and when the bar is gone you get to bake a new one. There is also an exploded anatomy view of a single square, a scroll progress bar, and a testimonials section where four honest people confess things about chocolate that most of us only think.

Demo

Live demo: vinimabreu.dev/chocodev

Click a chunk of the bar. The counter is watching.

Journey

The zero images rule sounded like a gimmick and turned into the whole education. A believable chocolate square needs light coming from somewhere, and without images that means layered box-shadows: an inset highlight on top, an inset shadow at the bottom, a drop shadow below. Once the light direction was consistent, everything suddenly looked edible. Before that it looked like brown buttons.

The part I am most proud of is not visible. The page respects prefers-reduced-motion in three places: the CSS animations, the stat counters that jump straight to their final value instead of counting up, and the parallax that simply declines to run. Every interactive element is a real button with a name a screen reader can speak, the document has a language and a title, and the menu wraps instead of clipping on narrow screens. Accessibility is a judging criterion in this challenge, and I think it should be the first one: a landing page that only works for some visitors is a poster.

What I would do next: a dark and milk chocolate theme toggle, and sound on the snap. Probably a mistake. Most good ideas about chocolate are.

Thanks for reading, and go eat something that makes you feel at home.

Everything that broke when I imported hand-written Make blueprints into a real workspace

Vinicius Pereira — Tue, 28 Jul 2026 17:52:42 +0000

The import dialog took my JSON without a single complaint. The first execution died on module 2 with BundleValidationError: Validation failed for 7 parameter(s).

That gap between "the import accepted it" and "the runtime will actually run it" is where I spent most of a day this week, and almost none of it is documented anywhere I could find. So here it is, in the order it broke.

Context, briefly. I write Make (ex-Integromat) scenarios as blueprint JSON by hand instead of assembling them in the canvas, because I want them in git, diffable, reviewable. The scenarios route an order flow between an ERP and a 3PL. Every decision that can go wrong (is this order complete, which carrier code does a typo'd shipping method mean, is this event a duplicate) lives in a typed FastAPI service with 83 pytest tests, and every response carries the same envelope: decision for the routers to branch on, reason for a human, evidence for the audit trail. The scenario routes; the service decides. To test the whole thing for real, I imported the blueprints into a live Make workspace and exposed the local service through a cloudflared tunnel.

Then the workspace started grading my homework.

1. Import is a parser, not a validator

The blueprints were clean JSON. Modules, routes, mappings, all structurally valid, and the import agreed. The first run did not: Validation failed for 7 parameter(s), naming seven fields I had never typed in my life, all missing from the HTTP module's mapper: serializeUrl, shareCookies, rejectUnauthorized, followRedirect, useQuerystring, gzip, useMtls.

These are the checkboxes the canvas quietly fills in when you drop an HTTP module onto a scenario. Write the module by hand and nobody fills them, and import does not care, because import only checks that the JSON parses into modules and routes. The module's own contract is enforced at execution time.

Fine. I added the seven and ran again.

Validation failed for 1 parameter(s): followAllRedirects.

The first error listed seven missing parameters when there were eight. Runtime validation happens in layers, and each run only surfaces the layer it died in. Budget one execution per layer of complaints; the error list in front of you is not the whole bill.

Here is the mapper that finally runs, straight from the blueprint:

"mapper": {
  "url": "https://YOUR-SERVICE-HOST/orders/validate",
  "method": "post",
  "headers": [{ "name": "Content-Type", "value": "application/json" }],
  "data": "{\"order_ref\": \"{{1.order_ref}}\", \"customer\": {{13.json}}, \"shipping_address\": {{14.json}}, \"shipping_method\": \"{{1.shipping_method}}\", \"lines\": {{12.json}}}",
  "serializeUrl": false,
  "shareCookies": false,
  "rejectUnauthorized": true,
  "followRedirect": true,
  "followAllRedirects": false,
  "useQuerystring": false,
  "gzip": true,
  "useMtls": false
}

Keep that block. It is the difference between a blueprint that imports and a blueprint that runs.

2. `{{1.body}}` does not exist, and the failure mode is silence

I had written the webhook mappings the way anyone with HTTP reflexes would: the webhook receives a POST, so the payload must live in {{1.body}}. It does not. Make's custom webhook parses the incoming JSON and exposes the fields at the top level of the module output: {{1.order_ref}}, {{1.shipping_method}}. There is no body.

And here is the part that costs hours: referencing a path that does not exist is not an error in Make. It renders as empty. Silently. My service received b'' first (the whole body reference resolved to nothing), and after a partial fix, the literal string null. To the validator those looked like genuinely broken requests, so it did its job and reported them as such, which pointed me at the wrong suspect. I went through the service looking for a bug that was not there. The service was fine. The bug was an absence.

The ten lines that ended the guessing

What broke the loop was not reading the scenario harder. It was logging what actually crossed the wire:

from fastapi import FastAPI, Request

app = FastAPI()

@app.middleware("http")
async def log_raw_body(request: Request, call_next):
    body = await request.body()
    print(f"{request.method} {request.url.path} raw={body!r}")
    return await call_next(request)

Behind a tunnel, this is a complete diagnostic rig for a visual tool. Every mapping experiment in the canvas shows up seconds later as the exact bytes it produced. My terminal reads like a fever chart of the session: raw=b'', then raw=b'null', then a perfect payload. Once I could see what Make sent instead of inferring it from downstream symptoms, every remaining bug fell in minutes.

3. Collections do not serialize into text fields

Next, the ops-alert branch. When validation fails, the scenario posts an alert carrying the list of gaps, and I mapped the gaps array straight into the raw JSON body of that request. Make rendered the collection as text, inside the quotes it happened to land in, and produced a body that was no longer JSON. The endpoint said 422, and it was right to.

A collection mapped into a text field does not become JSON. It becomes a string shaped like whatever Make's text rendering of that collection is, which inside a hand-built JSON body is a syntax error.

Two fixes, both in the final blueprints: run each collection through a JSON > Transform to JSON module and embed the result without surrounding quotes ({{12.json}}, not "{{12.json}}"), or keep alert bodies scalar-only (gap_count instead of the gaps array). Look back at the mapper above and you can see both rules at work: webhook scalars quoted, collections embedded bare.

4. The default schedule is a 15-minute tick

The scenario worked. Then it appeared to die. I sent test events and nothing happened. No error, no execution, no log line. Nothing.

Make's default schedule runs a scenario every 15 minutes. A webhook trigger without "Immediately as data arrives" queues events silently until the next tick. To anyone watching, a perfectly healthy integration looks down for up to fourteen minutes at a stretch. One dropdown. Nothing broken. This is the incident titled "the integration stopped working" that resolves itself before anyone finishes triaging it.

What the wire showed when it all worked

Every decision, reason, gap and score above is taken verbatim from the live runs described in this post.

Three routes, executed live through the imported scenario, values as returned:

A clean order came back VALID with 15 checks passed, the shipping method mapped MAPPED_EXACT to SM-STD-01, and the order was created at the 3PL. The happy path nobody writes posts about.
A broken order came back INVALID with the reason 7 gap(s) found, 8 check(s) passed (the same 15 checks, now split), and routed to the ops alert with gap_count 7. The router branched on decision; the reason string went into the alert text unchanged.
An order with the shipping method typed as "parcel" scored 0.60 on fuzzy match, below the 0.85 floor, so the service returned UNMAPPED with one ranked candidate and the reason returning 1 candidate(s) for human review instead of guessing. The scenario put the order on hold for a human.

That last route is the whole argument. A visual flow will route a wrong guess with the same confidence as a right one. The refusal to guess has to come from somewhere with a test suite.

The outage I did not schedule

Mid-session, cloudflared dropped the tunnel for real. Cloudflare error 1033, service unreachable. The Break error handler on the HTTP module (current Make docs call the directive Retry; the blueprint directive is still builtin:Break) parked the execution in Incomplete Executions with its mappings intact. Nothing lost, nothing half-applied, one click away from resolving once the service was reachable again. The exact failure mode this design exists to survive walked in unannounced, and the pattern held. I could not have scripted a better demo, and for once I didn't.

What this does not tell you

The middleware shows what arrives, not what Make intended. A mapping bug that happens to produce valid-shaped JSON with wrong values sails through both the wire log and the schema check. The semantic checks still have to live in the service.

None of this makes Make the villain. The event queue, the retry policy, the incomplete-executions safety net are better than what most hand-rolled webhook consumers ship. That is exactly why the routing belongs there, and the deciding does not.

Hand-writing blueprints is a real trade. The canvas would have filled those eight HTTP parameters for me and I would never have met BundleValidationError. I pay that tax to get scenarios in git with reviewable diffs. If you do not need the diff, click the modules.

And every Transform to JSON module is one more operation on every run, which is the unit Make bills in. Scalar alert bodies are not just simpler. They are cheaper.

The service, both importable blueprints with all eight parameters already in place, and these lessons written down live at github.com/vinimabreu/make-failsafe, mostly so I never rediscover any of this at 2 a.m.

The flow routes, the service decides, and the wire is the only place where both of them tell the truth.

Your LLM's confidence score is lying to you

Vinicius Pereira — Mon, 20 Jul 2026 11:31:33 +0000

The most expensive bug in an LLM system is not the output that is obviously broken. That one you catch. It is the fluent, well formed, confidently wrong answer that reads exactly like a correct one and walks straight into your system of record because nothing was standing between the model and production.

"Reads correct" and "is correct" are two different properties, and the model only optimizes the first. So the answer that is almost right, the near miss, comes back just as polished as the answer that is exactly right. You cannot tell them apart by looking. Neither can the model.

Similarity is not confidence

When teams want to gate on that, they reach for a number. Usually one of two: the model's own reported confidence, or the similarity score of whatever chunk it retrieved. Both feel like confidence. Neither one is.

Ask a model how sure it is and you get more generated text, sampled from the same distribution that just produced the wrong answer. It is a guess about a guess. And a similarity score measures proximity, not truth. The near miss scores high precisely because it sits close to the right answer. That is the whole trap: a similarity cutoff hands its highest marks to the single most dangerous output in the system, the confident near miss, and waves it through.

Compute it from what you can check

The way out is to stop reading confidence off the model and start computing it from signals you can verify without trusting the model at all. Three of them carry most of the weight.

Grounding asks whether the output is actually entailed by the evidence you gave it, not whether it reads well. An answer nobody can trace back to a source is not trustworthy, however clean it looks. A lexical-overlap baseline gets you moving, but the real seam is a (claim, evidence) -> float function, so you can swap in an NLI model or a cross-encoder the moment overlap is too blunt. Entailment over proximity.

Agreement resamples the generator on the same input and measures whether the answer survives. A model that flaps between three answers when you ask three times is telling you something a single clean draw hides. A minority draw is low confidence even when it parses perfectly. This is the one signal that costs you extra generations, so you price it in on purpose.

Validation is the cheap one, and the one people skip. Does the output satisfy the schema you declared, every required field present and the right type? A structurally invalid output should never reach the auto-accept ceiling no matter how good the other signals look, so one missing field drags the score down instead of passing silently.

The gate, as code

I wrote this up as a small reference implementation, confidence-gate, so the pattern has something runnable behind it. The generator goes in as a plain callable, which keeps the whole thing deterministic and runs it with no API key. Wiring it looks like this:

from pydantic import BaseModel

from confidence_gate import (
    AgreementSignal, GroundingSignal, Policy,
    ReviewQueue, Router, Task, ValidationSignal,
)


class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    total: str
    date: str


def generator(prompt: str) -> str:
    # Swap in a real model call. The gate does not care where the text
    # comes from, only whether it survives the signals.
    return '{"invoice_number": "INV-4021", "vendor": "Northwind Supplies", "total": "1840.00", "date": "2026-02-11"}'


queue = ReviewQueue(":memory:")
signals = [ValidationSignal(Invoice), GroundingSignal(), AgreementSignal(generator, k=3)]
router = Router(generator, signals, Policy(), queue)

verdict = router.route(
    Task(
        task_id="inv-1",
        prompt="extract invoice fields",
        evidence="Invoice INV-4021 from Northwind Supplies. Date 2026-02-11. Total 1840.00 USD.",
    )
)
print(verdict.decision, verdict.aggregate_confidence)

Run the bundled demo over three documents, one clean, one ambiguous, one with no supporting evidence, and it prints the decision for each. Captured output:

TASK            DECISION        CONF  SIGNALS
----------------------------------------------------------------
northwind-clean AUTO_ACCEPT     1.00  validation=1.00 grounding=1.00 agreement=1.00
globex-partial  HUMAN_REVIEW    0.50  validation=0.75 grounding=0.50 agreement=0.67
unknown-source  ABSTAIN         0.00  validation=1.00 grounding=0.00 agreement=1.00

review queue: 1 pending
  #1 globex-partial  conf=0.50  :: min aggregate 0.500 (accept>=0.75, abstain<=0.35) -> human_review

The middle row is the whole argument. globex-partial validates against the schema except for one missing field, only half of it is supported by the evidence, and the generator does not fully agree with itself on resampling. No single number screams "wrong", so the gate refuses to auto-accept and refuses to abstain. It routes the item to a person and keeps the receipt.

The bottom row is the one that would have burned you. unknown-source is structurally perfect, validation 1.00, and the model reproduces it on every resample, agreement 1.00. By any signal you could read off the model, it is maximally confident. Grounding is 0.00 because nothing in the evidence supports it. A confident, self-consistent answer with no external support, and the gate abstains.

Tune on the cost of a false accept

Look at how the aggregate falls out. It is the minimum of the signals, not the average. That is deliberate, and it comes from the cost structure rather than a taste for being strict. A cache miss or a re-run costs seconds. A false accept ships a confident wrong answer into the record with no model left in the loop to hedge it. Those two outcomes are not symmetric, so you do not tune the threshold to hit some acceptance rate. You tune it on what a single false accept costs you. A gate is only as strong as its weakest verified signal, so one failing signal is enough to deny auto-accept. Abstaining is cheap. Being confidently wrong is the bill you do not see coming.

That leaves three outcomes instead of two. Auto-accept when the external signals agree. Human review, with the full audit trail, when they are mixed and a person should look. Abstain when there is no support at all, because refusing to answer beats guessing when you cannot check the guess. The model proposes, the code disposes.

What this does not do

It does not make a wrong answer right. It refuses to ship one silently, which is a smaller and far more achievable goal. If your evidence is itself wrong, grounding will happily confirm a wrong answer against it, so the quality of your source text just became part of your trust boundary. The default entailment is lexical overlap, which is blunt: it catches "the answer cites nothing in the source" but not "the answer inverts the one number that mattered", so anything real gets a stronger function through that seam. Agreement costs you K generations per item, which is real latency and real money traded for a convergence signal. And none of this is a product with uptime promises. It is the shape of the design, made runnable.

A confident wrong answer and a confident right one are the same string until something outside the model checks. Be that something.

Clear the Lineup: the passage my RAG retrieved and then threw away in silence

Vinicius Pereira — Tue, 14 Jul 2026 23:24:30 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

rag-quality is a small, self-contained RAG pipeline: it retrieves passages from a corpus (sparse, dense, and hybrid modes), then generates a grounded answer that cites the passages it used and abstains when the context does not support one. It ships with an eval harness that scores retrieval quality.

Recently I commented on a dev.to thread about non-deterministic retrieval, the ways a RAG can fetch the right passage and still answer wrong. In reply, the author added one more check to his list: verify the context you asked for is the context you actually got. That check named the exact hole I had in this repo, one stage past retrieval, in assembly, against a bug I had shipped and not noticed.

Bug Fix or Performance Improvement

The generation step assembles retrieved passages into a context block and sends it to the model. Here is what it did:

def build_prompt(question: str, hits: list[Hit]) -> str:
    blocks = [f"[{h.chunk.id}] {h.chunk.text}" for h in hits]
    context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS]
    return f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"

MAX_CONTEXT_CHARS is 6000. It joins every retrieved passage, then hard-slices the string to 6000 characters. If the joined context runs longer, the tail is gone. The tail is the lowest-ranked hits, and the lowest-ranked hit is often the one that actually holds the answer: retrieval surfaced it, ranking put it last, the slice deleted it. No exception, no log, no return value that hints anything was cut. The caller gets a shorter prompt and believes it is complete.

What makes this one sting is the asymmetry a few lines down in the same file:

if message.stop_reason == "max_tokens":
    text += "\n[warning: answer truncated at the token limit]"

The code already knows truncation is dangerous. It shouts when the model's output is cut. It stays silent when the model's input is cut, which is the more dangerous of the two, because a truncated input can quietly remove the one passage the answer depended on and the model will confidently answer from what is left.

And the tests stayed green because nothing exercised the failure. The sample corpus is 17 short documents; its joined top-k context fits comfortably under 6000 characters, so the slice never removed anything in the bundled runs. The eval harness measures retrieval quality (hit@k, MRR, recall@k), whether the right passage was retrieved, not whether it survived assembly into the prompt. The bug lives in the gap between those two stages, which is precisely the gap no metric was watching.

Code

The whole change, the fix, the tests, and the optional Sentry and Gemini tooling, is one pull request:

Fix silent context truncation in build_prompt #1

vinimabreu posted on Jul 14, 2026

build_prompt joined every retrieved passage and hard-sliced the result to MAX_CONTEXT_CHARS, dropping the tail with no exception, log, or return signal. The tail is the lowest-ranked hits, and the lowest-ranked hit can be the one that holds the answer. The same file already warns on output truncation but stayed silent on the input side.

This packs whole passages in rank order up to the budget, never cuts a passage mid-text, and returns dropped_ids so the caller announces what was left out, mirroring the existing output-truncation warning.

Also adds:

env-gated Sentry spans across the pipeline (no-op without sentry-sdk and SENTRY_DSN), so an over-budget request shows total_context_chars, dropped_ids, and a context.truncated_chars measurement on the rag.build_prompt span
scripts/root_cause.py, an optional Gemini helper that localizes the loss from a trace plus the source
five tests that run key-free and offline

View on GitHub

My Improvements

Stop slicing a string. Pack whole passages in rank order until the next one would blow the budget, never cut one mid-text, and return the ids you dropped so the caller can announce them.

-def build_prompt(question: str, hits: list[Hit]) -> str:
-    blocks = [f"[{h.chunk.id}] {h.chunk.text}" for h in hits]
-    context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS]
-    return f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"
+def build_prompt(question: str, hits: list[Hit]) -> tuple[str, list[str]]:
+    separator = "\n\n"
+    budget = config.MAX_CONTEXT_CHARS
+    blocks = [f"[{hit.chunk.id}] {hit.chunk.text}" for hit in hits]
+
+    kept, dropped_ids, used = [], [], 0
+    for hit, block in zip(hits, blocks):
+        extra = len(block) + (len(separator) if kept else 0)
+        if used + extra <= budget:
+            kept.append(block)
+            used += extra
+        else:
+            dropped_ids.append(hit.chunk.id)
+
+    if dropped_ids:
+        log.warning("context truncated: %d passage(s) dropped: %s",
+                    len(dropped_ids), ", ".join(dropped_ids))
+
+    context = separator.join(kept)
+    prompt = f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"
+    return prompt, dropped_ids

That diff is the essence of the change. The shipped version also opens the rag.build_prompt span from the Sentry section around this same code, which is how those numbers reach the trace.

Generator.answer now unpacks the tuple and appends a note that mirrors the existing output warning, so the drop is visible in the answer itself:

if dropped_ids:
    note = ", ".join(dropped_ids)
    text += f"\n[warning: {len(dropped_ids)} passage(s) dropped from context: {note}]"

The budget stays in characters here on purpose. The stronger version counts the real token budget of the target model; I noted that in the docstring rather than building it, because the point of this fix is the missing signal, not a better ruler.

Proof. Same input in both runs: four bulky decoy passages plus a short answer chunk that is retrieved but ranked last, so the joined context is 7343 characters against a 6000 budget. Real console output, captured with the repo's own interpreter, no API key.

Before, the answer chunk is gone and nothing says so:

MAX_CONTEXT_CHARS      = 6000
full joined context len= 7343
answer chunk id        = expenses::answer
answer text present in FULL joined context? True
answer text present in BUILT prompt?        False
answer sentence in built prompt?            False
------------------------------------------------------------
RESULT: answer chunk SILENTLY DROPPED. No exception, no warning, no log. Caller gets a prompt missing the answer.

After, packing skips the one passage that will not fit, keeps the small answer chunk behind it, and reports the id it dropped:

context truncated: 1 passage(s) dropped to fit 6000-char budget: expenses::3
MAX_CONTEXT_CHARS      = 6000
full joined context len= 7343
answer chunk id        = expenses::answer
dropped_ids reported   = ['expenses::3']
answer text present in BUILT prompt?        True
answer sentence in built prompt?            True
------------------------------------------------------------
  expenses::0        -> KEPT (verbatim)
  expenses::1        -> KEPT (verbatim)
  expenses::2        -> KEPT (verbatim)
  expenses::3        -> dropped whole
  expenses::answer   -> KEPT (verbatim)
------------------------------------------------------------
RESULT: answer PACKED IN, and the passage that did not fit (['expenses::3']) is reported to the caller. No silent loss.

Four behavioural tests lock the contract: an over-budget answer chunk is either packed in or named in dropped_ids and never silently gone, over-budget context reports its dropped ids, no passage is ever cut mid-text, and within-budget context drops nothing. A fifth pins the tuple return type so a caller cannot regress to the old string signature. All five run with the repo's own interpreter, no key and no network.

The rule I kept: any place that decides what the model sees has to announce what it dropped. A retriever that finds the right passage and an assembler that quietly deletes it produce the same wrong answer, but only one of them leaves a trace. The output truncation in this file already knew that. The input side just needed to learn the same manners.

Best Use of Sentry

The fix for "silent" is instrumentation. I added env-gated Sentry spans across the pipeline (off unless SENTRY_DSN is set and sentry-sdk is installed, so the repo never hard-depends on it). The span that matters here is rag.build_prompt:

total_context_chars and max_context_chars
n_dropped and dropped_ids
a context.truncated_chars measurement
a warning-level event whenever n_dropped > 0

Now a wrong answer on a long-context request is not a mystery. The pipeline is instrumented end to end: rag.answer to rag.retrieve (hit_ids) to rag.build_prompt (dropped_ids) to rag.generate (stop_reason). The trace below exercises the assembly step directly, so it shows rag.answer to rag.build_prompt, where the truncation happens and the dropped ids surface.

On the over-budget request, the rag.build_prompt span reads total_context_chars 7343 against a max_context_chars of 6000, a context.truncated_chars measurement of 1343, and the dropped_ids, sitting directly under the rag.answer transaction. The truncation that used to be invisible is now a value on a span.

Best Use of Google AI

To make the localization repeatable I added scripts/root_cause.py: it takes a Sentry trace plus the source files that ran, and asks Gemini to point at the line. The prompt is fixed and paste-ready:

You are debugging a RAG pipeline that returned a wrong answer. Here is a Sentry trace and the relevant source. The retrieval span shows the correct passage was retrieved, but the final answer is wrong. Localize the exact line where the retrieved context is lost, explain the mechanism, and note whether the loss is signalled to the caller. Then propose a minimal fix.

The script is env-gated on GOOGLE_API_KEY or GEMINI_API_KEY and fails with a clear message when no key is set, so it never runs blind.

Run against the pre-fix source with the real trace, on gemini-2.5-flash, it returned this (verbatim):

The Sentry trace clearly indicates that the rag.retrieve span successfully identified expenses::answer as a relevant passage. However, the subsequent rag.build_prompt span shows a total_context_chars of 7343 and a max_context_chars of 6000, leading to 1343 characters being truncated. The symptom confirms that expenses::answer was "absent from the assembled prompt," which directly led to the LLM's abstention (I could not find this in the provided documents.).

1. Exact Line Where Context is Lost. The context is lost in app/generate.py, specifically at this line within the build_prompt function: context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS].

2. Mechanism of Loss: naive string slicing after concatenation. The function joins all blocks into a single long string, then applies a hard character limit by slicing that entire concatenated string. If the total length exceeds config.MAX_CONTEXT_CHARS, the slice cuts characters from the end of the combined string. Passages later in the hits list are more likely to be truncated, and even a highly relevant passage like expenses::answer can be cut off if it falls past the character limit.

3. Signalling of Loss. "No exception, warning, or dropped-id was reported to the caller." The function simply returns the truncated string; there are no checks, logging.warning calls, or exceptions that would inform the calling Generator.answer method. Sentry captured the context.truncated_chars measurement, but that is instrumentation-level reporting, not a signal within the application's business logic.

4. Minimal Fix. Adjust build_prompt to iteratively add passages, respecting config.MAX_CONTEXT_CHARS without splitting individual passages, so higher-ranked complete passages are prioritized. [It then produced a pack-to-fit loop that stops adding blocks once the budget is reached.]

The model landed on the same line I did, read the mechanism straight off the trace measurements, and proposed the same pack-to-fit shape as the fix above. Sentry said which chunk vanished and by how much; Gemini said which line threw it away and why nothing noticed.

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

The night I leaked my own payment key, and what caught it

Vinicius Pereira — Tue, 14 Jul 2026 21:30:52 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I build a SaaS for small businesses. Restaurants, small shops, the kind of owner who needs to track sales, stock, cash flow and finances without wrestling a heavyweight ERP. For the first months I was heads-down on features, because that is what pays: the product has to solve a real problem the day you ship it.

Then one night taught me something no feature ever could.

The mistake was entirely mine

During a deploy, a configuration of my own doing exposed a payment API key in production. The exposure window was short, but the risk was not theoretical. Anyone who found that credential could try to use it. In production. Against real money.

Here is the part I got right, and it was as much luck as foresight: weeks earlier I had built an emergency mechanism for exactly the kind of unlikely event I did not believe would ever happen to me.

Every payment integration ran behind a monitor that watched for abnormal usage. A sudden jump in transaction volume. Calls from origins it did not recognize. Sharp changes in behavior. Anything outside the profile it expected. If any of those tripped, the integration did not wait for a human. It moved itself into a protection mode, blocked new operations, and paged me immediately.

That is exactly what happened.

The circuit breaker did its job

The monitor flagged the anomaly and locked the integration into protection mode before a single improper transaction went through. The alert woke me in the middle of the night. I revoked the compromised credential, generated a new key, rolled it across the whole infrastructure, and validated everything before the first customer opened the app in the morning.

No customer was affected. No improper transaction went through. No financial data was exposed.

And none of that made the mistake any smaller.

What I actually learned

The lesson was not "be more careful." Careful is not a strategy. I had been careful, and I still shipped the mistake at the worst possible layer.

The real lesson was that security cannot rest on not making mistakes. It has to assume mistakes will happen, and it has to cap the blast radius when they do. The thing that saved me that night was not my discipline. It was a layer I had built specifically to survive my own discipline failing.

So I rebuilt the security architecture around that idea:

Automated rotation of sensitive credentials, so a leaked key has a short life by default.
Hard segregation between development and production, so the two can never bleed into each other.
Detailed audit logging of critical events, so every sensitive action leaves a trail.
Extra validation during deploy to stop secrets from being exposed in the first place.
Continuous monitoring of every external integration.
Real-time alerts for anything that looks abnormal.
A mandatory security review before any release reaches production.

The question I ask before I ship anything now

Every new feature I build now starts from one question:

If I make a mistake tomorrow at 3 a.m., will the system protect my customers before they notice?

That night changed what engineering means to me. Building is no longer just shipping features. It is building systems that expect human failure and protect the people who trust them anyway.

The bug that night was mine. The thing that smashed it was a layer I had built for a version of me that would eventually slip. Build that layer before you need it. You will need it.

My weekend challenge entry: an AI announcer for the beach-sport duels my futevôlei crew plays every day. Gemini writes the call, ElevenLabs voices it. Ring the bell yourself, the demo is live.

Vinicius Pereira — Fri, 10 Jul 2026 12:10:16 +0000

DEV Weekend Challenge: Passion Edition Submission

Vinicius Pereira

Jul 10

I gave my futevôlei crew an AI announcer (DuelUp Live)

#devchallenge #weekendchallenge #ai #googleaichallenge

4 min read

I gave my futevôlei crew an AI announcer (DuelUp Live)

Vinicius Pereira — Fri, 10 Jul 2026 09:52:42 +0000

This is a submission for the Weekend Challenge: Passion Edition

What I Built

Every Sunday my futevôlei crew meets on the sand in Niterói, Brazil. The rivalries are real: there are rematches people talk about all week, comebacks nobody is allowed to forget, and at least one duel per month that ends with someone buying açaí for everybody.

I am so deep into this passion that I have been building a whole platform for it: DuelUp, a competitive system for amateur beach sports, with ELO ratings, rank tiers from Sand to Legend, and virtual-coin stakes on real matches. It is not launched yet, but the crew already lives by its rules.

This weekend I built the piece every real rivalry deserves and none of us had: an announcer.

DuelUp Live is a fight bill for beach-sport duels. You pick the sport (footvolley, beach tennis, tennis or beach volleyball), write in the two sides with their nicknames and ranks, set the moment ("17-16, match point, blazing sun") and the bad blood ("rematch after last Sunday's comeback"). Then you ring the bell, and an electrifying AI sports announcer calls the match live, in English or in Portuguese with a Brazilian announcer voice, the kind we grew up hearing on the radio.

Gemini writes the play-by-play. ElevenLabs gives it the voice. The rank tiers come straight from DuelUp's real ELO system, and the announcer uses them as drama: when Silver faces Gold, you get an underdog story.

Demo

live.duelup.app

No signup, nothing to install. Click ⚡ Example to load a ready duel (each sport has three, they cycle), then 🔔 RING THE BELL.

Two things worth knowing while you try it:

Every ring is a fresh call, written and voiced on the spot. Ring the bell twice on the same duel and you will never hear the same broadcast.
It takes about 20 seconds, because it genuinely is an LLM writing a script and a TTS model performing it, live.

Code

github.com/vinimabreu/duelup-live

Next.js 14, App Router, zero extra runtime dependencies. Two API routes, one for the writing, one for the voice, both plain REST calls with no SDKs.

How I Built It

The writing (Gemini). The announcer is gemini-flash-latest with a prompt that treats narration as radio, not prose: present tense, short explosive sentences, 90 to 120 words, written for the ear because a TTS voice will perform it. Two decisions mattered more than the prompt itself:

Thinking disabled (thinkingBudget: 0). An announcer should not overthink. Without this, the model's reasoning tokens were eating the output budget and truncating calls mid-sentence. With it, calls come back complete and fast.
A model cascade. If gemini-flash-latest returns a 503 under load, the route falls through to gemini-flash-lite-latest, then gemini-2.0-flash. The demo has already survived a real Google traffic spike this way. A demo that only works when the weather is nice is not a demo.

Temperature is 1.0, which is why no two calls are alike. The prompt is also sport-aware: tennis gets called on the court, everything else on the sand, and the rank tiers get woven in as favorite-versus-underdog tension.

The voice (ElevenLabs). eleven_multilingual_v2 with stability set low, because announcers are not stable people. Portuguese calls go to Eduardo, a Brazilian voice that sounds like he has narrated a thousand matches. English calls go to Charlie, who brings the energy. The waveform player on the bill is wired to the real audio element: real progress, real duration, and the bars dance while he shouts.

The bill. The design is a vintage fight poster: paper texture, ink borders, Anton for the headline type, a black promoter plate with the real DuelUp logo, "TALE OF THE TAPE" for the context fields, and the rank badges from the actual game next to each fighter. The whole UI switches between English and Portuguese, announcer included.

Prize Categories

Best use of Google AI and Best use of ElevenLabs. They are not decorations here, they are the two halves of the product: Gemini is the announcer's brain, ElevenLabs is his lungs.

One personal note. Futevôlei is not a theme I picked for a challenge. It is the sport I play every single day, and the crew in this post is my real crew. Getting to spend a weekend building for it, and then writing about it here, was the rare kind of work that does not feel like work.

The platform came first, but the announcer went live before the platform did. That feels right. Passion projects do not follow roadmaps.

Built by Vinicius Pereira · vinimabreu.dev · github.com/vinimabreu

Lead Quorum: a multi-agent lead qualifier that refuses to guess (ADK + A2A)

Vinicius Pereira — Wed, 08 Jul 2026 07:35:25 +0000

This post is my submission for DEV Education Track: Build Multi-Agent Systems with ADK.

What I Built

Most lead scoring demos share a failure mode: you paste messy notes, a model returns a confident number, and nobody can answer why it is a 60 and not a 40. I built the opposite. Lead Quorum is a distributed multi-agent qualifier where the number is set by deterministic code, the explanation provably adds up to the score, and when two independent readers disagree about the input, the system abstains instead of guessing.

Three production failures it kills by construction:

Opaque scores. Every point is granted and explained on the same line of code, and a test parses the explanation and asserts the named points sum to the score. The reason cannot drift from the number.
One model grading its own homework. Two readers running two different Gemini models extract the same lead independently, as separate services. Agreement between different models is real signal, not a model agreeing with itself.
Fake confidence on thin input. "They mentioned possibly renewing" should not score like "they renewed in March."

What a run looks like. Clear notes, both models agree:

CONFIRMED  60/100: +35 team of 30 seats >= 25; +15 reachable decision maker (vp);
           +10 renewed at least once

Ambiguous notes, the models read "possibly renewing" differently:

EXCLUDED   readings disagree on which signals fire (prior_relationship: False vs True);
           abstaining instead of scoring a contradiction

That EXCLUDED is the feature. The score would differ depending on which reading you believe, so the honest output is no score, plus exactly what disagreed. A defensible abstention beats a fake-precise number built on a contradiction.

Cloud Run Embed

Open it, paste your own messy lead notes, and watch the audit trail: the score, the reason that reconciles to it, the two independent readings side by side, and the EXCLUDED abstention when they disagree about which rules fire. The two readers are separate Cloud Run services the orchestrator reaches over A2A.

Your Agents

Five roles. Two are LLMs, three are deterministic code, and that split is the design.

Enrichment reader (LlmAgent, gemini-flash-latest): extracts structured fields from raw notes. Temperature 0, pinned output schema, instructed to leave absent signals at their defaults instead of guessing.
Rederive reader (LlmAgent, gemini-2.5-flash-lite): reads the same notes from scratch, independently, on a deliberately different model. Runs as its own microservice, exposed over the A2A protocol with ADK's to_a2a() and consumed through its agent card with RemoteA2aAgent.
Scoring agent (custom BaseAgent, no LLM): applies the rubric. Each rule grants its points and writes its reason in the same branch, and the agent refuses to emit a result whose reason does not reconcile to its score.
Corroboration agent (custom BaseAgent, no LLM): compares the two readings in score-space. Same rules fire, values close: CONFIRMED. Same rules fire, readings drift: REVIEW, score stands, flagged. A rule flips between readings: EXCLUDED, no score, verdict names the flipped rule and both values.
Orchestrator (SequentialAgent + ParallelAgent): the two readers run concurrently, so wall-clock is one LLM round-trip, then scoring and corroboration run as pure code. Exactly two LLM calls per lead.

The readers deploy as independent Cloud Run services and the orchestrator reaches them over A2A, so the second opinion could be swapped for a different vendor or framework tomorrow without touching the pipeline.

Key Learnings

Put the LLM only where judgment lives. Multi-agent systems get expensive when every step is a model call. Two parallel calls per lead, everything downstream deterministic and unit-tested, made the system cheaper per unit of trust, not pricier.
Score-space beats value-space for corroboration. Two readings 500 dollars apart on the same side of a threshold produce the same score; that is drift, not contradiction. Comparing which rules fire, instead of raw field equality, lets abstention trigger only when the disagreement actually changes the answer.
Reconciliation has to be enforced, not intended. "Keep the reason next to the points" is a discipline, and disciplines rot. A failing test does not.
A2A is what makes independence credible. Behind an agent card, the second reader is a black box that could be any model, any framework, anywhere. That is the difference between a second opinion and an echo.
ADK's workflow agents are underrated. SequentialAgent and ParallelAgent gave me a deterministic, testable topology with no LLM routing where none was needed. The surprise of the build: RemoteA2aAgent has no output_key, so remote responses land in the event log, and a small capture adapter was the missing piece to keep the distributed pipeline identical to the local one.

Code: github.com/vinimabreu/lead-quorum, MIT, 16 tests, including a script that runs the full distributed topology locally.

Your LLM bill has two sides. Build the ledger that shows both.

Vinicius Pereira — Tue, 07 Jul 2026 20:57:08 +0000

Every RAG cost estimate starts the same way: input tokens equal top_k times chunk size, plus some overhead. Most of them stop there too. Then the invoice arrives, it is three times the estimate, and the team spends a sprint tuning chunk sizes while the actual money leaks somewhere that formula never touches.

The bill for an LLM system is set by total input and output tokens, summed across every call a query triggers. Not per call. Not input only. Every call, both directions. If you only remember one sentence from this post, that is the one.

Why chunk math lies

Four leaks, in the order I usually find them in real pipelines:

Calls you forgot exist. A query condenser rewriting the user question. A silent SDK retry. A JSON repair call after a malformed response. An agent loop that took three turns instead of one. Each is a full-priced API call that no chunk formula predicts, and some frameworks make them without telling you.

Scaffolding re-sent on every call. System prompt, tool schemas, formatting instructions, few-shot examples. This fixed overhead rides along on every single call, and in multi-call pipelines it often outweighs the retrieved chunks you are carefully tuning.

The output side, billed at a premium. Output tokens cost several times input on every major provider's list price, five times on current Claude models. A dashboard that tracks input tokens is watching the cheap half of the transaction.

Multi-call synthesis re-billing its own draft. Refine-style chains regenerate the full running answer at every step. Only the final draft is delivered, but all N drafts are billed at output rates, and each discarded draft is re-sent as input to the next call. The answer component of your cost scales with N, and no chunk tuning touches it. Prompt caching does not rescue it either: the growing answer sits mid-prompt, which breaks the prefix match.

Stop counting tokens. Start reading them.

The fix is not a better estimate. It is refusing to estimate: every provider already returns exact usage on every response, so the whole job is logging what the API tells you and adding it up per query.

Where you hook matters more than it looks:

The SDK call site is the reliable choke point. A thin wrapper around each create() sees server-reported usage for every call you make, streaming or not.
Framework callbacks are leakier. Some read server-reported usage properly and propagate through nested calls (LangChain's usage metadata callback does). Others estimate with a local tokenizer instead of reading the wire, LlamaIndex's TokenCountingHandler being the documented example. And none of them see calls made outside the framework.
HTTP-layer hooks have a streaming blind spot. An httpx response hook fires when headers arrive, before the body. For non-streaming calls you can read the body inside the hook; for SSE streams the usage arrives at the end of a body the hook cannot safely consume. If you want wire-level capture for streams, you need a logging proxy or a transport wrapper that tees the stream.

For most codebases the call-site wrapper is one afternoon of work and covers everything that matters.

The ledger

Columns that earn their keep, learned from what actually gets queried later:

column	why it exists
query_id	ties every call in one user request together; propagate it with a contextvar
call_index	0, 1, 2... within the query; the column that exposes hidden calls
model	mixed-model pipelines bill at mixed rates
input_tokens, output_tokens	the two sides of the bill, server-reported
cache_read, cache_write	cached input bills at its own rates; on Anthropic these are two separate fields billed differently
stop_reason	a truncated response often triggers a retry you will want to find
latency_ms	free to record, and cost and latency investigations are usually the same investigation
stage	"condense", "retrieve-answer", "repair"; the label that turns rows into a story

A minimal implementation, SQLite plus a contextvar:

import sqlite3, time, uuid
from contextvars import ContextVar

query_id: ContextVar[str] = ContextVar("query_id", default="")

db = sqlite3.connect("tokens.db")
db.execute("""CREATE TABLE IF NOT EXISTS calls(
    query_id TEXT, call_index INT, model TEXT,
    input_tokens INT, output_tokens INT,
    cache_read INT, cache_write INT,
    stop_reason TEXT, latency_ms INT, stage TEXT)""")

def record(model, usage, stop_reason, latency_ms, stage=""):
    qid = query_id.get() or str(uuid.uuid4())
    idx = db.execute(
        "SELECT COUNT(*) FROM calls WHERE query_id=?", (qid,)
    ).fetchone()[0]
    db.execute("INSERT INTO calls VALUES (?,?,?,?,?,?,?,?,?,?)",
        (qid, idx, model,
         usage["input"], usage["output"],
         usage.get("cache_read", 0), usage.get("cache_write", 0),
         stop_reason, latency_ms, stage))
    db.commit()

Wrapping a non-streaming Anthropic call:

import anthropic, time

client = anthropic.Anthropic()

def chat(stage="", **kwargs):
    t0 = time.monotonic()
    resp = client.messages.create(**kwargs)
    u = resp.usage
    record(
        model=resp.model,
        usage={
            "input": u.input_tokens,
            "output": u.output_tokens,
            "cache_read": getattr(u, "cache_read_input_tokens", 0) or 0,
            "cache_write": getattr(u, "cache_creation_input_tokens", 0) or 0,
        },
        stop_reason=resp.stop_reason,
        latency_ms=int((time.monotonic() - t0) * 1000),
        stage=stage,
    )
    return resp

Set the contextvar once at the top of each user request (query_id.set(str(uuid.uuid4()))), pass a stage label at each call site, and the ledger fills itself.

The streaming gotchas

Streaming is where naive capture quietly loses data, and the two big providers fail in different directions:

OpenAI only emits usage on a streamed call if you ask for it: pass stream_options={"include_usage": True}. The usage then arrives on the final chunk, with every earlier chunk carrying usage=None. Forget the flag and the ledger records nothing while the meter runs.

Anthropic splits usage across the stream: message_start carries the input side, and the output count arrives via message_delta near the end. If you use the SDK's streaming helper, the accumulated final message carries complete usage, so stream.get_final_message().usage is the easy path.

Field names do not agree across providers. prompt_tokens versus input_tokens, cached tokens under prompt_tokens_details.cached_tokens on one API and cache_read_input_tokens on another. And some self-hosted OpenAI-compatible servers omit usage entirely. Normalize into your own schema at write time, not at query time.

What the first week shows you

Having turned this on in production pipelines, the surprises arrive in a reliable order.

First surprise: call count. A rewrite step someone added in March, a retry that fires on every truncated response, an agent loop budgeted for one turn that averages 2.4. GROUP BY query_id and the queries with eight rows instead of two are your bill.

Second: the scaffolding. The fixed prompt overhead, multiplied by the call count you just discovered, frequently outweighs the retrieved context. People tune the 2,000 variable tokens and ignore the 3,500 fixed ones riding on every call.

Third, and only third: chunk size and top_k, the things everyone tunes first.

The ledger also makes redundancy visible. If your corpus has near-duplicate documents, they show up as repeating, near-identical input deltas across queries: you are paying to send the model the same paragraphs again and again, and the fix (dedup before the context window) is a quality fix that happens to cut the bill.

The whole analysis, honestly, is one query:

SELECT query_id, COUNT(*) AS calls,
       SUM(input_tokens) AS input,
       SUM(output_tokens) AS output,
       SUM(input_tokens + 5 * output_tokens) AS cost_proxy
FROM calls
GROUP BY query_id
ORDER BY cost_proxy DESC LIMIT 20;

The 5 is a list-price output-to-input ratio; swap in your provider's. The top 20 rows of that query are worth more than any cost-optimization blog post, including this one.

The companion package is on GitHub and PyPI: token-ledger is exactly this ledger, packaged. pip install llm-token-ledger, zero dependencies including the dashboard, one afternoon.

Sid Probstein made the argument from the architecture side this week, comparing what RAG stacks actually send across frameworks, and his closing line is the right summary of the whole subject: that number, not top_k, is your bill.

DEV Community: Vinicius Pereira

Python's casefold() merged two of my customers into one tenant

The line that was supposed to be boring

The tenant that was two tenants

Why this is worse than a normal access bug

Then the same bug again, wearing different clothes

Two fixes I tried and threw away

The fix

The general shape of it

Why the repo concentrates risk there on purpose

My exactly-once refund paid twice, and 373 passing tests could never have caught it

The claim

The bug

Why my tests were structurally incapable of finding it

The second thing it broke: a metric that was measuring the wrong noun

What the repo does now

vinimabreu / langgraph-production

A LangGraph support agent with the reliability layer around it: routing measured against labelled fixtures, a human gate before irreversible tools, durable resume with exactly-once effects, and a CI gate that fails when a number moves.

langgraph-production

The report this repo exists to produce

The takeaway I did not expect

The ChocoDEV Signature Bar: one chocolate bar, zero images

Inspiration

Demo

Journey

ChocoDEV: a chocolate bar you can eat, in pure CSS

What I Built

Demo

Journey

Everything that broke when I imported hand-written Make blueprints into a real workspace

1. Import is a parser, not a validator

2. {{1.body}} does not exist, and the failure mode is silence

The ten lines that ended the guessing

3. Collections do not serialize into text fields

4. The default schedule is a 15-minute tick

What the wire showed when it all worked

The outage I did not schedule

What this does not tell you

Your LLM's confidence score is lying to you

Similarity is not confidence

Compute it from what you can check

The gate, as code

Tune on the cost of a false accept

What this does not do

Clear the Lineup: the passage my RAG retrieved and then threw away in silence

Project Overview

Bug Fix or Performance Improvement

Code

Fix silent context truncation in build_prompt #1

My Improvements

Best Use of Sentry

Best Use of Google AI

The night I leaked my own payment key, and what caught it

The mistake was entirely mine

The circuit breaker did its job

What I actually learned

The question I ask before I ship anything now

My weekend challenge entry: an AI announcer for the beach-sport duels my futevôlei crew plays every day. Gemini writes the call, ElevenLabs voices it. Ring the bell yourself, the demo is live.

I gave my futevôlei crew an AI announcer (DuelUp Live)

I gave my futevôlei crew an AI announcer (DuelUp Live)

What I Built

Demo

Code

How I Built It

Prize Categories

Lead Quorum: a multi-agent lead qualifier that refuses to guess (ADK + A2A)

What I Built

Cloud Run Embed

Your Agents

Key Learnings

Your LLM bill has two sides. Build the ledger that shows both.

Why chunk math lies

Stop counting tokens. Start reading them.

The ledger

The streaming gotchas

What the first week shows you

2. `{{1.body}}` does not exist, and the failure mode is silence