DEV Community: Vinicius Pereira

Before you give: hear what a nonprofit's tax filings say, and what they can't

Vinicius Pereira — Sun, 06 Sep 2026 10:59:31 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

Type the name of a US nonprofit and hear, in a minute or two, what its public tax filings say: how much money moved, whether it spent more than it took in, how many months of reserves it holds, where the money comes from, and how that moved over the last decade. Then, in the same breath, what those filings can't tell you.

There's no score. That was the whole design decision.

Every charity rating site ends in a number, and the number has to weigh things the filing doesn't measure. A Form 990 records what came in and what went out. It doesn't record whether the work is any good, whether the reserves are locked to a purpose, or what happened after the fiscal year closed. A score built on it is a guess wearing a number, and a donor who trusts the number is trusting the guess.

So the unit here isn't a score. It's a fact with a limit attached:

Reserves about 12 months
Net assets of $3 billion would cover about 12 months of spending at the current pace.
limit: Net assets include buildings, equipment and gifts restricted to a purpose,
so the part it could actually spend is smaller. Treat this as an upper bound.

The limit is a required field. The code can't produce a fact without one.

The generosity angle is simple: give better. Most of us donate on impulse or on a friend's word. This gives you one minute of what's actually on file, in words you can take in on the way to deciding, and it's free, keeps no account, and rates nobody.

Demo

Live: https://before-you-give.vercel.app

Try "American Red Cross", then "Doctors Without Borders", then a small food pantry in your state. The three feel completely different once you hear them.

The voice runs on a spending cap I set for the weekend. If it tells you the budget is used up, the full narration is written out under "Read the narration" and everything else still works.

Code

vinimabreu / before-you-give

Hear what a nonprofit's public tax filings say before you donate. No score, no guessing.

before-you-give

Type the name of a US nonprofit and hear, in a minute or two, what its public tax filings say: how much money moved, whether it spent more than it took in, how many months of reserves it holds, where the money comes from, and how that has moved over a decade. Then, in the same breath, what those filings cannot tell you.

No score. No ranking. No language model writing the numbers.

Live: https://before-you-give.vercel.app

A real session on the live site: search, pick the organization, listen, follow the narration, scroll to the chart. The same clip as an MP4: assets/demo.mp4.

Screens

Landing	Reading

Year by year	Phone

Why there is no score

Every charity rating site ends in a number, and the number has to weigh things the filing does not measure. A Form 990 records what an organization took in and what it spent. It does…

View on GitHub

MIT. Python, FastAPI, one HTML page with no build step. 82 tests that run offline against real API responses, no key needed.

How I Built It

Data first, code second. Every number comes unchanged from the ProPublica Nonprofit Explorer API, which republishes IRS Form 990 extracts. Before writing any logic I pulled six real organizations and saved the responses as fixtures: a big 990 filer with 13 years of history, a private foundation (990-PF) that reports the same lines under different names, a food pantry on the short 990-EZ form, another on the full form, and one organization with no filings at all. The code was shaped by what the feed actually returns, not by what I assumed it would. Every field is optional and every fact only exists if its inputs do.

Facts with limits. reading.py is the whole arithmetic. A Fact has a display value, a plain sentence, the same sentence written for a voice, and a limit. The limit is what stops the page from lying by layout. When the feed doesn't carry the program / admin / fundraising split (it's on the filing PDF, not in the API), the page says so instead of pretending. When an organization has no numbers, the page explains that small organizations file a postcard and churches don't have to file at all, because absence of data is not a warning sign and a blank page would imply it is.

Two renderings of every number. $3.2 billion for the eye, 3.2 billion dollars for the voice. A text to speech engine reads $3.2B five different ways and none of them is what a listener wants. There's a test that walks every narration and asserts there is no $ and no % anywhere in it.

The narration is templates, not a language model. The script is assembled from the facts by plain string code, so it can't drift from the data even a little. It ends with the source line every time.

ElevenLabs, with the guardrails written before the first paid call. The narration renders with eleven_turbo_v2_5. Two rules sit in front of it:

cache by text: the mp3 is stored by the hash of the script, so the same organization never costs twice;
a hard cap on calls and characters, checked before every request. Past the cap the server answers 429 and keeps serving what's already cached. A small per-address limiter sits in front of that.

Without a key the app runs text-only and says so on the page. Nothing else changes.

On the voice itself: I spent part of this week tuning ElevenLabs settings on a different project, and the lesson carried over. The voice you pick matters less than stability and style. Defaults sound fast and flat; too much style and it starts oscillating between moods mid-sentence. For narration I landed on stability 0.6, style 0.2, speed 0.95, and it reads a balance sheet like a calm person, which is the point.

What the data told me that I didn't expect. The American Red Cross's fundraising events lost money in fiscal 2023: $3.1 million in, $4 million to run. My first version printed "kept -27%", which is true and useless. Now it says the events cost more than they raised, and the limit line notes that events often exist to build a community as much as to raise money. Also: a $3 billion organization holding about 12 months of reserves, and Doctors Without Borders USA running an $89 million deficit with 4 months of reserves and 97% of revenue from donations. None of that is a verdict. It's what's on file, and now you can hear it.

What I didn't build. No scraping of the filing PDFs for the expense split, no score, no "recommended" list. A weekend is enough to be honest about a data feed; it isn't enough to be right about a cause.

Prize Categories

Best use of ElevenLabs. The voice isn't decoration here, it's the accessibility layer. The people this is for are not the people who read walls of numbers. Every figure is formatted for speech, the script is deterministic so the audio can't say something the page doesn't show, and the cache and cap are what make it responsible to leave a public demo running on a personal key.

If you try it on a nonprofit you care about and the page gets something wrong, open an issue. The numbers are ProPublica's; the sentences are mine, and I'd rather fix a sentence than defend it.

A fake client sent me a GitHub repo. Running it cost me two days and every password I had.

Vinicius Pereira — Thu, 27 Aug 2026 18:45:33 +0000

This week a client on a freelancer platform sent me a repo and asked me to run it. I ran it. Two days later I had rotated every credential I own and wiped my machine.

I want to walk through exactly how it works, because it is not one clever trick, it is a whole family of them, and they are all aimed at us specifically. The people building these know that a developer will clone a repo and type npm run dev without thinking, the same way you would open a door for someone holding coffee. That reflex is the exploit.

How the main one works

The setup looks like a real job. A "client" posts something senior and well paid: a senior AI engineer role, a big fixed budget. They message you fast and warm. Then they share a GitHub repo. In my case it was an AI companion game, and it was genuinely well architected, clean separation, sensible structure, the kind of code that lowers your guard on purpose. They ask you to run it locally and send a screenshot proving it works, before any contract exists.

The malware was in postcss.config.js. Not in an obvious place, in a config file nobody reads, sitting on the same line as the real config behind a long run of spaces so it scrolls off the screen. It was obfuscated with fromCharCode so even if you looked you would see noise, not a URL. It fires the moment you run the dev server.

When it fires, it beacons out to a command-and-control server, pulls down a second stage, and goes straight for the browser. On a Mac it tries to read the "Chrome Safe Storage" key out of the keychain, which is the key that decrypts every password and cookie Chrome has saved. One keychain prompt, and if you approve it, the whole vault leaves the machine. To make the C2 hard to block, the address was not hardcoded, it was resolved through a dead-drop on the Ethereum blockchain, so a takedown or a blocklist does nothing.

This family has names if you want to read more: Contagious Interview, BeaverTail, InvisibleFerret. It is run at scale and it targets developers through job offers, on purpose, because our machines hold SSH keys, cloud credentials, and source access.

The part that actually scared me came that night. I found that it had not just run and exited. It had rewritten main.js inside VS Code, cli.js inside npm, and index.js inside Discord, three Electron apps whose code is plain editable JavaScript on disk. The npm one went from 407 bytes to 1.7 MB. Every time I opened any of those apps, or ran any npm command, it respawned the password stealer and a clipboard monitor. A reboot does nothing against that. "No LaunchAgent, no cron" is not the same as "no persistence."

The other flavors

The repo trick is the loud one. There are quieter versions arriving the same week, and I have watched every one of these land in my own feed in the last few days.

The malicious attachment. The job post has a .zip or .rar attached, labeled something like "requirements" or "project brief." Inside, a couple of harmless PNGs as decoy and one .vbs or .bat file. That script downloads the real payload from a hacked WordPress site and runs it. On a Mac a .vbs will not execute, but the file still travels, and if you also use Windows it will fire there.

The compromised real account. You check the client and it looks fine. Payment verified, phone verified, a five-star review, spending history, an account from years ago. Then it sends you a repo or an attachment with malware in it. Old, legitimate accounts get taken over and used as bait precisely because they pass the naive filter. Judge the message and the attachment, never the profile badge.

The off-platform pull. A warm reply to your proposal asking for your email so they can send the "official interview invitation," and often a tip to write your email with spaces so the platform's filter does not catch it. No real client ever coaches you to defeat the contact filter. That instruction alone is the whole answer.

The pay-to-test scam. A tester or QA role that, once you accept, needs you to fund a crypto wallet or a subscription "for payment testing, reimbursed separately." Crypto is irreversible and the reimbursement never comes. Real payment testing uses sandbox keys, never your own money.

Why the usual signals fail you here

The instinct is to trust the platform's checkmarks. Payment verified, top rated, high spend. Those tell you the account can pay and has a history. They tell you nothing about whether the file in front of you is safe, and they are exactly what a hijacked account still shows. The verification protects you against not getting paid. It does not protect you against running code.

The payment model itself kills the excuse. These platforms already hold the client's money and release it only when they approve your work, so a nervous client never needs to watch your machine run their project first. When someone insists you run their code and screenshot it before there is any contract at all, that is not caution and it is not a technical test. There is no legitimate reason for it. It is the attack.

The rules I follow now, no exceptions

Never run a stranger's code. Not a repo, not a script, not a notebook, not a "quick test task." Reading it is fine. Running it is the line, and it does not move for a good story or a big budget.
Read third-party code by view only. Read it raw over HTTP, no clone, no npm install, no pip install, no docker compose. And read the boring files first, because that is where it hides: config files like postcss, webpack, vite, the scripts block in package.json (watch postinstall and prepare), the Dockerfile, git hooks. Malware lives in the file you skip.
Treat attachments as text. Never double-click. A .docx or .xlsx is a zip, so unzip it and read the XML. A .zip or .rar, list the contents before extracting, and any .vbs, .bat, .js, .lnk, .scr, .hta, or executable inside means you stop there. Never go fetch the second stage to "see what it is." That just hands the attacker your IP and runs live malware.
A funded contract does not make their code safe to run. This is the bypass a scammer will reach for next, so be clear about it: a paid contract changes who is liable for the work, it does not change what the code does to your machine. If you ever genuinely have to run something unknown, it goes in a throwaway VM with no network and no access to your real files, and even a clean run there proves nothing, because some of this only fires on the right OS and some detects a VM. Reading the code is the only real check. Everything else is a second layer.
Stop saving passwords in the browser. That single keychain prompt is what turns one mistake into fourteen stolen logins. Use a password manager with its own master password, turn on 2FA everywhere, add passkeys where you can.
If it fires, assume the worst and wipe. Persistence hides in Electron apps and in npm, not only in the places a checklist looks. A reboot is not cleanup. If you cannot prove the machine is clean, you cannot trust it. I could not, so I formatted.

The uncomfortable part

The platforms are not catching these fast enough. I reported the job that got me, and the first reply said there was nothing wrong with it. The post came down a day later, from a different team, which tells you the front line and the safety team are not the same speed. You cannot outsource this to the marketplace. You are the last check.

The good news is that the defense is cheap. Skipping a scam costs you one lead you were never going to close anyway. Running one costs you your machine and days of your life. I know which trade I want now.

Be careful out there. If a client ever needs you to run their code before they will hire you, they are not hiring you.

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

The third identity merge wasn't Unicode. It was a placeholder.

Vinicius Pereira — Tue, 25 Aug 2026 22:19:50 +0000

Ann and Bob don't exist. They're two synthetic strangers in the test workspace of a CRM bridge I was about to publish. Ann is ann@x.example, Bob is bob@x.example, and both typed N/A into the phone field of a lead form, because that's what people type into a required phone field they'd rather not fill. My fake CRM merged them. Bob's upsert came back carrying Ann's contact id with Ann's conversation history hanging off it, and the audit ledger recorded contact_created twice. Two people created, one person stored, and the second creation line quietly naming the first one's id.

I caught it in a pre-publish review pass, not in production, which is the only comfortable part of this story. It's the third time in a few months I've hit the same family of bug, and this one stings in a specific way: it walked past a doctrine I had already written down twice.

The first incident was production. Python's casefold() merged two of my customers into one tenant: Unicode case folding is many-to-one, U+212A KELVIN SIGN folds to k, and two distinct identifiers became one entitlement key. The second was last week. My new repo argued against the exact bug it was carrying: the package preached ASCII-only folding while its own test fakes normalised emails with str.lower(). The doctrine existed; a caller just didn't call it.

This third one is worse in an instructive way, because the doctrine was right in both places you'd think to look.

The repo, for context

ghl-bridge is a policy-gated bridge between a GoHighLevel location and whatever generates reply drafts. Webhook in, dedupe, then a gate that only auto-sends a message when every policy passes, idempotency keyed on the event rather than the delivery, and a guard that raises if anything unapproved tries to leave. 379 tests, all offline, no account and no API key behind any of it.

Identity in this package follows two rules I've now paid for personally. Email folds ASCII case only, because str.lower() and str.casefold() apply Unicode tables that merge characters which were never case variants of each other. And phone is deterministic E.164 or an explicit refusal: a bare national number resolves only through the location's configured region, and when the rules can't determine the answer, the result is a PhoneNeedsReview value carrying the raw input and a named reason. A value, not an exception. The caller has to route it to a human queue on purpose, because guessing a country code merges strangers.

Both rules held. normalise_email was clean. normalise_phone was clean. Every caller I'd checked called them.

The door nobody was watching

The fake workspace models the platform's documented upsert semantics: within a location, an incoming contact that matches on email or phone updates the existing record instead of creating a new one. To do that it keeps an email index and a phone index, and before the review pass, its phone key helper had a fallback:

def _phone_key(self, state: _LocationState, raw: str) -> str:
    result = normalise_phone(raw, default_region=state.location.default_region)
    if isinstance(result, NormalisedPhone):
        return result.e164
    return raw.strip()

Look at how reasonable it is. The doctrine function is right there, called first. The fallback only fires for values the normaliser refused, and it just keeps them, tidied up, so the index stays total. Every visible key path in the package was doctrine-clean, and then the else branch of one private helper inside the test fake overruled the whole argument.

Run the sequence and the consequences arrive in order. Ann upserts with phone N/A; the normaliser refuses; the index files her under the literal string N/A. Bob arrives an hour later with the same placeholder; his email misses; his phone "matches". Merge. From then on, every lead whose phone field says N/A chains into that same contact record, which keeps getting fatter. Meanwhile the deduper, which computes its own doctrine-clean keys, saw nothing to match for Bob, so it logged contact_created while the store underneath merged him away. The audit trail and the data disagreed. In a real CRM this is one stranger reading another stranger's conversation history.

And the detail that made me laugh, the bitter kind: raw.strip() is the Unicode strip. The package's own identity.py defines a six-character ASCII whitespace constant precisely because bare str.strip() trims things like U+3000 IDEOGRAPHIC SPACE and merges identifiers that differ only there. So ext-12 and ext-12 followed by an ideographic space also folded onto one index key. The fallback didn't merely bypass the phone doctrine. It committed the exact sin from the email doctrine on its way past.

The fix is structural, and it's None

def _phone_key(self, state: _LocationState, raw: str) -> str | None:
    """The E.164 dedupe key, or None when the number does not determine
    one. An unresolvable phone is stored as a field but never indexed:
    indexing the raw string would merge two strangers who both typed
    "N/A" into the form, and trimming it with ``str.strip`` would break
    the ASCII-only doctrine everything else keys on. No key, no match,
    no merge."""
    result = normalise_phone(raw, default_region=state.location.default_region)
    if isinstance(result, NormalisedPhone):
        return result.e164
    return None

The callers grew None checks: indexing skips it, phone search returns empty. The raw value still gets stored on the contact as a field, so a human reading the record sees exactly what arrived; it just never becomes a match key. Three tests now pin the behaviour: test_two_strangers_with_the_same_unparseable_phone_do_not_merge is the Ann and Bob story, test_an_unparseable_phone_is_stored_as_a_field_but_never_indexed pins the stored-but-never-keyed split, and test_the_ascii_trim_doctrine_holds_for_phone_indexing_too carries a literal U+3000 in its source. You can't see the character, which is exactly the point.

Two lessons worth carrying out of the repo, because both apply anywhere identity gets deduped.

A normalisation failure is not an identity. When the normaliser says "I don't know", indexing the raw value turns every placeholder into a master key: every N/A equals every other N/A, every none, every -, every xxx. This isn't a CRM quirk. Any dedupe, ETL join, or entity-resolution pass keyed on a column that contains placeholders has this failure mode, and the placeholder density of a phone column is never zero. The irresolvable key has to index as nothing at all.

One rule, and it's only as strong as its least visible door. Three incidents, one shape: a many-to-one transformation over identity is a merge, and a merge is a grant. In the tenant incident the grant was entitlements. In the on-behalf repo it would have been document access. Here it's two strangers becoming one contact with one shared history. What moved between the incidents is where the door was. In incident two, the doctrine existed and a caller didn't call it. In incident three, the callers all called it, and the leak was the fallback path inside one of them, in a test fake, the component everyone extends and nobody audits because "it's just the fake". The fake is the thing your entire suite treats as ground truth. A doctrine that stops at the fake's front door is a lobby sign.

What the fix does not fix

A placeholder that happens to parse walks straight through the front door: 5555555555 under a US-region location resolves to a legitimate-looking E.164 and mints a real key, so two strangers typing it will still merge. Refusing the unparseable stops junk from acting as a wildcard; it does not detect junk. Catching parseable junk needs a known-junk list or an alarm on keys that match suspiciously often, and this repo ships neither yet. Second, the fake is my model of the documented upsert semantics; whether the live platform's own server-side matching indexes raw strings is exactly the kind of thing the RUNBOOK says to verify against a real workspace before trusting anything. Third, the fix has a stated cost: an N/A lead with an unknown email now creates a fresh contact every time. That's deliberate. It fails toward duplicates a human can merge instead of merges nobody can split, but your review queue will feel the difference.

The doctrine, the fake, the deduper and the three anchor tests are all in github.com/vinimabreu/ghl-bridge if you want to run the Ann and Bob case yourself.

A key you couldn't compute is not a key to tidy up. It's no key at all.

My new repo argued against the exact bug it was carrying

Vinicius Pereira — Tue, 25 Aug 2026 15:52:54 +0000

Not long ago, Python's casefold() merged two of my customers into one tenant. I wrote that incident up in its own post, and the rule I carried out of it felt permanent: when a normalized value becomes a security key, the normalization is attack surface. Any many-to-one transform over an identity is a merge, and in an entitlement check a merge is a grant.

This month I finished the sequel to that repo. on-behalf is permission-aware retrieval for RAG over SharePoint and Google Drive: instead of copying ACLs at ingestion, it asks the source whether the asking user can open each candidate document, at query time, as the user, before anything gets ranked. Emails are the join key between an asking user and a sharing grant there, so models.py carries the lesson from last time as one small function, docstring pointing back at the whole argument:

ASCII_WHITESPACE = " \t\n\r\v\f"

def normalise_email(value: str) -> str:
    trimmed = value.strip(ASCII_WHITESPACE)
    fold = str.maketrans("ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")
    return trimmed.translate(fold)

One correct implementation, in one place, born from a production incident. And in a review pass the day before publication, I found this in the package's own fakes:

user = GraphUser(user_id=user_id, email=email.strip().lower())

FakeGraph and FakeDrive, the deterministic permission models that ship inside the package, were normalizing every email with str.lower(). The repo was arguing against the exact bug it was carrying.

str.lower() is the polite one

lower() reads like the harmless sibling of casefold(). It skips the famous German fold: "ß".lower() is still "ß", where "ß".casefold() is "ss". But it is still a Unicode operation with many-to-one mappings inside it. U+212A is the Kelvin sign, a character that renders as a capital K in most fonts, and "K".lower() is a plain ASCII "k". One character, one method call, two identities become one.

The same two addresses through both folds. The left side is the bug; the right side is the doctrine, now anchored by a test.

I didn't leave that as a hypothetical. Before the fix, inside the fakes, a sharing link naming Kevlar@outside.example spelled with the Kelvin sign resolved to a PermissionProof for the ASCII user kevlar@outside.example. Two addresses the package's own normalise_email() keeps strictly apart, folded into one grant by the code whose job is to demonstrate the fence.

No test caught it going in, and the reason deserves a hard look: both sides of the comparison went through the same wrong fold. Link grantees were lowered, stored user emails were lowered, so every case-insensitivity test passed. A normalization bug in a join key is self-consistent. It doesn't crash and it never disagrees with itself. It just quietly merges identities that the doctrine one directory up says must never merge.

Fakes that demonstrate a guarantee are production code

This would be a smaller story if those were throwaway fixtures. They aren't. The fakes ship in on_behalf.fakes because they are the offline half of the product: they model SharePoint effective access (nested groups, folders that break inheritance, sharing links with expiry) and Drive's two auth modes, and they're what lets the demo and the 353-test suite run the full pipeline with no network, no credential, no key. When the suite proves "a sharing link is dead at the exact expiry instant", it proves it against these models. A test that validates the fence using a biased copy of the fence validates nothing.

So the fix was not a better fold in the fakes. It was deleting the second implementation. The fakes now import the same function everything else uses:

from ..models import normalise_email

user = GraphUser(user_id=user_id, email=normalise_email(email))

and the link-grantee matching goes through the same door:

user.email in {normalise_email(person) for person in link.people}

anchored by a test so the rule can't regress back into prose:

def test_link_people_matching_folds_ascii_only_never_unicode(
    tenant: GraphTenant,
) -> None:
    # "K" U+212A (Kelvin sign) lowercases to "k" under str.lower(); the fakes
    # must use the same ASCII-only fold as the rest of the package, so a link
    # naming the Kelvin-sign spelling is a different identity, not a grant.
    tenant.items["d-pub"].links.append(
        SharingLink(
            link_id="lnk-kelvin",
            scope=LinkScope.SPECIFIC_PEOPLE,
            people=("Kevlar@outside.example",),
        )
    )
    kevlar = tenant.add_user("u-kevlar", "kevlar@outside.example")
    decision = _resolve(tenant, kevlar, "d-pub")
    assert isinstance(decision, Denied)

The first character of that link's address is U+212A, not an ASCII K. They render identically on most screens, which is the whole problem, and why the comment spells it out.

Here's the part I keep coming back to. The doctrine existed the whole time. It was written down twice, once in the casefold post and once in the normalise_email docstring, and implemented correctly in exactly one place. The fakes just didn't call it. A security rule that lives in prose is advice. It becomes a property of the system on the day the function that implements it is the only door, and every caller, fakes included, has to walk through it.

The repo the bug was hiding in

The package deserved a cleaner arrival, because its own thesis is this same argument at a larger scale: a permission snapshot is a leak with a start date. Most "permission-aware" RAG connectors do this:

# ingestion, nightly
for doc in source.list_documents():
    index.add(doc, allowed_users=source.read_acl(doc))   # a copy

# query time
hits = [h for h in index.search(q, k) if user in h.allowed_users]   # the copy decides

That copy is wrong the day after it is right, in three distinct ways, and on-behalf ships a deliberately wrong SnapshotACLIndex (it raises a UserWarning at construction) so the three leaks are demonstrated by executable tests instead of described by paragraphs: a revoked direct grant the snapshot keeps honouring, a group the user left that the snapshot still counts them into, a sharing link that expired after ingestion. Each test asserts both directions, leak present in the snapshot and absent from the query-time path, so the demonstration can't pass by both sides being broken.

Same question, two indexes, after permissions moved. The snapshot believes 09:00 forever.

The offline demo runs one synthetic tenant through the same question at 09:00 and at 12:00, permissions moving at lunchtime, the index never re-ingested. Between the two runs one user loses a memo and another loses everything, while the index still holds all five documents, and the snapshot section closes on the sentence I'd put on a slide: the snapshot cannot see any of it, "because the event it needed to observe happened after the only moment it ever looked."

The whole demo, deterministic, no keys. Every number in this post comes from it.

The fix is ordering, twice over:

candidates = index.candidates(query)              # a lexical gate, unscored
decisions  = source.can_open_many(user, ids)      # the source answers, NOW
entitled   = [d for d in candidates if proven(d)] # proof or absence, no third state
return rank(query, entitled)[:k]                  # score last, cut last

Entitlement runs before ranking, so a document the user can't open never competes for the top-k window. The check runs at query time because that is the only moment the answer is true for. Whatever can't be proven is excluded and counted, never served on faith: a denial carries its reason, a source outage darkens that source's documents instead of widening access, and a redundant guard re-verifies every hit that reaches the context, raising EntitlementBreach instead of filtering silently, because a silent drop hides a real bug in the layer above.

What this does not settle

Exactness cuts both ways. A user whose grant was written under some exotic spelling of their address stays denied under an ASCII-only fold. That is the chosen trade-off, and it fails in the closed direction; the merge fails in the open one.

The live adapters sidestep folding almost entirely, by design. They do no ACL arithmetic: can_open fetches the item as the user, with a token minted on the user's behalf, and a 200 is the proof while 403 and 404 are the denial. The email fold matters most in the offline permission models, which is exactly the code most teams would wave off as "just fixtures".

And the live adapters have not yet been run against a real tenant. They implement the documented API contracts and pass the offline suite; the repo's RUNBOOK is the path to a real Microsoft 365 tenant and a Google Workspace domain, and until that walk happens the README refuses to claim it.

What changed in my process is small and concrete: fakes and fixtures now go through the same pre-publish sweep as shipped code, the grep for .lower( and .casefold( near anything identity-shaped covers the whole tree, and every doctrine function gets at least one test that exercises it through the fakes' own call path. The Kelvin test is the first of those. A rule is exactly as wide as the set of callers forced through the function that implements it; everything outside that set is a README.

The repo, with the demo, the three leak tests, and the RUNBOOK for a real tenant: github.com/vinimabreu/on-behalf

Python's casefold() merged two of my customers into one tenant

Vinicius Pereira — Wed, 12 Aug 2026 18:17:44 +0000

I spent a few days building a repo about multi-tenant retrieval. One knowledge base, many customers, and one promise: a customer can never retrieve another customer's documents. The filter runs before the candidate set is scored, the guard re-checks every chunk on the way out, every query lands in an audit log.

Then I ran an adversarial pass whose only job was to break the promise rather than confirm it. It broke it in four lines of setup, and the hole was in the function I had written specifically to prevent identity confusion.

The line that was supposed to be boring

Entitlement is compared on tenant identifiers, so identifiers get canonicalised once, at construction, and compared exactly afterwards:

def normalise_id(value: str) -> str:
    return value.strip().casefold()

The reasoning felt obvious. " ACME " and acme are the same customer, somebody will paste one with a trailing space eventually, and a fence that says "access denied" because of a space is a support ticket. casefold() rather than lower() because casefold is the Unicode-correct one, the one you are told to use for caseless comparison.

That is exactly why it is wrong here.

The tenant that was two tenants

The adversary created a customer called Straße-Werke and gave a completely unrelated principal a grant on strasse-werke. The unrelated principal read the documents.

>>> "Straße-Werke".casefold() == "strasse-werke".casefold()
True

casefold() is designed for caseless matching, and caseless matching is deliberately many-to-one. It is not a case conversion, it is a mapping onto a common form, and several of those mappings collapse characters that are not case variants of each other at all:

>>> "ß".casefold()          # one character becomes two
'ss'
>>> "K".casefold()     # KELVIN SIGN
'k'
>>> "ﬁ".casefold()     # LATIN SMALL LIGATURE FI
'fi'
>>> "ς".casefold() == "σ".casefold()   # final sigma
True

Every one of those is a merge. Feed casefold() two identifiers that an upstream registry considers distinct, and you get one entitlement key. Two customers become one customer.

Why this is worse than a normal access bug

Nothing failed. That is the part worth sitting with.

The filter ran correctly, against a key that was wrong. The candidate set was built correctly, from a key that was wrong. My guard, the deliberate second check that re-validates every chunk before it leaves, calls the same entitlement function, so it agreed. The audit log recorded a completely ordinary query:

QUERY principal=svc-partner allow=[strasse-werke/*/*] deny=[] returned=3

There is no exception, no anomaly, no unusual pattern. A reviewer reading that line sees a principal reading documents it is entitled to read, because by the time anything is logged, the two tenants are already the same tenant. Every layer of defence in depth inherits the same wrong key, so depth buys you nothing. The mistake happened before the first layer ran.

Then the same bug again, wearing different clothes

Having fixed the fold, I still had strip(). str.strip() with no argument removes all Unicode whitespace, not just the ASCII kind:

>>> "kronos\u3000".strip() == "kronos"   # IDEOGRAPHIC SPACE
True

The same applies to U+00A0, U+2007, U+1680, U+205F, U+0085 and U+2028. So a registry that enforces uniqueness on the raw string happily accepts kronos and kronos plus an ideographic space as two different customers, and my fence folds them into one.

The detail that makes this a good trap: zero-width space and the other invisible characters that people usually test for, U+200B, the soft hyphen, the byte order mark, are not stripped. So a test suite that covers "invisible characters" in the obvious way passes while the actual merge path stays open.

Two fixes I tried and threw away

Use lower() instead of casefold(). It reads like the conservative choice. It does not help:

>>> "K".lower()
'k'

The Kelvin sign folds under lower() too, because that mapping is simple case conversion, not full folding. Swapping the function narrows the hole without closing it, which is the worst outcome available: it feels fixed.

Reject non-ASCII identifiers at construction. This closes it, and it is wrong for a different reason. It refuses legitimate identifiers from most of the world in order to fix a bug in my normalisation, and it hides the actual rule behind a character-set restriction that has nothing to do with the problem.

The fix

Canonicalise as little as possible, and refuse anything that would need canonicalising to be safe:

ASCII_FOLD = str.maketrans(ascii_uppercase, ascii_lowercase)
ASCII_WHITESPACE = " \t\n\r\v\f"

def normalise_id(value: str) -> str:
    """Trim ASCII whitespace, fold ASCII case, and nothing else."""
    return value.strip(ASCII_WHITESPACE).translate(ASCII_FOLD)

Straße-Werke and strasse-werke now stay two tenants. A tenant named with non-ASCII characters keeps working, and keeps its identity.

Then the second half, which matters more than the first: anything still carrying whitespace or a non-printable character after the trim is refused at construction, naming the offending code point in the error. Not folded onto its neighbour, not silently accepted. A tenant id with an ideographic space in the middle of it is a data problem upstream, and the honest thing a fence can do is say so out loud instead of guessing which neighbour it meant.

The general shape of it

The lesson generalises past Unicode, and it is the reason I am writing this down rather than quietly pushing the patch.

When a normalised value becomes a security key, normalisation is part of the attack surface. Any many-to-one transform applied to an identity is a merge operation, and a merge between two identities is a privilege grant. Case folding, accent stripping, whitespace collapsing, punctuation removal, homoglyph mapping, lowercasing an email local part: all of them are helpful in a search box and all of them are dangerous the moment the output is compared for authorisation.

Two questions worth asking of any such function:

Can two inputs that the system elsewhere considers distinct produce the same output?
If they can, which layer is supposed to notice? If the answer is "the layer that compares the outputs", there is no such layer.

The reason I found this at all is that the adversarial pass was told to breach the fence, not to test it. A test written by the person who wrote the code asks "does it do what I meant". A test written to break it asks "what did I mean that was wrong". Those find different bugs, and the second kind is the kind that reaches production.

Why the repo concentrates risk there on purpose

The repo's actual argument is older and simpler than the Unicode story. Most multi-tenant retrieval filters after the ranking, which means a leak needs one caller, one cache, one new endpoint to forget. Filtering before the candidate set is scored moves all of that risk into a single function.

Which is precisely how I ended up here. Concentrating the risk is the right trade, and the bill it comes with is that the one function you concentrated it into is now the only thing worth attacking. That is why it carries 141 adversarial tests, and why four of them are about a character that is not a case variant of anything.

The repo is at github.com/vinimabreu/tenant-fence: the fence, the deliberately wrong version kept next to it so the suite can demonstrate the leak rather than describe it, and the tests that would have caught this on day one.

My exactly-once refund paid twice, and 373 passing tests could never have caught it

Vinicius Pereira — Tue, 04 Aug 2026 21:24:54 +0000

I spent a week building a repo about making LangGraph agents trustworthy in production. Routing measured against labelled fixtures instead of vibes. A human gate before anything irreversible. Durable resume, so a process that dies mid-refund comes back and does not pay the customer twice.

373 tests, all green. CI green on four Python versions. Then I ran an adversarial pass over it, whose only job was to break my claims rather than confirm them, and the first thing it broke was the one I had written the repo to prove.

The claim

The centre of the whole thing is one sentence from my README:

INSERT OR IGNORE then read. The claim and the check are one statement, so two attempts cannot both believe they are first.

Standard idempotency. Every effectful action gets a key, the key is inserted once, and whoever loses the race replays the stored result instead of running the handler again. My tests hammered it: eight concurrent processes on one key, one handler execution, seven replays. Crash injected at every boundary in the run, resumed from disk, effect count unchanged.

The bug

The key was built like this:

action_id = digest({
    "ticket": ticket_id,
    "tool": call.name,
    "args": call.args,      # <- what the model typed
})

call.args is raw model output. A language model asked for a $45.00 refund can write that amount in more than one way, and all of them are valid JSON that my schema happily accepts:

    4500  ->  action_id 1270c45d1612d5d1
  "4500"  ->  action_id 8f0f6363fea8adcf
  4500.0  ->  action_id b78a82d722dde95d

distinct idempotency keys for one identical $45.00 refund: 3

Three keys. Three refunds. Same customer, same invoice, same amount.

If a queue retries a ticket, or a customer submits twice, or the same conversation is driven again for any of the boring reasons production does that, and the model spells the number differently on the second pass, my exactly-once guarantee quietly becomes at-least-once. On money.

Why my tests were structurally incapable of finding it

This is the part I keep thinking about.

The test suite runs offline against a deterministic stand-in: a keyword classifier that, given the same ticket, emits byte-identical JSON every single time. That is a deliberate design choice and mostly a good one. It makes the suite fast, free, and reproducible, with no API key and no network.

It also means the second delivery of a ticket always produced exactly the same string as the first. The suite could never observe the failure, because the only component capable of producing the failure had been replaced with one that cannot.

My tests were not weak. They were blind by construction, in a way that green output cannot show you. Every assertion I had written was true. The thing I never asserted was that two equivalent calls collapse to one key, because with a deterministic generator, equivalent and identical are the same word.

The fix is one line of intent: derive the key from the arguments after the tool's own schema has parsed them, not from whatever the model typed.

parsed = spec.validate_args(call.args).model_dump(mode="json")
action_id = digest({
    "ticket": ticket_id,
    "tool": call.name,
    "args": parsed,          # parsed, not typed
})

Now 4500, "4500", 4500.0, " 4500 " and "+4500" all produce one key, while seven genuinely different refunds still produce seven. And there is a test that drives the same ticket twice with different spellings and asserts a single handler execution, so it stays fixed.

The second thing it broke: a metric that was measuring the wrong noun

While it was in there, the same pass killed a number I had been quoting proudly.

My report said tool choice 21/22 correct (95.5%). It compared the name of the tool the model picked against the expected name. Nothing anywhere compared the arguments.

So this ticket:

We were charged $90.00 on INV-10032 but only $45.00 of that was valid. Refund the difference.

with the model proposing issue_refund(invoice_ref="INV-10032", amount_cents=9000) scored as perfectly correct. Right tool, double the money. A version proposing $9,000 instead of $45 also scored perfectly correct.

"Chose the right action" and "chose the right tool name" are not the same claim, and I had been publishing the second one under the first one's label.

The fix added expected arguments to every fixture and a separate metric that counts them. Which is why my headline numbers got worse:

	before	after
routing accuracy	92.0%	90.2%
tool metric	21/22 name only	21/23 counting arguments
in-doubt stops	8	24
crash boundaries swept	176	208

Nothing regressed. The measurements stopped flattering me. The in-doubt count tripled because the audit also found two windows inside the effect where a crash could land and my crash-point list did not know they existed, which meant a line in my report was true by accident rather than by design.

What the repo does now

The whole thing in one pass. What follows is the same three acts, one at a time.

Routing is scored per branch with a confusion matrix, and the parser never raises: output it cannot trust becomes an escalation carrying one of nine named reasons, recorded in state.

Irreversible tools stop for a human, and the interrupt is measured in both directions. Missing a required stop is scored at threshold zero, because there is no acceptable rate for shipping a refund nobody approved. Stopping when you did not need to is scored separately with a real budget, because approval fatigue is how a gate stops meaning anything: interrupt people often enough for nothing and they start clicking approve without reading.

And the part that took the longest to get honest. The process can be killed at any of 208 points in a run and resumed from disk. 112 of the reachable boundaries come back byte-identical with the effect count unchanged. The other 24 land in a window that genuinely cannot be closed, between claiming the effect and recording its result, and there the system stops and says it does not know rather than guessing. A human reconciles it with the ledger in front of them.

Zero duplicates. Zero mismatches. A CI gate fails the build when any of those numbers moves.

vinimabreu / langgraph-production

A LangGraph support agent with the reliability layer around it: routing measured against labelled fixtures, a human gate before irreversible tools, durable resume with exactly-once effects, and a CI gate that fails when a number moves.

langgraph-production

A LangGraph agent with the reliability layer that decides whether it can ship.

Building the graph is the easy half. The half that decides whether it goes live is everything around it: what happens when the process dies between two nodes whether the refund gets paid twice on the way back up, whether an approval somebody gave on Tuesday can authorise a different action on Thursday, and whether anyone can say out loud how often the router is right.

This repo answers those four questions with code and with numbers, on a support desk for a fictional product. All data is synthetic.

The report this repo exists to produce

Real output from python -m langgraph_production.evaluation on this repository, not an illustration:

========================================================================
SUPPORT GRAPH EVALUATION
========================================================================
  fixtures      answer 13, escalate 15, tool 23  (total 51)
ROUTING
  accuracy      46/51 = 90.2%
  macro F1      0.907
  tool choice   22/23 correct (95.7%) on

…

View on GitHub

The takeaway I did not expect

I went in expecting the audit to find sloppy corners. It found the opposite: my careful parts were fine, and the failure was hiding in the seam between two decisions that were each individually correct.

Using a deterministic stand-in for tests: correct.
Keying idempotency off the tool call: correct.

Together they produce a system that is provably exactly-once against a generator that cannot vary, and at-least-once against the one you actually ship with.

So the question I now ask about any test suite, including yours: what can this suite not observe, by construction? Not what did I forget to assert. What has been designed out of the room. If you replace the non-deterministic component with a deterministic one for testing, every bug that only exists because of non-determinism is invisible to you, and it will stay invisible while the dashboard stays green.

Green tests are evidence about the world you built for them. It is worth being explicit about how much of the real one you left out.

Code, numbers, and the harness that produces them: github.com/vinimabreu/langgraph-production

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

The ChocoDEV Signature Bar: one chocolate bar, zero images

Vinicius Pereira — Sun, 02 Aug 2026 22:08:15 +0000

This is a submission for Frontend Challenge: Comfort Food Edition, CSS Art.

Inspiration

Yesterday I submitted a landing page for ChocoDEV, a chocolate brand that exists only in CSS. A brand needs a product shot, and this brand has a rule: no images, ever. So the product shot had to be drawn in the only medium the brand allows.

It is one chocolate bar with a bite taken out of it, because a perfect untouched bar is a lie. Nobody photographs their comfort food before tasting it.

Demo

Live: vinimabreu.dev/chocodev/bar

The fallen chunk reacts if you click it. That is the entire JavaScript budget, three lines, and the piece is complete without them.

Journey

Every chunk is a molded facet: four trapezoid faces built from angular gradients, so the light lands differently on each slope, the way a real mold presses chocolate. The letters are not printed on top, they are pressed in, one dark shadow above and one thin highlight below, and that two-shadow trick is the whole illusion of depth.

The bite was the hard part. A clean clip-path zigzag looked like scissors had done it, so the exposed edge got fracture facets in lighter matte browns and a few crumbs, because chocolate never breaks politely.

There is exactly one animation, a slow sheen that crosses the bar like it is turning under a studio light, and it sits behind a prefers-reduced-motion guard. When your system asks for stillness, the bar just stands there looking edible.

Everything is gradients, box-shadow, border-radius and clip-path. Open devtools, it is chocolate all the way down.

ChocoDEV: a chocolate bar you can eat, in pure CSS

Vinicius Pereira — Sun, 02 Aug 2026 21:22:19 +0000

This is a submission for Frontend Challenge: Comfort Food Edition, Perfect Landing.

What I Built

ChocoDEV is a landing page for a chocolate brand that does not exist, and I am fine with that. Chocolate is my comfort food, and it turns out it is also South America's: the oldest traces of cacao were found in the upper Amazon, which as a Brazilian I consider a home win.

The page has one rule: zero images. No PNGs, no SVGs, no background URLs. Every square of chocolate, every drip, every bevel is gradients, box-shadow and keyframes. The footer makes the claim and devtools can check it, which is my favorite kind of claim.

The centerpiece is a chocolate bar you can actually eat. Each chunk is a button, each bite updates a counter, and when the bar is gone you get to bake a new one. There is also an exploded anatomy view of a single square, a scroll progress bar, and a testimonials section where four honest people confess things about chocolate that most of us only think.

Demo

Live demo: vinimabreu.dev/chocodev

Click a chunk of the bar. The counter is watching.

Journey

The zero images rule sounded like a gimmick and turned into the whole education. A believable chocolate square needs light coming from somewhere, and without images that means layered box-shadows: an inset highlight on top, an inset shadow at the bottom, a drop shadow below. Once the light direction was consistent, everything suddenly looked edible. Before that it looked like brown buttons.

The part I am most proud of is not visible. The page respects prefers-reduced-motion in three places: the CSS animations, the stat counters that jump straight to their final value instead of counting up, and the parallax that simply declines to run. Every interactive element is a real button with a name a screen reader can speak, the document has a language and a title, and the menu wraps instead of clipping on narrow screens. Accessibility is a judging criterion in this challenge, and I think it should be the first one: a landing page that only works for some visitors is a poster.

What I would do next: a dark and milk chocolate theme toggle, and sound on the snap. Probably a mistake. Most good ideas about chocolate are.

Thanks for reading, and go eat something that makes you feel at home.

Everything that broke when I imported hand-written Make blueprints into a real workspace

Vinicius Pereira — Tue, 28 Jul 2026 17:52:42 +0000

The import dialog took my JSON without a single complaint. The first execution died on module 2 with BundleValidationError: Validation failed for 7 parameter(s).

That gap between "the import accepted it" and "the runtime will actually run it" is where I spent most of a day this week, and almost none of it is documented anywhere I could find. So here it is, in the order it broke.

Context, briefly. I write Make (ex-Integromat) scenarios as blueprint JSON by hand instead of assembling them in the canvas, because I want them in git, diffable, reviewable. The scenarios route an order flow between an ERP and a 3PL. Every decision that can go wrong (is this order complete, which carrier code does a typo'd shipping method mean, is this event a duplicate) lives in a typed FastAPI service with 83 pytest tests, and every response carries the same envelope: decision for the routers to branch on, reason for a human, evidence for the audit trail. The scenario routes; the service decides. To test the whole thing for real, I imported the blueprints into a live Make workspace and exposed the local service through a cloudflared tunnel.

Then the workspace started grading my homework.

1. Import is a parser, not a validator

The blueprints were clean JSON. Modules, routes, mappings, all structurally valid, and the import agreed. The first run did not: Validation failed for 7 parameter(s), naming seven fields I had never typed in my life, all missing from the HTTP module's mapper: serializeUrl, shareCookies, rejectUnauthorized, followRedirect, useQuerystring, gzip, useMtls.

These are the checkboxes the canvas quietly fills in when you drop an HTTP module onto a scenario. Write the module by hand and nobody fills them, and import does not care, because import only checks that the JSON parses into modules and routes. The module's own contract is enforced at execution time.

Fine. I added the seven and ran again.

Validation failed for 1 parameter(s): followAllRedirects.

The first error listed seven missing parameters when there were eight. Runtime validation happens in layers, and each run only surfaces the layer it died in. Budget one execution per layer of complaints; the error list in front of you is not the whole bill.

Here is the mapper that finally runs, straight from the blueprint:

"mapper": {
  "url": "https://YOUR-SERVICE-HOST/orders/validate",
  "method": "post",
  "headers": [{ "name": "Content-Type", "value": "application/json" }],
  "data": "{\"order_ref\": \"{{1.order_ref}}\", \"customer\": {{13.json}}, \"shipping_address\": {{14.json}}, \"shipping_method\": \"{{1.shipping_method}}\", \"lines\": {{12.json}}}",
  "serializeUrl": false,
  "shareCookies": false,
  "rejectUnauthorized": true,
  "followRedirect": true,
  "followAllRedirects": false,
  "useQuerystring": false,
  "gzip": true,
  "useMtls": false
}

Keep that block. It is the difference between a blueprint that imports and a blueprint that runs.

2. `{{1.body}}` does not exist, and the failure mode is silence

I had written the webhook mappings the way anyone with HTTP reflexes would: the webhook receives a POST, so the payload must live in {{1.body}}. It does not. Make's custom webhook parses the incoming JSON and exposes the fields at the top level of the module output: {{1.order_ref}}, {{1.shipping_method}}. There is no body.

And here is the part that costs hours: referencing a path that does not exist is not an error in Make. It renders as empty. Silently. My service received b'' first (the whole body reference resolved to nothing), and after a partial fix, the literal string null. To the validator those looked like genuinely broken requests, so it did its job and reported them as such, which pointed me at the wrong suspect. I went through the service looking for a bug that was not there. The service was fine. The bug was an absence.

The ten lines that ended the guessing

What broke the loop was not reading the scenario harder. It was logging what actually crossed the wire:

from fastapi import FastAPI, Request

app = FastAPI()

@app.middleware("http")
async def log_raw_body(request: Request, call_next):
    body = await request.body()
    print(f"{request.method} {request.url.path} raw={body!r}")
    return await call_next(request)

Behind a tunnel, this is a complete diagnostic rig for a visual tool. Every mapping experiment in the canvas shows up seconds later as the exact bytes it produced. My terminal reads like a fever chart of the session: raw=b'', then raw=b'null', then a perfect payload. Once I could see what Make sent instead of inferring it from downstream symptoms, every remaining bug fell in minutes.

3. Collections do not serialize into text fields

Next, the ops-alert branch. When validation fails, the scenario posts an alert carrying the list of gaps, and I mapped the gaps array straight into the raw JSON body of that request. Make rendered the collection as text, inside the quotes it happened to land in, and produced a body that was no longer JSON. The endpoint said 422, and it was right to.

A collection mapped into a text field does not become JSON. It becomes a string shaped like whatever Make's text rendering of that collection is, which inside a hand-built JSON body is a syntax error.

Two fixes, both in the final blueprints: run each collection through a JSON > Transform to JSON module and embed the result without surrounding quotes ({{12.json}}, not "{{12.json}}"), or keep alert bodies scalar-only (gap_count instead of the gaps array). Look back at the mapper above and you can see both rules at work: webhook scalars quoted, collections embedded bare.

4. The default schedule is a 15-minute tick

The scenario worked. Then it appeared to die. I sent test events and nothing happened. No error, no execution, no log line. Nothing.

Make's default schedule runs a scenario every 15 minutes. A webhook trigger without "Immediately as data arrives" queues events silently until the next tick. To anyone watching, a perfectly healthy integration looks down for up to fourteen minutes at a stretch. One dropdown. Nothing broken. This is the incident titled "the integration stopped working" that resolves itself before anyone finishes triaging it.

What the wire showed when it all worked

Every decision, reason, gap and score above is taken verbatim from the live runs described in this post.

Three routes, executed live through the imported scenario, values as returned:

A clean order came back VALID with 15 checks passed, the shipping method mapped MAPPED_EXACT to SM-STD-01, and the order was created at the 3PL. The happy path nobody writes posts about.
A broken order came back INVALID with the reason 7 gap(s) found, 8 check(s) passed (the same 15 checks, now split), and routed to the ops alert with gap_count 7. The router branched on decision; the reason string went into the alert text unchanged.
An order with the shipping method typed as "parcel" scored 0.60 on fuzzy match, below the 0.85 floor, so the service returned UNMAPPED with one ranked candidate and the reason returning 1 candidate(s) for human review instead of guessing. The scenario put the order on hold for a human.

That last route is the whole argument. A visual flow will route a wrong guess with the same confidence as a right one. The refusal to guess has to come from somewhere with a test suite.

The outage I did not schedule

Mid-session, cloudflared dropped the tunnel for real. Cloudflare error 1033, service unreachable. The Break error handler on the HTTP module (current Make docs call the directive Retry; the blueprint directive is still builtin:Break) parked the execution in Incomplete Executions with its mappings intact. Nothing lost, nothing half-applied, one click away from resolving once the service was reachable again. The exact failure mode this design exists to survive walked in unannounced, and the pattern held. I could not have scripted a better demo, and for once I didn't.

What this does not tell you

The middleware shows what arrives, not what Make intended. A mapping bug that happens to produce valid-shaped JSON with wrong values sails through both the wire log and the schema check. The semantic checks still have to live in the service.

None of this makes Make the villain. The event queue, the retry policy, the incomplete-executions safety net are better than what most hand-rolled webhook consumers ship. That is exactly why the routing belongs there, and the deciding does not.

Hand-writing blueprints is a real trade. The canvas would have filled those eight HTTP parameters for me and I would never have met BundleValidationError. I pay that tax to get scenarios in git with reviewable diffs. If you do not need the diff, click the modules.

And every Transform to JSON module is one more operation on every run, which is the unit Make bills in. Scalar alert bodies are not just simpler. They are cheaper.

The service, both importable blueprints with all eight parameters already in place, and these lessons written down live at github.com/vinimabreu/make-failsafe, mostly so I never rediscover any of this at 2 a.m.

The flow routes, the service decides, and the wire is the only place where both of them tell the truth.

Your LLM's confidence score is lying to you

Vinicius Pereira — Mon, 20 Jul 2026 11:31:33 +0000

The most expensive bug in an LLM system is not the output that is obviously broken. That one you catch. It is the fluent, well formed, confidently wrong answer that reads exactly like a correct one and walks straight into your system of record because nothing was standing between the model and production.

"Reads correct" and "is correct" are two different properties, and the model only optimizes the first. So the answer that is almost right, the near miss, comes back just as polished as the answer that is exactly right. You cannot tell them apart by looking. Neither can the model.

Similarity is not confidence

When teams want to gate on that, they reach for a number. Usually one of two: the model's own reported confidence, or the similarity score of whatever chunk it retrieved. Both feel like confidence. Neither one is.

Ask a model how sure it is and you get more generated text, sampled from the same distribution that just produced the wrong answer. It is a guess about a guess. And a similarity score measures proximity, not truth. The near miss scores high precisely because it sits close to the right answer. That is the whole trap: a similarity cutoff hands its highest marks to the single most dangerous output in the system, the confident near miss, and waves it through.

Compute it from what you can check

The way out is to stop reading confidence off the model and start computing it from signals you can verify without trusting the model at all. Three of them carry most of the weight.

Grounding asks whether the output is actually entailed by the evidence you gave it, not whether it reads well. An answer nobody can trace back to a source is not trustworthy, however clean it looks. A lexical-overlap baseline gets you moving, but the real seam is a (claim, evidence) -> float function, so you can swap in an NLI model or a cross-encoder the moment overlap is too blunt. Entailment over proximity.

Agreement resamples the generator on the same input and measures whether the answer survives. A model that flaps between three answers when you ask three times is telling you something a single clean draw hides. A minority draw is low confidence even when it parses perfectly. This is the one signal that costs you extra generations, so you price it in on purpose.

Validation is the cheap one, and the one people skip. Does the output satisfy the schema you declared, every required field present and the right type? A structurally invalid output should never reach the auto-accept ceiling no matter how good the other signals look, so one missing field drags the score down instead of passing silently.

The gate, as code

I wrote this up as a small reference implementation, confidence-gate, so the pattern has something runnable behind it. The generator goes in as a plain callable, which keeps the whole thing deterministic and runs it with no API key. Wiring it looks like this:

from pydantic import BaseModel

from confidence_gate import (
    AgreementSignal, GroundingSignal, Policy,
    ReviewQueue, Router, Task, ValidationSignal,
)


class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    total: str
    date: str


def generator(prompt: str) -> str:
    # Swap in a real model call. The gate does not care where the text
    # comes from, only whether it survives the signals.
    return '{"invoice_number": "INV-4021", "vendor": "Northwind Supplies", "total": "1840.00", "date": "2026-02-11"}'


queue = ReviewQueue(":memory:")
signals = [ValidationSignal(Invoice), GroundingSignal(), AgreementSignal(generator, k=3)]
router = Router(generator, signals, Policy(), queue)

verdict = router.route(
    Task(
        task_id="inv-1",
        prompt="extract invoice fields",
        evidence="Invoice INV-4021 from Northwind Supplies. Date 2026-02-11. Total 1840.00 USD.",
    )
)
print(verdict.decision, verdict.aggregate_confidence)

Run the bundled demo over three documents, one clean, one ambiguous, one with no supporting evidence, and it prints the decision for each. Captured output:

TASK            DECISION        CONF  SIGNALS
----------------------------------------------------------------
northwind-clean AUTO_ACCEPT     1.00  validation=1.00 grounding=1.00 agreement=1.00
globex-partial  HUMAN_REVIEW    0.50  validation=0.75 grounding=0.50 agreement=0.67
unknown-source  ABSTAIN         0.00  validation=1.00 grounding=0.00 agreement=1.00

review queue: 1 pending
  #1 globex-partial  conf=0.50  :: min aggregate 0.500 (accept>=0.75, abstain<=0.35) -> human_review

The middle row is the whole argument. globex-partial validates against the schema except for one missing field, only half of it is supported by the evidence, and the generator does not fully agree with itself on resampling. No single number screams "wrong", so the gate refuses to auto-accept and refuses to abstain. It routes the item to a person and keeps the receipt.

The bottom row is the one that would have burned you. unknown-source is structurally perfect, validation 1.00, and the model reproduces it on every resample, agreement 1.00. By any signal you could read off the model, it is maximally confident. Grounding is 0.00 because nothing in the evidence supports it. A confident, self-consistent answer with no external support, and the gate abstains.

Tune on the cost of a false accept

Look at how the aggregate falls out. It is the minimum of the signals, not the average. That is deliberate, and it comes from the cost structure rather than a taste for being strict. A cache miss or a re-run costs seconds. A false accept ships a confident wrong answer into the record with no model left in the loop to hedge it. Those two outcomes are not symmetric, so you do not tune the threshold to hit some acceptance rate. You tune it on what a single false accept costs you. A gate is only as strong as its weakest verified signal, so one failing signal is enough to deny auto-accept. Abstaining is cheap. Being confidently wrong is the bill you do not see coming.

That leaves three outcomes instead of two. Auto-accept when the external signals agree. Human review, with the full audit trail, when they are mixed and a person should look. Abstain when there is no support at all, because refusing to answer beats guessing when you cannot check the guess. The model proposes, the code disposes.

What this does not do

It does not make a wrong answer right. It refuses to ship one silently, which is a smaller and far more achievable goal. If your evidence is itself wrong, grounding will happily confirm a wrong answer against it, so the quality of your source text just became part of your trust boundary. The default entailment is lexical overlap, which is blunt: it catches "the answer cites nothing in the source" but not "the answer inverts the one number that mattered", so anything real gets a stronger function through that seam. Agreement costs you K generations per item, which is real latency and real money traded for a convergence signal. And none of this is a product with uptime promises. It is the shape of the design, made runnable.

A confident wrong answer and a confident right one are the same string until something outside the model checks. Be that something.

Clear the Lineup: the passage my RAG retrieved and then threw away in silence

Vinicius Pereira — Tue, 14 Jul 2026 23:24:30 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

rag-quality is a small, self-contained RAG pipeline: it retrieves passages from a corpus (sparse, dense, and hybrid modes), then generates a grounded answer that cites the passages it used and abstains when the context does not support one. It ships with an eval harness that scores retrieval quality.

Recently I commented on a dev.to thread about non-deterministic retrieval, the ways a RAG can fetch the right passage and still answer wrong. In reply, the author added one more check to his list: verify the context you asked for is the context you actually got. That check named the exact hole I had in this repo, one stage past retrieval, in assembly, against a bug I had shipped and not noticed.

Bug Fix or Performance Improvement

The generation step assembles retrieved passages into a context block and sends it to the model. Here is what it did:

def build_prompt(question: str, hits: list[Hit]) -> str:
    blocks = [f"[{h.chunk.id}] {h.chunk.text}" for h in hits]
    context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS]
    return f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"

MAX_CONTEXT_CHARS is 6000. It joins every retrieved passage, then hard-slices the string to 6000 characters. If the joined context runs longer, the tail is gone. The tail is the lowest-ranked hits, and the lowest-ranked hit is often the one that actually holds the answer: retrieval surfaced it, ranking put it last, the slice deleted it. No exception, no log, no return value that hints anything was cut. The caller gets a shorter prompt and believes it is complete.

What makes this one sting is the asymmetry a few lines down in the same file:

if message.stop_reason == "max_tokens":
    text += "\n[warning: answer truncated at the token limit]"

The code already knows truncation is dangerous. It shouts when the model's output is cut. It stays silent when the model's input is cut, which is the more dangerous of the two, because a truncated input can quietly remove the one passage the answer depended on and the model will confidently answer from what is left.

And the tests stayed green because nothing exercised the failure. The sample corpus is 17 short documents; its joined top-k context fits comfortably under 6000 characters, so the slice never removed anything in the bundled runs. The eval harness measures retrieval quality (hit@k, MRR, recall@k), whether the right passage was retrieved, not whether it survived assembly into the prompt. The bug lives in the gap between those two stages, which is precisely the gap no metric was watching.

Code

The whole change, the fix, the tests, and the optional Sentry and Gemini tooling, is one pull request:

Fix silent context truncation in build_prompt #1

vinimabreu posted on Jul 14, 2026

build_prompt joined every retrieved passage and hard-sliced the result to MAX_CONTEXT_CHARS, dropping the tail with no exception, log, or return signal. The tail is the lowest-ranked hits, and the lowest-ranked hit can be the one that holds the answer. The same file already warns on output truncation but stayed silent on the input side.

This packs whole passages in rank order up to the budget, never cuts a passage mid-text, and returns dropped_ids so the caller announces what was left out, mirroring the existing output-truncation warning.

Also adds:

env-gated Sentry spans across the pipeline (no-op without sentry-sdk and SENTRY_DSN), so an over-budget request shows total_context_chars, dropped_ids, and a context.truncated_chars measurement on the rag.build_prompt span
scripts/root_cause.py, an optional Gemini helper that localizes the loss from a trace plus the source
five tests that run key-free and offline

View on GitHub

My Improvements

Stop slicing a string. Pack whole passages in rank order until the next one would blow the budget, never cut one mid-text, and return the ids you dropped so the caller can announce them.

-def build_prompt(question: str, hits: list[Hit]) -> str:
-    blocks = [f"[{h.chunk.id}] {h.chunk.text}" for h in hits]
-    context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS]
-    return f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"
+def build_prompt(question: str, hits: list[Hit]) -> tuple[str, list[str]]:
+    separator = "\n\n"
+    budget = config.MAX_CONTEXT_CHARS
+    blocks = [f"[{hit.chunk.id}] {hit.chunk.text}" for hit in hits]
+
+    kept, dropped_ids, used = [], [], 0
+    for hit, block in zip(hits, blocks):
+        extra = len(block) + (len(separator) if kept else 0)
+        if used + extra <= budget:
+            kept.append(block)
+            used += extra
+        else:
+            dropped_ids.append(hit.chunk.id)
+
+    if dropped_ids:
+        log.warning("context truncated: %d passage(s) dropped: %s",
+                    len(dropped_ids), ", ".join(dropped_ids))
+
+    context = separator.join(kept)
+    prompt = f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"
+    return prompt, dropped_ids

That diff is the essence of the change. The shipped version also opens the rag.build_prompt span from the Sentry section around this same code, which is how those numbers reach the trace.

Generator.answer now unpacks the tuple and appends a note that mirrors the existing output warning, so the drop is visible in the answer itself:

if dropped_ids:
    note = ", ".join(dropped_ids)
    text += f"\n[warning: {len(dropped_ids)} passage(s) dropped from context: {note}]"

The budget stays in characters here on purpose. The stronger version counts the real token budget of the target model; I noted that in the docstring rather than building it, because the point of this fix is the missing signal, not a better ruler.

Proof. Same input in both runs: four bulky decoy passages plus a short answer chunk that is retrieved but ranked last, so the joined context is 7343 characters against a 6000 budget. Real console output, captured with the repo's own interpreter, no API key.

Before, the answer chunk is gone and nothing says so:

MAX_CONTEXT_CHARS      = 6000
full joined context len= 7343
answer chunk id        = expenses::answer
answer text present in FULL joined context? True
answer text present in BUILT prompt?        False
answer sentence in built prompt?            False
------------------------------------------------------------
RESULT: answer chunk SILENTLY DROPPED. No exception, no warning, no log. Caller gets a prompt missing the answer.

After, packing skips the one passage that will not fit, keeps the small answer chunk behind it, and reports the id it dropped:

context truncated: 1 passage(s) dropped to fit 6000-char budget: expenses::3
MAX_CONTEXT_CHARS      = 6000
full joined context len= 7343
answer chunk id        = expenses::answer
dropped_ids reported   = ['expenses::3']
answer text present in BUILT prompt?        True
answer sentence in built prompt?            True
------------------------------------------------------------
  expenses::0        -> KEPT (verbatim)
  expenses::1        -> KEPT (verbatim)
  expenses::2        -> KEPT (verbatim)
  expenses::3        -> dropped whole
  expenses::answer   -> KEPT (verbatim)
------------------------------------------------------------
RESULT: answer PACKED IN, and the passage that did not fit (['expenses::3']) is reported to the caller. No silent loss.

Four behavioural tests lock the contract: an over-budget answer chunk is either packed in or named in dropped_ids and never silently gone, over-budget context reports its dropped ids, no passage is ever cut mid-text, and within-budget context drops nothing. A fifth pins the tuple return type so a caller cannot regress to the old string signature. All five run with the repo's own interpreter, no key and no network.

The rule I kept: any place that decides what the model sees has to announce what it dropped. A retriever that finds the right passage and an assembler that quietly deletes it produce the same wrong answer, but only one of them leaves a trace. The output truncation in this file already knew that. The input side just needed to learn the same manners.

Best Use of Sentry

The fix for "silent" is instrumentation. I added env-gated Sentry spans across the pipeline (off unless SENTRY_DSN is set and sentry-sdk is installed, so the repo never hard-depends on it). The span that matters here is rag.build_prompt:

total_context_chars and max_context_chars
n_dropped and dropped_ids
a context.truncated_chars measurement
a warning-level event whenever n_dropped > 0

Now a wrong answer on a long-context request is not a mystery. The pipeline is instrumented end to end: rag.answer to rag.retrieve (hit_ids) to rag.build_prompt (dropped_ids) to rag.generate (stop_reason). The trace below exercises the assembly step directly, so it shows rag.answer to rag.build_prompt, where the truncation happens and the dropped ids surface.

On the over-budget request, the rag.build_prompt span reads total_context_chars 7343 against a max_context_chars of 6000, a context.truncated_chars measurement of 1343, and the dropped_ids, sitting directly under the rag.answer transaction. The truncation that used to be invisible is now a value on a span.

Best Use of Google AI

To make the localization repeatable I added scripts/root_cause.py: it takes a Sentry trace plus the source files that ran, and asks Gemini to point at the line. The prompt is fixed and paste-ready:

You are debugging a RAG pipeline that returned a wrong answer. Here is a Sentry trace and the relevant source. The retrieval span shows the correct passage was retrieved, but the final answer is wrong. Localize the exact line where the retrieved context is lost, explain the mechanism, and note whether the loss is signalled to the caller. Then propose a minimal fix.

The script is env-gated on GOOGLE_API_KEY or GEMINI_API_KEY and fails with a clear message when no key is set, so it never runs blind.

Run against the pre-fix source with the real trace, on gemini-2.5-flash, it returned this (verbatim):

The Sentry trace clearly indicates that the rag.retrieve span successfully identified expenses::answer as a relevant passage. However, the subsequent rag.build_prompt span shows a total_context_chars of 7343 and a max_context_chars of 6000, leading to 1343 characters being truncated. The symptom confirms that expenses::answer was "absent from the assembled prompt," which directly led to the LLM's abstention (I could not find this in the provided documents.).

1. Exact Line Where Context is Lost. The context is lost in app/generate.py, specifically at this line within the build_prompt function: context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS].

2. Mechanism of Loss: naive string slicing after concatenation. The function joins all blocks into a single long string, then applies a hard character limit by slicing that entire concatenated string. If the total length exceeds config.MAX_CONTEXT_CHARS, the slice cuts characters from the end of the combined string. Passages later in the hits list are more likely to be truncated, and even a highly relevant passage like expenses::answer can be cut off if it falls past the character limit.

3. Signalling of Loss. "No exception, warning, or dropped-id was reported to the caller." The function simply returns the truncated string; there are no checks, logging.warning calls, or exceptions that would inform the calling Generator.answer method. Sentry captured the context.truncated_chars measurement, but that is instrumentation-level reporting, not a signal within the application's business logic.

4. Minimal Fix. Adjust build_prompt to iteratively add passages, respecting config.MAX_CONTEXT_CHARS without splitting individual passages, so higher-ranked complete passages are prioritized. [It then produced a pack-to-fit loop that stops adding blocks once the budget is reached.]

The model landed on the same line I did, read the mechanism straight off the trace measurements, and proposed the same pack-to-fit shape as the fix above. Sentry said which chunk vanished and by how much; Gemini said which line threw it away and why nothing noticed.

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

The night I leaked my own payment key, and what caught it

Vinicius Pereira — Tue, 14 Jul 2026 21:30:52 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I build a SaaS for small businesses. Restaurants, small shops, the kind of owner who needs to track sales, stock, cash flow and finances without wrestling a heavyweight ERP. For the first months I was heads-down on features, because that is what pays: the product has to solve a real problem the day you ship it.

Then one night taught me something no feature ever could.

The mistake was entirely mine

During a deploy, a configuration of my own doing exposed a payment API key in production. The exposure window was short, but the risk was not theoretical. Anyone who found that credential could try to use it. In production. Against real money.

Here is the part I got right, and it was as much luck as foresight: weeks earlier I had built an emergency mechanism for exactly the kind of unlikely event I did not believe would ever happen to me.

Every payment integration ran behind a monitor that watched for abnormal usage. A sudden jump in transaction volume. Calls from origins it did not recognize. Sharp changes in behavior. Anything outside the profile it expected. If any of those tripped, the integration did not wait for a human. It moved itself into a protection mode, blocked new operations, and paged me immediately.

That is exactly what happened.

The circuit breaker did its job

The monitor flagged the anomaly and locked the integration into protection mode before a single improper transaction went through. The alert woke me in the middle of the night. I revoked the compromised credential, generated a new key, rolled it across the whole infrastructure, and validated everything before the first customer opened the app in the morning.

No customer was affected. No improper transaction went through. No financial data was exposed.

And none of that made the mistake any smaller.

What I actually learned

The lesson was not "be more careful." Careful is not a strategy. I had been careful, and I still shipped the mistake at the worst possible layer.

The real lesson was that security cannot rest on not making mistakes. It has to assume mistakes will happen, and it has to cap the blast radius when they do. The thing that saved me that night was not my discipline. It was a layer I had built specifically to survive my own discipline failing.

So I rebuilt the security architecture around that idea:

Automated rotation of sensitive credentials, so a leaked key has a short life by default.
Hard segregation between development and production, so the two can never bleed into each other.
Detailed audit logging of critical events, so every sensitive action leaves a trail.
Extra validation during deploy to stop secrets from being exposed in the first place.
Continuous monitoring of every external integration.
Real-time alerts for anything that looks abnormal.
A mandatory security review before any release reaches production.

The question I ask before I ship anything now

Every new feature I build now starts from one question:

If I make a mistake tomorrow at 3 a.m., will the system protect my customers before they notice?

That night changed what engineering means to me. Building is no longer just shipping features. It is building systems that expect human failure and protect the people who trust them anyway.

The bug that night was mine. The thing that smashed it was a layer I had built for a version of me that would eventually slip. Build that layer before you need it. You will need it.

DEV Community: Vinicius Pereira

Before you give: hear what a nonprofit's tax filings say, and what they can't

What I Built

Demo

Code

vinimabreu / before-you-give

Hear what a nonprofit's public tax filings say before you donate. No score, no guessing.

before-you-give

Screens

Why there is no score

How I Built It

Prize Categories

A fake client sent me a GitHub repo. Running it cost me two days and every password I had.

How the main one works

The other flavors

Why the usual signals fail you here

The rules I follow now, no exceptions

The uncomfortable part

The third identity merge wasn't Unicode. It was a placeholder.

The repo, for context

The door nobody was watching

The fix is structural, and it's None

What the fix does not fix

My new repo argued against the exact bug it was carrying

str.lower() is the polite one

Fakes that demonstrate a guarantee are production code

The repo the bug was hiding in

What this does not settle

Python's casefold() merged two of my customers into one tenant

The line that was supposed to be boring

The tenant that was two tenants

Why this is worse than a normal access bug

Then the same bug again, wearing different clothes

Two fixes I tried and threw away

The fix

The general shape of it

Why the repo concentrates risk there on purpose

My exactly-once refund paid twice, and 373 passing tests could never have caught it

The claim

The bug

Why my tests were structurally incapable of finding it

The second thing it broke: a metric that was measuring the wrong noun

What the repo does now

vinimabreu / langgraph-production

A LangGraph support agent with the reliability layer around it: routing measured against labelled fixtures, a human gate before irreversible tools, durable resume with exactly-once effects, and a CI gate that fails when a number moves.

langgraph-production

The report this repo exists to produce

The takeaway I did not expect

The ChocoDEV Signature Bar: one chocolate bar, zero images

Inspiration

Demo

Journey

ChocoDEV: a chocolate bar you can eat, in pure CSS

What I Built

Demo

Journey

Everything that broke when I imported hand-written Make blueprints into a real workspace

1. Import is a parser, not a validator

2. {{1.body}} does not exist, and the failure mode is silence

The ten lines that ended the guessing

3. Collections do not serialize into text fields

4. The default schedule is a 15-minute tick

What the wire showed when it all worked

The outage I did not schedule

What this does not tell you

Your LLM's confidence score is lying to you

Similarity is not confidence

Compute it from what you can check

The gate, as code

Tune on the cost of a false accept

What this does not do

Clear the Lineup: the passage my RAG retrieved and then threw away in silence

Project Overview

Bug Fix or Performance Improvement

Code

Fix silent context truncation in build_prompt #1

My Improvements

Best Use of Sentry

Best Use of Google AI

The night I leaked my own payment key, and what caught it

2. `{{1.body}}` does not exist, and the failure mode is silence