I planted a security bug to test my own CI. My merge gate let it through.

I built Nudge for the TestSprite Hackathon (Season 4). It's invoicing for freelancers with one feature I've always wanted: read receipts. Every client gets a private portal that shows only their own invoices. The moment a client opens one, the freelancer's dashboard flips from Sent to Viewed, live, without a refresh.

That feature is the easy part. The hard part is the sentence before it: only their own invoices.

This post covers how I made the tests prove that on every pull request. Then I deliberately broke it, and found out my safety net had a hole in it.

• Live demo (one click, no sign-up): https://nudge-spike.vercel.app
• Code: https://github.com/hackid02/nudge-spike
• 2-minute video: in the repo README

The bug I was most afraid of

Picture one freelancer, Maya, with two clients: Northwind and Lumen Labs.

Cross-freelancer isolation is easy to remember. Most apps get it right because every query already filters by the logged-in user. Cross-client isolation inside one freelancer's account is where real apps leak. Both clients belong to the same tenant, so a lazy where freelancer_id = ? looks correct and isn't.

In Nudge, ownership lives in the SQL WHERE clause, not in the page:

select i.id, i.status, ...
  from invoices i
  join users u on u.id = i.client_id
 where i.id = ${invoiceId}
   and i.client_id = ${clientId}   -- the line this whole post is about
   and i.status <> 'draft'

If that returns nothing, the page renders one generic message: "Invoice not found." That message is the same for another client's invoice, another tenant's invoice, your own unsent draft, and an ID that doesn't exist. Anything more specific would confirm that the invoice exists.

Tests that assert absence, not "an error appeared"

A test that checks "the page shows an error" passes on a 200 response that happens to show someone else's data next to a toast. So the isolation tests assert the absence of specific values:

Sign in as Client A. Open /portal/invoices/INV-1002 (Lumen's invoice). Verify the page shows "Invoice not found." and does not contain INV-1002 or $640.00.

The full suite is 15 TestSprite end-to-end tests:

• tenant isolation, including URL guessing, drafts, another freelancer's data, and both clients' CSV exports
• role routing and signed-out redirects
• an exact wrong-password message
• invoice totals to the cent
• send, record payment, void (refused on paid invoices), and delete with confirmation
• the read receipt itself

Every PR gets its own database

Shared test data is how you get flaky end-to-end tests. If two PRs run against the same database, one run's "send invoice" breaks the other run's "invoice is a draft".

So each pull request gets:

1. Its own Vercel preview at a predictable URL: nudge-spike-git-{branch}-….vercel.app
2. Its own Neon database branch, created by the Neon–Vercel integration.
3. A fresh seed at build time. vercel-build runs a reset script before next build. The script guards against ever touching production:

if (process.env.VERCEL_ENV !== 'preview') {
  console.log('[preview-reset] not a preview, skipping (guard)')
  process.exit(0)
}

When the deployment reports ready, the TestSprite GitHub App runs the suite against that preview, posts a TestSprite V3 status and a comment on the PR, and a ruleset on main requires that status.

Fixtures are split, too. Read-only invoices (INV-10xx) are never modified by any test. Each test that changes data owns exactly one disposable invoice (INV-11xx). Seed dates are relative to now(), so nothing drifts into Overdue halfway through judging.

What the suite caught first

Before any of the planted-bug drama, a real result: the test "Maya can delete a draft invoice after confirming" came back Blocked.

Delete used the browser's native window.confirm(). Automated browsers close native dialogs on their own, so the Cancel path could never be exercised, by TestSprite or anyone else's harness. I replaced it with an in-page confirmation dialog (role="alertdialog", Cancel and Escape close it, Confirm submits). On the next PR the test clicked Cancel, checked the invoice was still there, clicked Delete again, confirmed, and checked it was gone. Blocked became Passed.

It's a small thing, but it's the loop doing its job: the test exposed a UX choice that was also a testability bug.

Then I tried to break it on purpose

Every PR so far had been green. That bothered me. A gate you've never seen fail isn't a gate. It's a decoration.

So I opened PR #10, titled "DO NOT MERGE: planted tenant leak". It deletes exactly one line:

  where i.id = ${invoiceId}
-   and i.client_id = ${clientId}
    and i.status <> 'draft'

Locally, my HTTP isolation script confirmed the leak: Client A could now read Lumen's INV-1002 and another freelancer's INV-2001.

TestSprite ran on the PR's own preview and database. It caught it:

Ran into trouble — 1 failed. Client A could open another client's invoice directly. The page displayed invoice INV-1002, billed to Lumen Labs. The total shown was $640.00.

Then I looked at the status check.

TestSprite V3: success — 14/15 passed · 1 failed

Success. With a failing security test. GitHub said the PR was mergeable.

The hole in my safety net

The tests worked. The ruleset worked. The missing piece was a toggle on the TestSprite trigger: Merge gate: "Report results as a status check, blocking merge on failure." It was off. With it off, the run reports its findings in a comment but sends GitHub a passing status. My required check was guaranteed to be green no matter what the tests found.

Every PR since TestSprite was wired in had come back green, and I believed my merges were gated. They weren't. Those PRs happened to pass, so nothing slipped through, but I only learned the gate was open because I tried to walk through it.

I switched the toggle on and re-ran the same code:

TestSprite V3: failure — 14/15 passed · 1 failed, marked Required, and the merge button greyed out. I closed the PR unmerged. main and production were never touched.

What I'd tell anyone wiring up PR testing

1. Plant a bug before you trust the gate. One deleted line, one PR, closed unmerged. It's the cheapest proof you can get, and it found a real misconfiguration in my setup.
2. Check the status, not just the comment. The bot comment said "1 failed". The status GitHub enforces said "success". Those are two different things.
3. Give every PR its own data. Per-branch databases made every red check reproducible and every green check meaningful.
4. Assert absence. For access control, "the secret value is not on the page" beats "an error appeared".
5. Write down what isn't built. Nudge doesn't create invoices, invite clients, send email or take payments. I spent the week making isolation and the invoice lifecycle provably correct instead, and the README says so.

The rest of the setup

• A second, credit-free test layer: GitHub Actions runs npm test (21 checks on the money maths: half-up rounding, tax in basis points, formatting) on every PR.
• The live demo is shared by every visitor, so a nightly GitHub Action reseeds it to the known fixtures.
• Stack: Next.js (App Router), Postgres on Neon, Tailwind v4, Vercel previews, TestSprite.

Try the attack yourself: open the live demo, click Client A, and go to /portal/invoices/INV-1002. You'll get "Invoice not found." That's the point.

Built for the TestSprite Hackathon, Season 4.

Comments

[Ananya Soni]

solid breakdown on testing the CI gate itself rather than just trusting the green check. finding out the hard way that the status check toggle was off is a classic production reality check. asserting the absence of unauthorized data in access control tests instead of just looking for an error toast is also the right call.

curious what else you are working on besides this. I spend most of my time building artificial general intelligence, multimodal AI architectures, and trying to replicate human brain mechanisms as founder and CEO of AGI systems directorate.

also totally agree on Vercel for preview deployments. I run my own server infrastructure for core workloads, but I use Vercel for quick preview releases whenever my own server is down for maintenance.

platforms like dev.to often feel like quiet documentation diaries where new posts get zero initial engagement, so figured I would drop a proper comment. always down to connect and collaborate with other developers actually shipping code. what is next on your roadmap?
And a literally want to connect with other developers working in the field