DEV Community: Ender Yentar

Our test clicked a button our users couldn't reach

Ender Yentar — Sat, 19 Sep 2026 02:21:16 +0000

We shipped a small blog editor inside our dashboard. The same evening I opened it and the form was cut off at the bottom of the screen. The body field was half visible. Everything under it, the FAQ section and the publish options, was simply gone. The mouse wheel did nothing. Page Down did nothing.

The end-to-end test for that exact form was green. It filled the form, clicked "Add question" at the very bottom, published the post, and checked that it appeared on the public blog. Every step passed.

So the test could reach a button that no human could reach. This post is about how that happens, and the one-line check that would have caught it.

The bug itself

Our dashboard has a mail view that fills the screen exactly, like a desktop mail client. The list and the reader each scroll on their own, and the outer shell never scrolls. The shell looks like this:

.dash {
  display: grid;
  height: 100vh;
  overflow: hidden;
}

Most pages don't render inside that full-height mode. They render inside a normal scrolling container instead. The code picks between the two with a condition, and that condition was a list of exclusions:

const isMailFull =
  !billing && !profile && !admin &&
  mode === "person" && nav === "inbox-mail";

The blog editor was a new top-level page. I added it to the router and forgot to add it to this list. It opened in the mail mode, so it was rendered inside the fixed shell with overflow: hidden. The form was taller than the screen, and the bottom of it was clipped with no way to scroll down.

The fix was two words, && !blog, plus a comment on that line telling the next person to add new pages there. The interesting part is why the test said everything was fine.

`overflow: hidden` does not mean "cannot scroll"

This is the part I had wrong in my head. An element with overflow: hidden is still a scroll container. It clips its content and it hides the scrollbar. It does not listen to the mouse wheel or to the keyboard. But its scrollTop is still a live property, and anything that scrolls it from code still works: element.scrollTop = 500, element.scrollIntoView(), and focus moving to an element inside it.

Playwright's click() runs actionability checks before it clicks. One of them is: if the element is not in view, scroll it into view. It does that from code, so an overflow: hidden ancestor is no obstacle at all.

I didn't want to put that in a post based on my reading of docs, so I measured it in a tiny page:

import { chromium } from "playwright";

const html = `
<style>
  body { margin: 0 }
  .shell { height: 100vh; overflow: hidden }
  .form { height: 2000px; position: relative }
  #btn { position: absolute; top: 1800px }
</style>
<div class="shell"><div class="form">
  <button id="btn" onclick="window.clicked = true">Add question</button>
</div></div>`;

const browser = await chromium.launch();
const page = await browser.newPage({ viewport: { width: 1200, height: 800 } });
await page.setContent(html);
const shell = page.locator(".shell");

// What a person can do
await page.mouse.move(600, 400);
await page.mouse.wheel(0, 1500);
await page.keyboard.press("End");
console.log("user scrollTop:", await shell.evaluate((e) => e.scrollTop));

// What the test does
await page.locator("#btn").click();
console.log("clicked:", await page.evaluate(() => window.clicked));
console.log("test scrollTop:", await shell.evaluate((e) => e.scrollTop));

await browser.close();

The output:

user scrollTop: 0
clicked: true
test scrollTop: 1200

The wheel and the End key moved the page by zero pixels. The button stayed below the fold. Then click() scrolled the same container by 1200 pixels on its own and clicked the button. From the test's point of view nothing was wrong, because for the test nothing was.

Why nothing else caught it either

This one got past every layer we have. The unit tests don't render the layout. The production build doesn't care about CSS. The end-to-end suite drove the real form in a real browser and the form worked.

My first attempt at a fix was also wrong. I saw a cramped form, decided it was a width problem, and widened the editor. It shipped, it looked better, and the form was still cut off. I had checked the thing I expected instead of the thing that was reported. "I can't get to the bottom of the form" is a scrolling complaint, not a width complaint.

The check that catches it

The test now asserts the thing the user actually needs: the page is in the scrolling container, not the fixed one, and that container really scrolls.

await expect(page.locator(".content .blog-page")).toHaveCount(1);
await expect(page.locator(".content-fill .blog-page")).toHaveCount(0);

const scroller = page.locator(".content").first();
const canScroll = await scroller.evaluate((el) => {
  el.scrollTop = el.scrollHeight;
  return el.scrollTop > 0;
});
expect(canScroll, "blog editor cannot be scrolled").toBe(true);

A caveat on that last check: setting scrollTop from code also works on an overflow: hidden element, which is the whole point of this post. It is meaningful here only because it targets .content, which is supposed to be overflow-y: auto, together with the two assertions above it. If you want a check that doesn't depend on knowing your class names, drive the page the way a person does and then look:

await page.mouse.move(600, 400);
await page.mouse.wheel(0, 5000);
await expect(page.getByRole("button", { name: "Add question" })).toBeInViewport();

mouse.wheel goes through the browser's real scroll handling, so a clipped container stays at zero and toBeInViewport() fails.

Before trusting the new test I put the bug back, reverting the two-word fix, and ran it. It went red. Then I restored the fix and it went green. A regression test that you have never seen fail hasn't shown you anything yet.

What I took from it

Playwright is doing the right thing. Auto-scrolling before a click is what you want in almost every test, otherwise every test would be full of manual scrolling. But it means click() answers "does this button work?" and not "can a person get to this button?". Those are different questions, and a layout bug lives entirely in the gap between them.

So now, for any page that is taller than the screen, one test reaches the bottom the way a user would: with the wheel, and with an assertion on what's visible, before it clicks anything.

We also have a rule written next to that condition now: a new top-level page in the dashboard means checking that list. It's a list of exclusions, and a list of exclusions fails quietly when you forget to add to it. That is probably the real bug, and a good candidate for the next refactor.

I'm building MailFlat, a permanent encrypted inbox with an email API for developers and AI agents. I write about bugs that pass locally and fail in production.

We show your API key once, and here is the option we turned down

Ender Yentar — Wed, 09 Sep 2026 20:59:54 +0000

We shipped a change that takes something away. You used to be able to open an inbox in the dashboard and reveal your API key whenever you wanted. Now the full key appears once, at the moment it is created, and after that you only ever see mf_sk_••••PeH0.

Nobody asked for that. We did it on purpose, and the reasoning is more interesting than the change.

A schema task with a product decision inside it

The task on the board said: drop two plaintext columns. It read like housekeeping. Migration, tests, done.

Before touching it we went looking for everything that reads those columns. One of them fed a button. The inbox modal had a "Show" control that revealed the whole key, and a "Copy" that copied the whole key. Both of them read the plaintext column directly.

So dropping the column was not housekeeping. It answered a question nobody had written down:

can a user see their key again after they create it?

That is a product decision. It was sitting inside a schema ticket, unlabelled, and it would have been decided by whoever happened to run the migration.

What we actually store

Before choosing, we wrote down what is already true, because two of the three options only make sense in contrast to it:

What	How	Reversible
API keys	HMAC-SHA256 with a pepper	no
Passwords	scrypt 32768:8:1 with a salt	no
Messages	RSA-OAEP-2048 plus AES-256-GCM	yes, but the private key lives in your browser

One note, because it comes up every time: base64 is not encryption. It is an envelope, not a lock. If a key is base64 in your database, it is plaintext with extra steps.

Three options

A. Show it once. The key is displayed at creation, with a warning, and never again. Lost key means revoke and create a new one.

B. Encrypt instead of hash. Store the key encrypted with a server-side key, decrypt on request, show it whenever the user asks.

C. Encrypt with the user's own key. We already run end-to-end encryption for messages: the public key encrypts, the private key stays in the browser. The API key could be stored the same way and decrypted client side, with the server still blind.

We chose A.

Why B was rejected

B is the option that keeps every feature. It is also the one that quietly cancels the reason the work existed.

The whole point of hashing is a sentence we get to say: if our database leaks, the keys in it are not usable. Option B replaces that with if our database leaks, the keys are encrypted, and so is the key that decrypts them, which our application can reach.

An attacker with database access is usually one step from application access. B does not add a wall, it adds a drawer next to the wall.

The way we phrased it internally: A lets us say "we cannot see this key either". B turns that into "we cannot see it, but we do keep it". Those are not the same promise, and only one of them survives a bad day.

Why C is on a shelf and not in the bin

C is genuinely good. The infrastructure exists, it is running in production for message bodies, and it would give users their key back without the server ever reading it.

It has one real cost: it forks the system.

Users can turn end-to-end encryption off, and keys created through the API have no browser to decrypt in. So C is not "the same thing, better". It is a second branch that has to be built, tested and explained alongside the first. Two paths where there is currently one.

So we wrote C up as its own plan, and then wrote down what would have to be measured before we build it:

Users regularly landing in a revoke-and-recreate loop, visible as short-lived key rows on the same inbox.
Real support requests along the lines of "I cannot see my key".

Both are currently zero. Not "we think they are low". Zero, with a query behind it.

That last part is the bit worth stealing. "We might do this later" is a way of avoiding a decision. "We will do this when this number stops being zero" is a decision, and it has an owner and a trigger.

Shipping it

Four code paths read or wrote the plaintext column: the "does this inbox already have a key" check, the write on creation, the clear on revoke, and the list endpoint, which was returning the full key in its response body.

Authentication was not one of them. It had been moved to the hash a few weeks earlier, which is the only reason this change did not touch the login path at all. Being able to say that, with a line number, was worth more than any amount of confidence.

The revoke button stayed exactly where it was. If you lose your key, that is your exit, and removing the reveal makes revoke more important rather than less.

The proof that counted

Production verification was five measurements on a throwaway inbox:

Inbox creation returns no key
Key generation returns the plaintext once
Both plaintext columns are null afterwards
The list endpoint returns a masked string, not the key
A live HTTP request using that key returns 200

The first four say nothing leaks. The fifth says it still works. Either one alone is a trap. A system that leaks nothing because it is broken passes 1 through 4 perfectly.

We are still running with the old columns in place. They will be dropped in a separate step, after a week of the new path being live, with a separate approval. There is no reason to combine an irreversible migration with a behaviour change on the same day.

What we would tell ourselves a week earlier

Before you run a migration, grep for what reads the column. Not to check the code compiles, but because a column that feeds a button is not a storage detail, it is a promise to a user, and someone should decide about it on purpose.

We build MailFlat this way throughout: the server holds ciphertext it cannot read, and the parts we cannot recover for you are the parts we are proudest of.

Your backfill is a photograph

Ender Yentar — Wed, 09 Sep 2026 03:24:07 +0000

We were getting ready to make a column NOT NULL. Standard preparation: count the rows that would violate it. The count was not zero.

Seven API keys had no owning organization. That looked like a key problem, so we went looking at how keys are created.

It was not a key problem.

The keys were innocent

All seven belonged to the same account. And that account had no organization either. Widening the query, two accounts were in that state. Both had signed up recently, within the same three week window.

That reframed the question. Keys read organization_id off the user who owns them. If the user has none, every key that user creates is born without one. The keys were downstream of something else.

The gap

Two weeks earlier we had run a migration that created an organization for every existing user and linked them up. It worked. Every row we had at that moment was correct.

Then we shipped it and moved on.

Nobody wired the signup path.

The proof took one command:

grep -rn "Organization(" backend/app/

Zero hits in application code. The only places that constructed an Organization were the migration itself and the tests. So from the moment the migration finished, every new account was born outside the ownership chain, and the data drifted a little further from correct with each signup.

Why nobody noticed for two weeks

Because nothing read the column yet.

The organization was groundwork for permission checks that had not shipped. No request failed. No error was logged. No user saw anything wrong. The system behaved exactly as it had before, because the broken part was not load bearing.

This is the uncomfortable shape of the bug: it produced no symptom, and the absence of a symptom is what let it grow. It only surfaced because we went looking for something else, and it had been quietly blocking that something else the whole time.

We had already written the lesson down

While reading the code, we found this in a helper module, in a comment above a completely different function:

# A backfill is a photograph. A dual write is keeping the photograph current.

Someone on this project learned that lesson, wrote it down, applied it to inboxes, and then did not apply it to users. The knowledge was in the repository. It just was not attached to the thing that needed it.

A backfill answers "what is true right now". It cannot answer "what stays true tomorrow". Those are two different jobs and shipping the first one feels like finishing.

The fix we did not make

The obvious fix is to create the organization at signup. We counted the paths that create a user:

Normal email signup
Google sign-in
Two separate billing paths
Admin seeding
The demo system user

Six call sites. Adding the same three lines to six places would have made today's bug six times more likely, not less, because the seventh path gets written next month by someone who never reads this post.

So we took the guarantee off the caller entirely. It is a session level hook that runs before every flush:

@event.listens_for(Session, "before_flush")
def _attach_personal_organization(session, flush_context, instances):
    for obj in session.new:
        if isinstance(obj, User) and obj.organization_id is None:
            # Assign the relationship, not the foreign key. See below.
            obj.organization = Organization()

It only looks at session.new, so flushing an existing user a second time does not open a second organization. If organization_id is already set, it leaves it alone, which is what lets the backfill migration and the tests do their own thing.

Now it does not matter which path creates the user, or whether that path knows organizations exist.

There is a real cost to this and it is worth saying out loud: grep -rn "Organization(" still finds nothing useful in the application code. We traded an explicit call for an invisible one. The mitigation is signposting: the model module says in its docstring that organizations are born in a before_flush event and that grepping will not find the call. If you use this pattern, write that sign, because the next person will grep first.

The test says the same thing out loud. It does not go through the signup endpoint, because that would only prove the endpoint works:

def test_guarantee_does_not_depend_on_the_caller(client):
    s = SessionLocal()
    u = User(email="raw-insert@test.com", username="rawinsert", password="x", plan="free")
    s.add(u)
    s.commit()
    assert u.organization_id is not None

If someone later moves the guarantee back into the signup handler, this test goes red. That is its entire job.

Two things that bit us on the way

You cannot flush inside before_flush. The first version added the organization, flushed to get its id, and assigned it to the foreign key. SQLAlchemy raises "Session is already flushing" and all four tests fail at once. The fix was to stop thinking in ids: assign the relationship, and let SQLAlchemy work out the insert order and the key.

The backfill migration passed locally and failed on Postgres. It used sa.table(), the lightweight construct, which has no primary key definition, so inserted_primary_key comes back empty and indexing it raises. sa.Table() with an explicit primary key column fixes it.

We never would have seen this locally, because our test suite runs on SQLite and Alembic does not run there at all. A small staging box with the production schema caught it. That is a separate story, and it is the same story.

What we do now

After any backfill, three questions before it counts as shipped:

What creates new rows of this kind, and does that path set the field?
If the answer is "nothing reads it yet", what will read it, and when?
Is the guarantee attached to a caller, or to the data?

The first question is the one we skipped. The third is the one that actually fixed it.

Production is clean now: no accounts without an organization, no keys without an owner, no broken links between the two. The proof was taken inside a transaction that was rolled back, so verifying the fix did not create a row to explain later.

We hit this while building MailFlat, where every API key belongs to an inbox, which belongs to an account, which belongs to an organization. A chain is only as good as the day someone forgets to attach the next link.

SQLite doesn't enforce foreign keys by default, and it cost us three bugs

Ender Yentar — Tue, 08 Sep 2026 21:17:38 +0000

Our test suite runs on SQLite. Production runs on PostgreSQL. That is a common setup and for a long time it felt free: tests were fast, isolated, and green.

Over about three weeks we hit three separate bugs that all came from the same gap. None of them were caught by 1200+ passing tests, because the tests were running on an engine that quietly forgives what production rejects.

Here they are, in the order we found them.

1. Two tests that were red in production and green locally

We were making a schema change, so we ran the suite against a real Postgres instance for once. Two tests failed. On SQLite the same commit was 1241/1241 green.

First reaction: today's change broke something. We stashed it and re-ran against a fresh Postgres database. The same two tests failed. So they had been broken for a while and nobody had seen it.

Both were test bugs, not product bugs:

# To simulate "an inbox owned by someone else"
other_user_id = box.user_id + 999   # a user id that does not exist

SQLite accepts that row. Postgres rejects it with inboxes_user_id_fkey. And the test was weak on top of being broken: it wanted to prove "another person's inbox is not accessible", but what it actually created was "an inbox owned by nobody".

The second one was subtler:

owner = session.query(User).filter_by(is_admin=True).first()   # no ORDER BY

An unordered .first(). SQLite happened to return the admin the test needed. Postgres returned a different admin, and the test failed looking for an inbox that belonged to someone else.

Neither bug was in the product. Both were in tests that had been passing for weeks while measuring the wrong thing.

2. A migration that could not fail locally, because it never ran locally

Later we wrote a data migration to backfill some rows. It looked fine. It passed review. Then we ran it against Postgres and it blew up:

organizations = sa.table("organizations", sa.column("name"), sa.column("region"))
res = conn.execute(organizations.insert().values(...))
org_id = res.inserted_primary_key[0]   # IndexError: tuple index out of range

sa.table() is SQLAlchemy's lightweight table construct. It has no primary key definition, so the driver never asks for the generated id and inserted_primary_key comes back empty. The fix is a real sa.Table() with an explicit primary key column.

The point is not the API detail. The point is this: our test suite does not run Alembic at all. Tests create the schema directly. So migration code the code that runs against production data, once, with no undo was the least tested code in the repository. Not under-tested. Zero lines executed.

We caught this one on a staging box with real Postgres. If we had not had one, we would have found it in production.

3. Deleted rows that came back to life

The third one was the strangest.

We changed a uniqueness check to ask the api_keys table instead of a legacy column:

if session.query(ApiKey).filter_by(inbox_id=inbox.id, revoked=False).first():
    return {"error": "This inbox already has a key."}

Suddenly, brand new inboxes started reporting that they already had a key.

The chain took a while to see:

api_keys.inbox_id has ondelete="CASCADE". In production Postgres, deleting an inbox deletes its key rows.
SQLite does not enforce foreign keys unless you turn them on. So in tests, deleting an inbox left its key rows behind as orphans.
SQLite reuses row ids. A newly created inbox could take the id of a deleted one, and inherit its orphaned key rows.

So a fresh inbox "already had a key" a key belonging to an inbox that had been deleted.

The old code never saw this, because the old check read a column on the inbox itself, and on a new inbox that column is NULL. The bug was not new. It became visible.

The fix, and what it does not fix

One line, in the engine setup:

@event.listens_for(engine, "connect")
def _sqlite_pragmas(dbapi_connection, connection_record):
    cursor = dbapi_connection.cursor()
    cursor.execute("PRAGMA foreign_keys=ON")
    cursor.close()

SQLite ships with foreign key enforcement off for backwards compatibility. It has been that way for years and it is documented, but the default is silence: no warning, no error, just a database that accepts rows Postgres would refuse.

Turning it on made our tests behave like production. Case 3 disappeared. Case 1 would have been caught the day it was written.

But be clear about what one pragma does not do:

It does not make SQLite run your migrations. Case 2 is still invisible locally.
It does not change id allocation. SQLite still reuses ids; Postgres sequences do not.
It does not make the two engines equivalent. It closes one specific gap.

What we changed beyond the pragma

Migrations get a real Postgres before they run anywhere else. Not the full suite, just the migration, against a database that has the production schema. Ours runs on a small staging machine. Twice now, that box caught something that SQLite structurally could not: once the migration above, once a duplicate revision id that had forked the migration tree.

"Green" now has a scope. A green suite means "green on SQLite". That sentence used to be implicit and it was doing a lot of quiet damage. Writing it down changed how we read the result.

The uncomfortable version of the lesson: for three weeks our tests were measuring a different database than the one our users touch. They were not lying. We were reading them as if they said more than they did.

We hit these while building MailFlat, an email API where agents and test suites get real inboxes. The bug journal these came from is kept per-incident, symptom first, which is why three separate weeks turned out to be one story.