A Redirect Is Not a Receipt: The Automation Bug That Told Me It Succeeded

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The worst bugs don't crash. They smile at you, hand you a green checkmark, and send a "success!" email while the actual work quietly didn't happen. I shipped one of those. It cost me trust before it cost me anything else — because for a while, I believed it.

Here's the story of a false-success bug in a browser automation, why the check passed when the action had failed, and the one-line reframe that fixes this whole class of mistake: a redirect is not a receipt.

The setup

I have a small Playwright automation that drives a web editor: open a draft, fill it in, click Publish, done. To decide whether the publish worked, I used the most natural signal I could see — the URL. In this app, a draft lives at a URL ending in /edit. Once you publish, the app navigates you away from /edit to the published post. So my success check was, essentially, "did we leave the edit page?"

// click Publish, then wait to leave the editor
await publishButton.click();
await page.waitForURL((u) => !u.pathname.endsWith("/edit"), { timeout: 25000 });
// got here? must be published. 🎉
return page.url();

It worked in every test. It worked for weeks. Then one day it lied to my face.

The day it lied

I ran the automation. Console said PUBLISHED. I got the "🟢 published" email. I went to look at the live post to admire it — and it wasn't there. The content was still sitting in the draft, untouched.

The action had been rejected — the platform rate-limited me — but my automation reported a clean success anyway. Worse than a crash: a crash tells the truth. This handed me a confident lie and an email to match.

Why the check passed when the publish failed

I pulled the real URL the automation ended on and saw it:

/p/<id>/submission?redirectUrl=%2Fp%2F<id>%2Fedit

Read that carefully. When the publish was rejected, the app bounced the request through an intermediate /submission URL whose query string contained the edit path (redirectUrl=…%2Fedit), before dumping me back on the editor with a red banner.

Now look at my check again: !u.pathname.endsWith("/edit"). The pathname of that bounce URL is /p/<id>/submission — it does not end in /edit. The /edit is in the query string, which pathname ignores. So my "did we leave the edit page?" test returned true… on a page that was, functionally, still the edit page mid-rejection.

I had confused a side effect (the URL changed) with a confirmation (the post is live). Those are not the same thing, and the gap between them is exactly where this bug lived. My "success" signal was a proxy for success, and the day the proxy and the reality disagreed, I believed the proxy.

The fix: confirm the postcondition, don't infer it

The repair wasn't a smarter URL regex. It was to stop inferring success from navigation and instead poll for the real outcome — either a genuinely published post URL, or the rejection banner that tells me it didn't work:

// Resolve the TRUE outcome: a real published URL, or an explicit rejection.
let outcome = { published: false, reason: "timeout — no nav, no banner" };
for (let i = 0; i < 20; i++) {
 await page.waitForTimeout(1000);
 const s = await page.evaluate(() => {
 const txt = document.body.innerText || "";
 // the platform's own rejection / rate-limit banner
 const rejected = /maximum of.* in the past 24 hours/i.test(txt);
 return { path: location.pathname, rejected };
 });
 if (s.rejected) { outcome = { published: false, reason: "rate-limited" }; break; }
 // a bounce lands on /submission or stays on /edit — NOT success
 const stillEditing =
 s.path.endsWith("/edit") || s.path.includes("/submission");
 if (!stillEditing) { outcome = { published: true, reason: "live URL" }; break; }
}
return outcome;

Two changes carry all the weight:

1. Positive confirmation, not absence of the old state. "Not on /edit" is not proof of publication. "On a real post URL" is. I now require the success state to be present, not merely the failure state to be gone.
2. Make failure loud. The platform was already telling me it rejected the publish — there was a banner. I just wasn't reading it. The automation now watches for that banner and reports rate-limited honestly, with a truthful email, instead of faking a win.

Before / after

• Before: URL no longer ends in /edit → "published" → success email. Silently wrong the moment the app used a redirect bounce.
• After: poll until either a real post URL (published) or the rejection banner (not published); report exactly what happened. No more confident lies.

What I learned

The lesson generalizes far past this one script. Any automation that decides success from an easy-to-observe side effect — a redirect, an HTTP 200 on a page that renders its real error in the body, a file that exists but is half-written, a job that "started" — is one edge case away from cheerfully reporting success on a failure. The failure modes hide in the gap between "something changed" and "the thing I wanted is true."

So: verify the postcondition, not a proxy for it. And when the system is already shouting its failure at you in a red banner, read the banner. A navigation is not a confirmation. A redirect is not a receipt.

The bug that smiles at you is worse than the one that screams. This one is smashed — now it screams when it should.