DEV Community

Jennifer Smith
Jennifer Smith

Posted on

I saw the broken email in the preview, blamed my phone, and approved the send anyway.

Three days after I sent an email to the subscriber list of one of my brands, a reader replied with a screenshot. The subject line was fine. The sender was fine. The body was a smear: every line of text stacked onto the same narrow band of pixels, sentence printed over sentence until the whole message was one unreadable stripe.

That screenshot was the first evidence of what the send had actually done inside real inboxes. The template bug itself had been caught and fixed the same day the email went out — I will come back to that, because it makes this story worse rather than better — but the fix faced forward, toward future sends, and nobody looked back at the copies already delivered. So the first report from the receiving end was not a bounce report, a monitor, or a dashboard alert. It was a human being, doing unpaid quality assurance for a list they had voluntarily joined, three days after the fact.

This is the postmortem, and the four guards that now exist so no reader ever holds that job again.

What actually broke

The email platform I use has its own template format, and that format expresses line height as a percentage of font size. I had given it the value 1.65 — a perfectly ordinary number if you have spent your life reading it as a CSS ratio, where it means "165% of the font size, comfortable reading rhythm."

The template format read it as 1.65 percent.

At 1.65 percent of the font size, every line of the email is drawn essentially on top of the line before it. The platform did nothing wrong: it stored the value I gave it and served it faithfully to every inbox on the list. One number, valid in one language and catastrophic in a neighboring dialect, and no part of my pipeline knew the difference.

Why every gate waved it through

The inventory is uncomfortable, and the pattern only becomes visible when you line the gates up.

The words themselves were right. I had read them, revised them, and read them again, and they said exactly what I meant. But every check that email ever received was a check of words, and words are not what an inbox renders.

I approved the send from my phone, between other things, and the preview in front of me looked wrong. The lines were crushed together, and I could see that they were. I decided the phone was the problem — small screens do strange things to email previews all the time — and I approved the send anyway. That is the worst line in this inventory, so I am not going to soften it: the system showed me the failure before it shipped, and I explained the evidence away.

The platform accepted the template upload without complaint, because 1.65 is a legal value for the field. Legal and correct are also different sentences.

And nothing anywhere in the pipeline ever looked at the published artifact — the thing the platform actually stored after the upload and would actually serve — as opposed to the thing I intended to upload. Twelve people received the smear. When the bug surfaced later that same day, the template got fixed and the pipeline moved on, but the twelve copies already sitting in inboxes belonged to nobody's checklist. The email had announced a time-limited offer, its original window expired while the announcement was illegible, and the offer ended up being extended. For three days, the only sensor pointed at the subscriber side of the system was the goodwill of readers.

The pattern underneath

This estate has published a postmortem before. In that one, an automated pipeline shipped a two-year-old news story as if it were new — and the gate that should have caught it was not missing; it existed and stood at the wrong door. This failure is the same species, and I think its most general form, because once again nothing was absent. Every gate I had was present, functioning, and pointed at the wrong object.

Every gate audited intent. None audited the artifact.

By "artifact" I mean the thing that ships: what the third-party platform stored and will serve to a real inbox — not the local file, not the draft in the editor, and not the words I approved. Config files, deploy logs, reviews, and approvals are all claims about a system. The artifact is the fact.

Once you see the pattern, you also see how common it is. Anyone who publishes through a platform they don't control — an email service, a CMS, an app store, a template engine — has a gap between the thing they approved and the thing that ships, and most pipelines audit only the near side of it.

The four guards

I pointed my agent at the problem the same day, and we built four layers. I contributed the part that twenty-five years of infrastructure actually trains: knowing the names of the guards an organization would deploy, and insisting on all of them instead of the first one.

1. Round-trip every publish. The sync script that pushes a template to the platform now immediately fetches back what the platform stored and validates that, not the local copy. A bad value can no longer survive a publish silently, because the publish itself reads its own result.

2. Gate the send on the artifact, and treat "cannot audit" as "do not send." The bulk-send tool now fetches the live template and lints it before the first recipient. It refuses to send if the check fails — and it equally refuses if the check cannot run, because the template is unpublished, the API errors, or the format is one the linter cannot parse. Unverifiable is not safe. That sentence took me three days and a screenshot to learn, and it is the one I most want you to take.

3. Make approval mean "I saw what recipients see." The send command now delivers one real copy to my own inbox and stops. The bulk send fires only when I run it again with a flag whose only meaning is "I looked at the proof." This is the guard that covers everything the other three cannot, because it catches the failure modes no linter anticipates yet — and it is aimed squarely at the excuse I used, because a real delivery rendered in a real mail client is the artifact itself, not a preview you can second-guess. If the proof in my inbox looks broken, there is no "it's probably just the phone" left to reach for.

4. Sweep daily. A watchdog now lints every live template every morning and raises a flag only on failure. Templates can drift between sends — a stray edit in the platform's UI that no script ever saw — and the sweep exists so drift gets caught by a machine before breakfast instead of by a subscriber after dinner.

There is also a fifth item, which is not a guard but a discipline: name your blind spots out loud. Part of this platform's content lives in automation sequences the API cannot read, so no script can lint it. The guard prints that limitation in every single run instead of silently skipping it, because an audit that quietly narrows its own scope reports coverage it does not have. After any manual edit to those sequences, the check is my own eyes, and the tooling reminds me of exactly that.

The watchdog's first sweep ran the day the guards landed and came back clean across every live template. The house rule holds here as everywhere: it is measured, not promised.

Steal this

The condensed version, for any pipeline that publishes through a platform you don't control:

  • After every push to a third-party platform, read back what it stored and validate that.
  • Bulk operations refuse to run when the pre-flight audit fails, and equally when it cannot run at all.
  • Approval requires seeing the rendered artifact: proof-to-yourself before send-to-everyone.
  • A daily sweep of the live artifacts, alarming on failure only.
  • Any surface the audit cannot reach gets named in the audit's own output, every run.

The honest coda

The embarrassing part of this story is not the attribute. Numbers that mean different things in adjacent formats are a tale as old as software, and the platform behaved exactly as documented.

The embarrassing part is that I saw it. The preview on my phone showed me the damage before the send, and I dismissed it as a rendering quirk of the device, because the alternative was inconvenient. Twenty-five years of infrastructure taught me that an anomaly is a claim that deserves investigation before dismissal, and I have said versions of that sentence to entire organizations. At my own kitchen table, with my own send, I treated the anomaly as noise. Every professional instinct I have would have flagged that override in someone else's operation, and I performed it in my own without a second thought.

The subscriber who replied did a job my monitoring should have been doing. They did it for free, three days after the send, because on the receiving side of my system a reader with a screenshot was the only sensor actually deployed. The bitterest detail is that the bug was already fixed by then — fixed for the future, on the very day it shipped, while the past sat unread in twelve inboxes. Repairing a system and repairing its consequences turn out to be different jobs, and I had only staffed the first one. The four guards exist so that the next render failure — and there will be one; there is always one — is caught by a machine at seven in the morning instead of a reader at dinner.

This estate is supposed to run while I sleep. The readers should get to sleep too.

Top comments (1)

Collapse
 
reidmarlow profile image
Reid Marlow

That phone-preview bit is painfully familiar. The guard I keep coming back to is saving a rendered artifact, not just approving the editor state. For email that probably means one screenshot per target client, tied to the exact campaign id, so a fix after send cannot quietly rewrite history.